Fsdss-536

FSDSS-536 — Implementation & QA Guide

8. Documentation & Runbook

  • README: setup, env vars, migrations, seed data.
  • Runbook: steps for investigating failed sync job, tail logs, requeue job, clear cache.
  • On-call guide: escalation, known issues, rollback.

3. Implementation Plan (Sprint-sized tasks)

Sprint 1 (2 wk)

  1. Schema migrations: create tables.
  2. Backend: basic read APIs (GET /syncs, /syncs/id).
  3. Unit tests for DB layer.
  4. Frontend: dashboard skeleton, fetch & display list.

Sprint 2 (2 wk)

  1. Worker to aggregate status and write to sync_jobs.
  2. Caching (Redis) + TTL invalidation on updates.
  3. WebSocket/SSE for live updates.
  4. E2E tests for list+details.

Sprint 3 (1 wk)

  1. History endpoint, pagination.
  2. Alerts integration (Slack/PagerDuty) on failure states.
  3. Performance tuning and load tests.
  4. Documentation and runbooks.

1. Overview

| Item | Description | |--------------------------|-------------| | Ticket Summary | Add support for bulk import of transaction records from CSV files into the Financial‑Services‑Data‑Sync‑Service (FSDSS). | | Primary Goal | Enable users to upload a CSV (up to 50 k rows) and have the service parse, validate, and persist the records atomically, returning a detailed import report. | | Scope | – New REST endpoint /api/v1/transactions/import
– CSV parser utility (CsvTransactionParser)
– Validation layer (TransactionValidator)
– Bulk‑insert service (TransactionBulkService)
– Import‑report DTO (ImportReportDto)
– Integration tests and Swagger documentation. | | Related Tickets | FSDSS‑421 (single‑record import), FSDSS‑487 (audit‑log enhancements). | | Branch | feature/FSDSS-536-bulk-import | FSDSS-536

The change introduces a substantial new capability that will be heavily used by downstream reporting tools, so correctness, performance, and observability are critical. FSDSS-536 — Implementation & QA Guide 8

B. Configuration diff (pre‑ vs post‑incident)

| Property | Pre‑Incident (v3.2.5) | Post‑Incident (v3.2.7) | |----------|----------------------|------------------------| | enable.auto.commit | true | true | | auto.commit.interval.ms | 5000 | 5000 (unchanged) | | max.poll.records | 500 | 500 | | Added | – | commit.retry.backoff.ms = 2000
commit.max.retries = 5 | README: setup, env vars, migrations, seed data

Components

  • Backend API (HTTP/JSON)
  • Data store (Postgres)
  • Cache (Redis)
  • Frontend dashboard (React)
  • Worker(s) for status aggregation
  • Alerting (PagerDuty/Slack)

6. Security

  • File Validation – Only text/csv MIME type is accepted. The controller also checks the file extension for .csv.
  • Input Sanitisation – All fields are trimmed and validated before persisting. No direct SQL concatenation; JPA parameter binding is used.
  • Rate Limiting – Not yet implemented on this endpoint; could be a vector for DoS (large files repeatedly). Recommend adding a per‑user rate limit (e.g., 5 imports per hour).

6. Deployment & Rollout

  • Feature-flag backend and frontend (ff: fsdss_536).
  • Canary: Deploy backend to 10% of traffic for 48h; monitor errors, latency, Redis hit rate.
  • Gradual rollout: 25% → 50% → 100% over 3 days if metrics stable.
  • Rollback plan: revert flag + deploy previous release; run DB downgrade only if safe.

6. Actual Result

The process hangs at “Scanning directory” after processing ~9 950 files. No further progress is logged. The UI shows a spinner indefinitely.