Skip to main content

Distributed DICOM Routing & Processing Platform

Project Overview​

A scalable, production-grade platform for distributed routing, processing, and storage of DICOM (and other medical) files. The system is designed to solve the pain points of hospital IT: complex manual routing, fragmented storage, lack of centralized management, and poor automation.

Key capabilities:

  • Distributed SCP/SCU clusters for parallel image ingestion and delivery
  • Dynamic load balancing and scaling via automated HAProxy configuration
  • Modular processing pipeline: from metadata extraction to future AI/ML-powered image analysis
  • Real-time system monitoring and notification stack
  • Centralized admin interface for routing, node management, and live observability
  • Extensible design, ready to handle any medical file format and future cloud integration (S3/MinIO)

Architecture​

DICOM Platform Architecture

The platform consists of several core service groups:

  • SCP/SCU Cluster:
    Multiple SCP (Service Class Provider) and SCU (Service Class User) instances process and route incoming DICOM files, balancing load and ensuring fault tolerance.
  • HAProxy:
    Load balancer automatically configured and managed via a custom Docker Helper service, allowing horizontal scaling of SCP instances based on real-time database settings.
  • Processing Pipeline (Workers):
    Distributed worker nodes extract DICOM metadata (tags), persist them to PostgreSQL, and prepare files for further actions (storage, forwarding, or AI analysis). The architecture enables rapid integration of AI/ML models for tasks like segmentation, anomaly detection, or data anonymization.
  • Centralized Storage:
    Files are stored locally at first; the design allows seamless expansion to cloud storage backends (S3, MinIO, or remote servers) and supports backup/archival strategies.
  • Monitoring & Notifications:
    Full observability via Prometheus (metrics), Grafana (dashboards), Loki & Promtail (logs), and cAdvisor (container health). Real-time system status, node workload, and notifications are available via the web UI.
  • Web UI & API:
    Built with React and FastAPI, provides a centralized admin panel for configuring routing rules, managing node health, monitoring load, and handling live notifications (via WebSockets).
  • DevOps & Automation:
    All components orchestrated via Docker Compose; the Docker Helper service handles initialization, database migrations (Alembic), and dynamic reconfiguration.

Technologies Used​

  • Backend: Python (FastAPI, FastStream, pynetdicom), Redis (stream/caching), PostgreSQL (metadata, routing)
  • Processing: Modular worker nodes, AI/ML-ready pipeline (plug-and-play for future deep learning integration)
  • Orchestration: Docker Compose (multi-container, multi-node), custom Docker Helper for HAProxy and lifecycle automation
  • Frontend: React (dashboard, admin, notifications), WebSockets for real-time updates
  • Load Balancing: HAProxy (dynamic configuration)
  • Monitoring: Prometheus, Grafana, Loki, Promtail, cAdvisor
  • APIs & Integrations: REST API (FastAPI), future SCU integrations (C-MOVE, C-FIND)
  • Security & Access: Ready for further extension (role management, certificate management, etc.)

Key Features​

  • Distributed Routing:
    Incoming DICOM files are received by a cluster of SCP instances, balanced by HAProxy. Each file is stored locally and routed according to dynamic policies set via the web interface.
  • Automated Forwarding & Backup:
    Flexible routing engine allows forwarding to external PACS, cloud storage, or backup destinations. Local storage is always performed first to ensure data safety.
  • Processing Pipeline:
    Worker nodes extract DICOM metadata and can easily be extended to perform advanced analytics (AI/ML, anonymization, etc.).
  • Centralized Management UI:
    React-based dashboard enables real-time monitoring, configuration, and operational control. Live notifications and system health status are available via WebSockets.
  • Scalable & Fault-Tolerant:
    Horizontal scaling for both SCP/SCU and worker nodes. HAProxy and Docker Compose ensure smooth addition/removal of nodes.
  • Observability:
    Metrics, logs, and system health are collected, visualized, and can be sent to the admin interface for proactive management.
  • Future-Proof Design:
    Architecture is modular, supporting new processing stages, additional data formats, cloud storage, and advanced routing scenarios.

Typical Workflow​

  1. Ingestion:
    DICOM files arrive from PACS sources to the HAProxy endpoint.
  2. Load Balancing:
    HAProxy distributes incoming files among SCP instances.
  3. Initial Storage & Tag Extraction:
    Each SCP saves files locally and forwards metadata extraction tasks into the Redis stream.
  4. Distributed Processing:
    Worker nodes asynchronously extract tags, store metadata in PostgreSQL, and handle further actions (e.g., AI inference, backup, cloud upload).
  5. Routing & Forwarding:
    Based on admin-configured policies, files can be routed to SCU nodes for delivery to external systems, cloud storage, or backups.
  6. Observability & Notifications:
    All node statuses, processing logs, and metrics are streamed to Prometheus/Grafana/Loki and made available in real time via the Web UI.

Real-World Use Case​

Designed for hospitals and research centers needing robust, automated DICOM file routing and processing, with minimal manual intervention. The system dramatically reduces IT workload, minimizes human error in file routing/backup, and provides an upgrade path for AI/ML-based analytics and compliance reporting.

Extensibility & Roadmap​

  • AI/ML Integration:
    Plug-and-play architecture for deep learning models (diagnostics, segmentation, classification).
  • Cloud Storage:
    Native S3/MinIO support for backup, archiving, and hybrid storage.
  • Advanced Analytics:
    Custom processing workers for anonymization, report generation, or cross-system data aggregation.
  • User Management:
    Future implementation of roles, certificates, and audit logging for compliance.
  • Alerting & Incident Response:
    Integration with external notification systems (Slack, Email, etc.) for critical events.
  • API Expansion:
    Enhanced REST and DICOM protocol (C-FIND, C-GET, etc.) for deeper interoperability.

My Role & Achievements​

  • Designed and implemented system architecture (from Docker Compose to distributed data pipeline and monitoring stack).
  • Built modular FastAPI/React components for maximum reusability and ease of extension.
  • Automated node configuration and scaling via dynamic HAProxy and custom Docker Helper service.
  • Integrated real-time monitoring and notification infrastructure using best-in-class observability tools.
  • Performed load testing and system validation under simulated high-traffic scenarios.
  • Planned roadmap for AI/ML integration and long-term system evolution.

User Interface​

DICOM Platform UI

Centralized admin interface for real-time monitoring, configuration, and operational control of the distributed DICOM routing system.

Note​

Due to NDA and client confidentiality, source code and production configurations cannot be shared.
This document and diagrams are provided for demonstration of my engineering approach, architectural skills, and ability to deliver scalable, production-ready healthtech solutions.