ETLPIPELINE
Messy data in. Meaningful data out.
Choose a numbered part or a stage below to explore the system.
EXTRACT / TRANSFORM / LOADRobust multi-threaded engine processing 12,000+ transactions with real-time CDC and dynamic currency conversion.
- Python
- PostgreSQL
- Kafka
- HDFS
- Spark
- AirFlow
- Azure Cloud
Build notes
Multi-threaded processing engine handling over 12,000 transactions.
Real-time Change Data Capture (CDC) integration.
Dynamic currency conversion and audit logging system.
- Capacity
- 12K+
- Process
- MultiThreading
- Cloud
- Airflow