FloatChat – Oceanographic Data Engineering Pipeline
Designed and implemented a scalable data engineering pipeline for processing ARGO oceanographic datasets stored in NetCDF format. Built a multi-stage ETL workflow that automatically downloads, validates, transforms, and stores float data as optimized Parquet datasets while generating metadata for PostgreSQL and ChromaDB indexing. Implemented parallel processing and hash-based incremental updates to eliminate redundant computation and handle inconsistencies across large scientific datasets. The system enables efficient querying and downstream analytics on previously difficult-to-use oceanographic data.
Impact: Automated ingestion and transformation of large-scale scientific datasets, significantly reducing manual preprocessing effort while improving processing throughput through concurrent execution and incremental updates.
- Python
- PostgreSQL
- ChromaDB
- NetCDF
- Pandas
- Joblib
- ThreadPoolExecutor