Prodigy Education - Senior Data Engineer

Improving the reliability of Prodigy’s data pipelines and data lakehouse.

• Currently embedded in a product team working to expose modelled data to users
• Worked across teams to build and scale an event pipeline to ingest thousands of messages per second.
• Developed processes for deploying ml pipelines using MLflow, implemented PII redaction, and added automation tools
• Mentoring, training and on-boarding new teammates.
• Expanding my understanding of data systems, reliability, data warehousing/data lakes

Technologies: DBT, SQL, Python, Spark, AWS, Databricks, Docker, Airflow, Terraform, Datadog, Kubernetes

Skills Developed :
Data pipeline scheduling and automation
DBT model development and optimization
Automating backfill processes
Updating and testing packages
Developing production ML pipeline processes