Sr Data Engineer – Data Ingestion Framework

Experience: 10–12 Years

Work Arrangement: Hybrid (preferred local to Boston)

Engagement: Data Platform Modernization / Data Ingestion Framework

We are looking for an experienced Lead Data Engineer with strong expertise in Python and dbt to design and develop a scalable, reusable, and technology-agnostic data ingestion framework.

The role involves replacing existing SnapLogic ETL capabilities with an open-source technology stack leveraging Python, dbt, and orchestration technologies such as Dagster or Apache Airflow. The solution will utilize AWS S3 as the data lakehouse storage layer and Snowflake as the target data platform.

The ideal candidate should have hands-on experience building enterprise-grade data ingestion frameworks, developing reusable connectors, implementing metadata-driven pipelines, and integrating heterogeneous source systems with modern cloud data platforms.

1. Data Ingestion Framework Development

• Develop a reusable, scalable, and technology-agnostic data ingestion framework using Python.
• Design modular and extensible components to support multiple source systems.
• Develop metadata-driven and configuration-based ingestion capabilities.
• Build reusable connectors and adapters for different source technologies.
• Implement full and incremental data ingestion patterns, including CDC where applicable.
• Implement robust error handling, logging, auditing, retry, and reprocessing mechanisms.

2. Data Integration & Lakehouse Architecture

• Develop ingestion pipelines to integrate data from heterogeneous source systems into AWS S3 and Snowflake.
• Implement data ingestion from SQL Server, Oracle, PostgreSQL, REST APIs, SFTP, AWS S3, and other source systems.
• Design and implement data lakehouse ingestion patterns using AWS S3.
• Work with file formats such as CSV, JSON, Parquet, and other structured/semi-structured formats.
• Implement schema handling, data validation, partitioning, and data movement strategies.

3. Python & dbt Engineering

• Develop production-grade Python applications and reusable ingestion libraries.
• Build and maintain dbt models for data transformation and integration.
• Implement dbt testing, documentation, incremental models, and reusable macros.
• Optimize Python-based ingestion processes and dbt transformations for performance and scalability.
• Follow engineering best practices for modularity, maintainability, and code quality.

4. Workflow Orchestration

• Implement and manage workflow orchestration using Dagster or Apache Airflow.
• Develop reusable workflows for scheduling, dependency management, retries, and pipeline monitoring.
• Integrate orchestration workflows with Python ingestion components and Snowflake.

Required Technical Skills

• Python: Expert-level hands-on experience in Python development, including object-oriented programming, reusable libraries, APIs, error handling, and performance optimization.
• dbt: Strong hands-on expertise in dbt Core or dbt Cloud, including models, macros, tests, incremental processing, and deployment practices.
• Data Engineering: Extensive experience building enterprise-scale ETL/ELT pipelines and reusable data ingestion frameworks.
• Snowflake: Strong experience with Snowflake architecture, data loading, stages, Snowpipe, SQL, and performance optimization.
• AWS: Hands-on experience with AWS S3, IAM, and cloud-based data integration patterns.
• Data Integration: Experience integrating relational databases, REST APIs, SFTP, and cloud-based storage systems.
• SQL: Advanced SQL development and query optimization skills.
• CI/CD: Experience with Git-based version control, automated builds, testing, and deployment pipelines.

Qualifications

• 10–12 years of overall experience in data engineering and data integration.
• Strong architectural and hands-on engineering capabilities.
• Experience working on enterprise data modernization initiatives.
• Excellent communication and stakeholder management skills.
• Ability to work independently in a distributed onshore/offshore delivery model.

Mandatory Skills

1. Expert-level Python development.
2. Strong hands-on dbt experience.
3. Enterprise data ingestion framework development.
4. Snowflake and AWS S3 experience.
5. Strong SQL and data integration experience.

Preferred: Dagster, Apache Airflow, Apache Iceberg, and SnapLogic migration experience.