Lead Data Engineer

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by New York Technology Partners, so some details may not match this job. The employer's own page has the real description.

Overview

The Lead Data Engineer at New York Technology Partners will play a crucial role in establishing and maintaining the architecture and infrastructure needed to ensure the effective collection, storage, and processing of data. This position requires an individual who is not only technically proficient but also capable of leading a team. The ideal candidate will oversee the development and implementation of data solutions that meet organizational needs while promoting best practices in data governance and management.

Job Responsibilities

• Design, develop, and maintain robust data pipelines and architectures for optimal data collection and processing.
• Lead efforts in data strategy, architecture decisions, and tool selections aligned with business goals.
• Collaborate with data scientists, analysts, and other stakeholders to identify data requirements and translate them into technical specifications.
• Monitor and optimize data storage costs, performance, and scalability.
• Ensure data quality and integrity through rigorous testing and validation processes.
• Provide mentorship and guidance to junior data engineers to foster professional growth and team cohesion.
• Stay informed on the latest industry trends, technologies, and best practices in data engineering.
• Document architecture, data flow, and data transformation processes in detail.

Qualifications

• Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field.
• Five or more years of experience in data engineering or a related role.
• Proficiency in SQL and at least one programming language such as Python, Scala, or Java.
• Strong experience with data warehousing solutions and ETL tools.
• Familiarity with cloud platforms such as AWS, Azure, or Google Cloud Platform.
• Solid understanding of data governance principles and data modeling.
• Excellent analytical, problem-solving, and organizational skills.
• Demonstrated ability to lead cross-functional teams effectively.

Technologies & Tools

The Lead Data Engineer will engage with a variety of technologies and tools, including relational and NoSQL databases (such as PostgreSQL, MongoDB), data warehousing solutions (e.g., Snowflake, Redshift), and ETL pipelines (using tools like Apache Airflow and Talend). Familiarity with big data technologies, including Apache Spark and Hadoop, and cloud services such as AWS S3 and Azure Data Lake, will also be essential.

Ideal Candidates

Ideal candidates for the Lead Data Engineer position will possess not only strong technical skills but also exceptional leadership abilities. They will be characterized by a strategic mindset, an innovative approach to problem-solving, and a strong commitment to collaboration. Successful candidates will demonstrate adaptability in a fast-paced environment and a passion for leveraging data to drive business success.