Data Engineer

About the Role

Our client is seeking a Data Engineer to design, develop, and maintain scalable data pipelines and applications supporting complex data processing and analysis. This role involves working with large datasets, implementing ETL processes, maintaining data quality, and developing data solutions using technologies such as Palantir Foundry, Databricks, Python, Spark, Scala, and SQL.

The ideal candidate has experience supporting Department of War (DoW) missions and associated datasets, along with an active Secret clearance and hands-on data engineering experience.

Key Responsibilities

• Analyze science, engineering, business, and other data processing challenges to develop solutions for complex applications, system administration, and network-related concerns.
• Perform systems management and integration activities, improve existing computer systems, and evaluate system capabilities, workflows, and scheduling limitations.
• Evaluate and recommend commercially available software when appropriate.
• Implement and enforce data governance policies and procedures in accordance with regulatory requirements and industry best practices.
• Troubleshoot data-related issues and develop solutions to complex data challenges.
• Collaborate with data scientists, analysts, and stakeholders to provide accurate and timely data insights for decision-making.
• Use big data technologies to process and analyze large datasets efficiently.
• Perform data cleansing, integration, and quality assurance to maintain data accuracy and integrity.
• Build and optimize scalable, reliable, high-performance data pipelines.
• Work with cross-functional teams to understand data requirements and develop appropriate data models and architectures.
• Design, develop, and implement end-to-end data pipelines using ETL processes and technologies such as Databricks, Python, Spark, Scala, JavaScript/JSON, and SQL.

Required Qualifications

• 3–10 years of directly related experience.
• Bachelor’s or master’s degree in a related field.
• 3+ years of experience as a Data Engineer.
• Active Secret security clearance.
• Experience building applications in Palantir Foundry, including Workshop, Ontology Manager, and Gaia.
• Experience with machine learning models and/or Generative AI.
• Experience working with APIs.
• Previous experience with Advana or the War Data Platform.
• Experience supporting Department of War missions and working with associated datasets.
• Understanding of data modeling, visualization, database design principles, and data governance practices.
• Experience processing and analyzing large datasets using big data technologies.
• Proficiency in ETL processes and hands-on experience with Databricks, Python, Spark, Scala, JavaScript/JSON, SQL, and Jupyter Notebooks.

Preferred Qualifications

• The source lists the following as preferred qualifications as well; these overlap with several of the stated required qualifications:
• Experience with machine learning models and/or Generative AI.
• Experience working with APIs.
• Previous experience with Advana or the War Data Platform.
• Experience supporting Department of War missions and associated datasets.
• Knowledge of data modeling, visualization, database design, and data governance.
• Experience processing large datasets using big data technologies.
• Proficiency in ETL processes and technologies such as Databricks, Python, Spark, Scala, JavaScript/JSON, and SQL.

Work Arrangement & Security Clearance

• Remote position.
• An active Secret clearance is required.