Sr. Spark/AWS Engineer

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by Steneral Consulting, so some details may not match this job. The employer's own page has the real description.

Job Overview

The Senior Spark/AWS Engineer at Steneral Consulting will play a pivotal role in the design, implementation, and management of data processing frameworks using Apache Spark on the AWS platform. This position requires a skilled engineer who can leverage big data technologies to build robust data pipelines and ensure high performance in data parsing, transformation, and analytics. The ideal candidate will collaborate closely with cross-functional teams, contributing to the enhancement of data-driven decision-making processes across the organization.

Job Responsibilities

• Design and develop scalable data processing solutions using Apache Spark within the AWS ecosystem.
• Build and optimize data ingestion pipelines that interface with various data sources and formats.
• Collaborate with data engineers and data scientists to define and implement data models and ETL processes.
• Ensure high availability and performance of data processing workflows, conducting regular health checks and monitoring systems.
• Implement best practices for data security, integrity, and compliance adhering to organizational policies.
• Participate in technical documentation and provide training to team members on Spark and AWS tools.
• Stay abreast of emerging technologies in big data and suggest improvements to processes and workflows.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, or a related field; Master’s degree preferred.
• A minimum of 5 years of professional experience in software engineering, with a strong focus on big data technologies.
• Proficiency in Apache Spark and experience with its ecosystem components (e.g., Spark SQL, Spark Streaming).
• Strong understanding of AWS services, including but not limited to S3, EMR, Lambda, and Redshift.
• Experience with programming languages such as Java, Scala, or Python.
• Knowledge of data warehousing concepts and technologies, such as Redshift or Snowflake.
• Familiarity with containerization and orchestration technologies like Docker and Kubernetes is a plus.

Technologies & Tools

The Senior Spark/AWS Engineer will be primarily working with technologies such as Apache Spark, AWS (Amazon Web Services) components including S3, EMR, and Lambda, as well as databases like Redshift and DynamoDB. Additional tools may include data processing frameworks, ETL tools, and development environments that support the implementation and monitoring of data workflows.

Ideal Candidates

The ideal candidate for this position is a proactive problem-solver with a strong analytical mindset and the ability to handle complex data scenarios. They should possess excellent communication skills, working collaboratively within diverse teams, to effectively convey technical concepts to non-technical stakeholders. A passion for continuous learning and adaptability to new technologies will also be essential in thriving within the fast-paced innovative environment at Steneral Consulting.