Sr. Spark/AWS Engineer

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by Steneral Consulting, so some details may not match this job. The employer's own page has the real description.

Job Overview

The Senior Spark/AWS Engineer at Steneral Consulting will play a critical role in designing, developing, and optimizing big data solutions utilizing Apache Spark in conjunction with Amazon Web Services (AWS). This position requires an adept understanding of cloud computing principles and extensive hands-on experience in data processing, analysis, and pipeline architecture. The successful candidate will work collaboratively with cross-functional teams to deliver high-quality data-driven solutions that meet business needs and enhance overall data analytics capabilities.

Job Responsibilities

• Design and develop scalable Spark applications on AWS to process and analyze large datasets.
• Optimize existing Spark applications for performance improvements and cost efficiency.
• Implement ETL (Extract, Transform, Load) processes to facilitate comprehensive data integration.
• Collaborate with data scientists and analysts to understand their analytical needs and provide tailored solutions.
• Ensure data quality and integrity by developing robust data validation frameworks.
• Monitor and troubleshoot production data pipelines to resolve any issues promptly.
• Stay up to date with new Spark and AWS features and best practices to continually improve existing systems.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• A minimum of 5 years’ experience in software engineering, with a focus on big data technologies.
• Extensive experience with Apache Spark, including Spark SQL and DataFrames API.
• Proficiency in AWS services, specifically S3, EMR, Lambda, and Redshift.
• Strong programming skills in languages such as Java, Scala, or Python.
• Familiarity with data modeling concepts and relational databases.
• Experience with version control systems such as Git.
• Excellent problem-solving skills and ability to work in a fast-paced environment.

Technologies & Tools

The Senior Spark/AWS Engineer will work with a variety of technologies and tools, including but not limited to Apache Spark for big data processing, AWS services such as S3 for storage, EMR for processing large datasets, and Lambda for serverless computing solutions. Familiarity with data warehousing solutions like Redshift and integration tools such as Apache Kafka or Apache NiFi will be advantageous. Proficiency in version control systems like Git and experience with containerization tools such as Docker may also be required.

Ideal Candidates

Ideal candidates for the Senior Spark/AWS Engineer position will exhibit strong analytical and technical skills coupled with excellent communication abilities. They should demonstrate a proactive approach to problem-solving and possess a keen attention to detail. Candidates who thrive in collaborative settings, welcome feedback, and adapt easily to changing technologies and project requirements will find themselves well-suited for this role. A strong passion for data engineering and a continuous desire for learning and growth within the big data ecosystem is essential.