AWS Cloud Data Engineer with Pyspark, Python

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by Infinite Computer Solutions, so some details may not match this job. The employer's own page has the real description.

Overview

As an AWS Cloud Data Engineer at Infinite Computer Solutions, you will be responsible for designing, developing, and maintaining scalable data pipelines to support data-driven decision-making across the organization. Your expertise in PySpark and Python will enable you to efficiently process and analyze large data sets on the AWS platform. You will collaborate closely with data scientists, analysts, and other stakeholders to build robust data solutions that enhance business intelligence and operational efficiency.

Job Responsibilities

• Design and implement data pipelines using AWS services and frameworks such as Lambda, Glue, and S3.
• Develop ETL processes to extract, transform, and load data from various sources into AWS environments.
• Utilize PySpark for distributed data processing and large-scale data transformations.
• Collaborate with cross-functional teams to understand data needs and provide technical guidance on best practices.
• Monitor and optimize data processing workflows to ensure efficiency and reliability.
• Perform data quality checks and manage metadata to ensure accuracy and consistency within data sets.
• Implement security and compliance measures for data handling in the AWS environment.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or a related field.
• 3+ years of experience in data engineering or a similar role with a focus on cloud technologies.
• Proficiency in Python and experience with the PySpark library for data analysis.
• Strong understanding of AWS cloud services, particularly Data Pipeline, Glue, and Redshift.
• Familiarity with data warehousing concepts and experience with SQL and NoSQL databases.
• Experience in data modeling and data architecture best practices.
• Knowledge of version control systems, preferably Git.

Technologies & Tools

This position will utilize various technologies and tools, including but not limited to AWS services (such as EC2, S3, Lambda, Glue, Redshift), PySpark, Python, SQL, data visualization tools, and version control systems like Git. Familiarity with container orchestration platforms, such as Docker, may also be beneficial.

Ideal Candidates

Ideal candidates for this position should possess a strong analytical mindset and be detail-oriented, demonstrating a passion for data analytics and cloud technologies. Effective communication skills and the ability to work collaboratively in a team-oriented environment are essential. The candidate should be proactive in identifying solutions and embracing learning opportunities to keep pace with evolving technologies within the data engineering landscape.