Senior Data Engineer

About the position

• Data Engineers are responsible for building and maintaining Big Data Pipelines using Data Platforms.


• Data Engineers are custodians of data and must ensure that data is shared in line with the information classification requirements on a need-to-know basis.


• Data Engineers are also expected to: Stay up to date with the latest data engineering tools, technologies and industry trends. Identify opportunities for process improvements and automation to enhance the efficiency and reliability of data pipelines.


• Explore and evaluate new data engineering approaches and technologies to drive innovation within the organisation.


• Mentor, train and upskill members in the team.

Minimum Requirements:

Education

• Relevant IT / Business / Engineering Degree


• Candidates with one or more of the certifications are preferred: AWS Certified Cloud Practitioner, AWS Certified SysOps Associate, AWS Certified Developer Associate, AWS Certified Architect Associate, AWS Certified Architect Professional, HashiCorp Certified Terraform Associate


• Experience in working with Enterprise Collaboration tools such as Confluence, JIRA, etc.  


• Experience developing technical documentation and artefacts.


• Knowledge of data formats such as Parquet, AVRO, JSON, XML, CSV, etc.  


• Experience working with Data Quality Tools such as Great Expectations.


• Experience developing and working with REST API's is a bonus.


• Basic experience in Networking and troubleshooting network issues.  


• Knowledge of the Agile Working Model.

 



Techstack:



Cloud & Infrastructure

• Terraform, AWS CloudFormation


• AWS services: S3, Glue, Athena, CloudWatch, SNS, Lambda, Step Functions, DynamoDB


• AWS Parameter Store, Secrets Manager


• CodeBuild, Code Pipeline

Programming & Scripting

• Python 3.x, PySpark, Boto3


• SQL (Oracle, PostgreSQL)


• PowerShell, Bash


• Linux / Unix environments

Data Engineering & Big Data

• ETL pipeline development


• Big Data processing and analytics


• Kafka, Kinesis Streams, Kinesis Firehose


• AWS EMR, Amazon Redshift

DevOps & Containerisation

• Docker

Data Modelling & BI

• Technical data modelling and schema design (engineered, non-drag-and-drop)


• Business Intelligence (BI) experience

Enterprise Data Platforms

• Cloud Data Hub (CDH)


• CDEC Blueprint

Desired Skills:

• Data
• Data Engineering
• AWS