Job Description

Job Description:

As a Data Engineer with expertise in PySpark, Databricks, and Microsoft Azure, you will be responsible for designing, developing, and maintaining robust and scalable data pipelines and processing systems. You will work closely with data scientists, analysts, and other stakeholders to ensure our data solutions are efficient, reliable, and scalable.

Responsibilities:

Design, develop, and optimize ETL pipelines using PySpark and Databricks to process large-scale data on the Azure cloud platform.

• Implement data ingestion processes from various data sources into Azure Data Lake and Azure SQL Data Warehouse

• Develop and maintain data models, data schemas, and data transformation logic tailored for Azure.

• Collaborate with data scientists and analysts to understand data requirements and deliver highquality datasets.

• Ensure data quality and integrity through robust testing, validation, and monitoring procedures.

• Optimize and tune PySpark jobs for performance and scalability within the Azure and Databricks environments

• Implement data governance and security best practices in Azure.

• Monitor and troubleshoot data pipelines to ensure timely and reliable data delivery.

• Document data engineering processes, workflows, and best practices specific to Azure and Databricks.

Works in the area of Software Engineering, which encompasses the development, maintenance and optimization of software solutions/applications.
1. Applies scientific methods to analyse and solve software engineering problems.
2. He/she is responsible for the development and application of software engineering practice and knowledge, in research, design, development and maintenance.
3. His/her work requires the exercise of original thought and judgement and the ability to supervise the technical and administrative work of other software engineers.
4. The software engineer builds skills and expertise of his/her software engineering discipline to reach standard software engineer skills expectations for the applicable role, as defined in Professional Communities.
5. The software engineer collaborates and acts as team player with other software engineers and stakeholders.

Job Description - Grade Specific

Is highly respected, experienced and trusted. Masters all phases of the software development lifecycle and applies innovation and industrialization. Shows a clear dedication and commitment to business objectives and responsibilities and to the group as a whole. Operates with no supervision in highly complex environments and takes responsibility for a substantial aspect of Capgemini’s activity. Is able to manage difficult and complex situations calmly and professionally. Considers ‘the bigger picture’ when making decisions and demonstrates a clear understanding of commercial and negotiating principles in less-easy situations. Focuses on developing long term partnerships with clients. Demonstrates leadership that balances business, technical and people objectives. Plays a significant part in the recruitment and development of people.

Skills (competencies)

Verbal Communication