Job Description
We are seeking a Data Engineer (Databricks) to join PERSOL in Singapore. This role focuses on building and operating large-scale AWS data platforms using Databricks and Informatica to transform raw data into reliable, analytics-ready pipelines.
As a Data Engineer, you will design, build, and maintain scalable data pipelines that empower data scientists, analysts, and business users to derive timely insights. You will collaborate with data architects and engineers to implement robust data models, ETL/ELT processes, and automated data quality checks across cloud environments.
You will monitor pipeline performance, optimize data processing for speed and cost efficiency, enforce data governance and security controls, and champion best practices for CI/CD in a cloud data platform. The ideal candidate thrives in a fast-paced, cross-functional team and has hands-on experience with Databricks, AWS, and Informatica.
This is a fantastic opportunity to grow your career in data engineering while delivering tangible business impact through scalable, reliable data products. Join us to accelerate data-driven decision making across the organisation.
Responsibilities
- Design, develop, and maintain scalable data pipelines on AWS using Databricks and Informatica.
- Collaborate with data scientists, analysts, and engineers to translate business requirements into reliable data models and ETL/ELT processes.
- Optimize data ingestion, processing, and storage to ensure high performance and cost efficiency.
- Implement data quality checks, data lineage, and governance controls to ensure accuracy and compliance.
- Monitor pipeline health, troubleshoot issues, and implement automated testing and CI/CD for data assets.
- Contribute to data architecture decisions, including lakehouse design, schema evolution, and data modeling.
- Provide mentoring and best practices for SQL, Python/Scala, and Spark transformations.
- Communicate progress and results to stakeholders and collaborate with cross-functional teams in an agile environment.
Qualifications
- Bachelor's degree in Computer Science, Engineering, or a related field; or equivalent practical experience.
- Hands-on experience with Databricks on AWS and building scalable data pipelines.
- Proficiency in SQL and one or more programming languages (Python, Scala).
- Experience with ETL/ELT tools, preferably Informatica, and data integration patterns.
- Strong understanding of data warehousing, data modeling, and data governance concepts.
- Familiarity with cloud data security, access controls, and compliance requirements.
- Strong problem-solving, communication, and collaboration skills; ability to work in an Agile environment.
- Self-motivated with a passion for delivering high-quality data products and continuous learning.