Responsibility
Data Pipeline Development and Architecture
a) Translate data requirements from business users into technical specifications.
b) Build out data products as part of a data team.
c) Architect and build ingestion pipelines to collect, clean, merge, and harmonise
data from different source systems.
d) Construct, test, and update useful and reusable data models based on data
needs of end users.
e) Design and build secure mechanisms for end users and systems to access data
in the data warehouse.
f) Research, propose, and develop new technologies and processes to improve
agency data infrastructure.
Data Quality, Testing and Validation
a) Implement data quality checks and validation processes to ensure data accuracy
and consistency across all pipeline stages.
b) Design and execute test cases for data pipelines, covering functional, regression,
and exploratory testing to validate data transformations and business logic.
c) Develop and maintain automated test coverage for data pipelines to prevent bugs
and defects from reaching production.
d) Perform data reconciliation and integrity testing to validate end-to-end data flows
across source and target systems.
e) Support User Acceptance Testing (UAT) by coordinating with stakeholders to
validate that data products and pipelines meet business requirements.
Operations, Governance and Continuous Improvement
a) Conduct day-to-day monitoring of databases and ETL systems, including capacity
planning, maintenance, performance tuning, and issue diagnosis.
b) Collaborate with partner agency IT teams on technology stack, infrastructure, and
security alignment.
c) Collaborate with data stewards to establish and enforce data governance policies,
best practices, and procedures.
d) Maintain a data catalogue to document data assets, metadata, and lineage.
e) Implement and enforce data security best practices, including access control,
encryption, and data masking, to safeguard sensitive data.
f) Drive continuous improvement in CI/CD and Site Reliability Engineering (SRE)
practices.
Experience and Skills Needed
Qualifications and Experience
a) Bachelor's Degree, preferably in Computer Science, Software Engineering,
Information Technology, or related disciplines.
b) Deep understanding of system design, data structures and algorithms, data
modelling, data access, and data storage.
c) Experience in designing, building, and maintaining batch and real-time data
pipelines.
d) Experience with orchestration frameworks such as Airflow and Azure Data
Factory.
e) Proficiency in working with Python, Shell Scripts, and SQL.
f) Demonstrated ability in using cloud technologies such as AWS, Azure, and Google Cloud.
g) Experience with Databricks.
Data Quality and Testing
a) Experience designing and executing data quality checks, validation processes,
and test cases for data pipelines.
b) Familiarity with test automation approaches for data pipelines, including
integration with CI/CD pipelines.
c) Experience with defect tracking and test management tools such as JIRA or
equivalent.
DevOps and CI/CD Practices
a) Familiarity with building and using CI/CD pipelines.
b) Familiarity with DevOps tools such as Docker, Git, and Terraform.
c) Experience implementing technical processes to enforce data security, data
quality, and data governance.
Domain Knowledge (Added Advantage)
a) Familiarity with government systems and policies relating to data governance,
data management, data infrastructure, and data security.
b) Experience in Climate and Weather domains will be an advantage.
