1. Data Pipeline Development & Management
· Design, build, and maintain robust data pipelines usingAWS Glue
· Implement ETL/ELT processes for data ingestion frommultiple sources
· Optimize data workflows for performance and scalability
· Monitor and troubleshoot data pipeline failures andperformance issues
2. Data Infrastructure & Engineering (with IPOS ITDepartment)
· Manage and optimize AWS Redshift data warehouse operations
· Configure and maintain data storage solutions (AWS S3,data lakes)
· Implement data partitioning, indexing, and compressionstrategies
· Support Infrastructure as Code (IaC) for datainfrastructure deployment
3. CI/CD & DevOps for Data (with AWS partners and IPOS ITDepartment)
· Develop and maintain CI/CD pipelines for data workflowsusing GitLab
· Implement automated testing for data pipelines and dataquality
· Support version control and deployment strategies for dataassets
· Configure AWS Lambda functions for data processingautomation
4. Monitoring & Support (with IPOS IT Department)
· Set up monitoring and alerting for data pipeline health
· Provide technical support for data-related issues
· Collaborate with technical teams on data architecturerequirements
· Optimize query performance and database operations
5. Documentation & Reporting (with IPOS IT Department)
· Document data pipeline architectures and technicalspecifications
· Maintain runbooks and operational procedures
· Conduct monthly progress meetings (1 hour) to report onsystem health
· Track engineering tasks through SHIP-HATS Jira
· Maintain technical documentation on SHIP-HATS Confluence
RequiredSkills & Experience:
· Strong background in data engineering and data pipelinedevelopment
· Proficiency in SQL, Python, and shell scripting
· Extensive experience with AWS data services (Redshift, S3,Glue, Lambda, CloudWatch)
· Data warehouse design and optimization experience
· Strong CI/CD pipeline knowledge (GitLab preferred)
· Infrastructure as Code (IaC) experience (Terraform,CloudFormation)
· Knowledge of data modeling and database design principles
· Strong troubleshooting and performance optimization skills
Thankyou!