Job Description: Lead Data EngineerRole OverviewWe are looking for a Lead Data Engineer to design, build, and scale our data automation and ingestion capabilities. This role will play a critical part in shaping how data is collected, processed, and operationalized across the organization.You will lead the development of scalable, reusable data collection and processing systems using Python-based pipelines, APIs, and workflow-driven automation primarily focussing on data automation, pipeline reliability, and production-grade model execution.This is a hands-on role for an experienced data engineer who enjoys building robust systems, improving standards, and partnering closely with analytics, backend, UI, and data operations teams.Key ResponsibilitiesData Ingestion & AutomationDesign and develop scalable, reusable data ingestion systems using Python, APIs, web scraping, and file-based ingestion.Build and maintain data collection methods for diverse data types and sources, including APIs, Excel, flat files, HTML etc.Ensure ingestion pipelines are automated, fault-tolerant, and self-healing, minimizing manual intervention.Pipeline Integration & ExecutionIntegrate ingestion pipelines with enterprise data storage and processing layers.Leverage Decisions (workflow automation software) and internal workflow services to orchestrate and execute data transformation and manipulation workflows.Partner with internal platform and workflow teams to ensure pipelines run consistently at scale.Data Quality, Governance & StandardsDefine and enforce data quality checks, validation rules, and reconciliation logic across pipelines.Ensure strong data governance and cross-system alignment, working closely with backend, analytics, and UI teams.Establish and promote development standards, including Script and module structure, Logging, error handling, Unit testing and validation frameworksReduce technical risk and future rework through consistent engineering practices.Model Operationalization & Data Ops EnablementWork closely with Data Ops and Analytics teams to help transition Python-based models from: Analyst-led prototypesTo production-ready, scalable, and reliable operational modelsEnsure models can be executed efficiently within workflow-driven environments (e.g., Decisions), with proper dependency management, performance optimization, and monitoring.Enable end-to-end execution of data pipelines and models, supporting growing data volumes and increased execution frequency.Provide Technical leadership, Guiding design decisions, reviewing code and architectureRequired Skills & ExperienceSenior-level data engineer (typically 10+ years), with strong hands-on experience and the ability to lead and shape systems rather than just implement tasks.Strong hands-on experience with Python in production data engineering environments.Proven experience building data ingestion pipelines, ETL/ELT-style processes, and automation frameworks using code (not GUI-based tools).Strong knowledge of data manipulation and scraping libraries, including Pandas, NumPy, Requests, BeautifulSoup and Selenium (or similar browser automation tools)Experience working with APIs, authentication mechanisms, and structured/unstructured data formats.Solid understanding of data quality, validation, and governance practices.Experience designing pipelines that are Scalable, Reliable, Idempotent, Easy to monitor and debugFamiliarity with workflow-based execution and orchestration concepts.Strong ability to collaborate across teamsAbility to translate analytical needs into robust, production-grade data solutions.
