Senior Data Engineer

Requirements

• 5+ years of experience in data engineering
• 3+ years of production experience with Databricks and Delta Lake
• Strong expertise in medallion architecture and event-driven asynchronous pipeline design
• Advanced proficiency in Python, PySpark, and SQL
• Deep hands-on experience with Databricks, including Unity Catalog, Lakeflow Jobs/Workflows, and job optimization
• Strong AWS expertise including S3, IAM, KMS, and services such as AWS Batch, ECR, EventBridge, Lambda, or API Gateway
• Experience with infrastructure as code and CI/CD using Terraform and Databricks Asset Bundles
• Practical experience with Git workflows, pull request reviews, and automated testing
• Experience configuring secure CI pipelines using OIDC or service principals
• Practical MLflow experience including experiment tracking, model registration, and reproducibility practices
• Strong understanding of data quality, lineage, provenance, schema contracts, and versioning
• Experience with data quality frameworks such as Great Expectations, Soda, pyDeequ, or Databricks Labs DQX
• Experience with monitoring, observability, and cloud cost optimization
• Upper-Intermediate English level or higher

WILL BE A PLUS

• Experience packaging workloads in Docker containers and running batch jobs on AWS
• Familiarity with experiment optimization tools such as scikit-optimize, BoTorch, Ax, and scipy
• Experience building feature sets and train/test splits that avoid data leakage
• Experience with regulated or high-security environments including data residency requirements
• Experience disabling or restricting AI features in secured environments

Personal Profile

• Strong analytical mindset
• Ownership and accountability
• Ability to communicate complex technical concepts clearly
• Proactive approach to problem-solving
• Attention to scalability and maintainability
• Collaborative and team-oriented attitude

Responsibilities

• Design and build scalable cloud-native data platforms from greenfield to production
• Design and implement near-real-time ingestion pipelines using event-driven patterns
• Define and enforce data platform standards including Data Lake/Lakehouse principles, medallion architecture, and data contracts
• Refactor, optimize, and modernize Spark and PySpark scripts for performance and maintainability
• Introduce best practices for code quality, testing, and CI/CD across data pipelines
• Drive adoption of AI tooling and agentic workflows within the data engineering team
• Ensure data quality, observability, and reliability across pipelines and platforms
• Design and deliver self-service tooling and microservices that abstract platform complexity
• Collaborate directly with the Customer’s scientific and engineering teams
• Monitor platform performance, reliability, and cloud infrastructure costs