Middle/Senior Data Engineer (Social Shopping Platform)

Requirements

• 3+ years of strong experience with Python as a programming language for data pipelines and related tools
• Proven strong track record of building data platforms and managing infrastructure for Airflow and Databricks
• Familiarity and understanding of distributed data processing with Spark for data pipeline optimization and monitoring workloads
• Proven strong track record of building data transformations using data build tools
• Excellent implementation of data modeling and data warehousing best practices
• Good written and spoken English communication skills
• Familiarity with software engineering best practices: testing, PRs, Git, code reviews, code design, releasing
• Proven strong track record of building data platforms and managing infrastructure for Airflow and Databricks

Would be a plus

• Strong Data Domain background – understanding of how data engineers, data scientists, analytics engineers, and analysts work to be able to work closely with them and understand their needs
• Experience with DAGs and orchestration tools
• Experience in developing event-driven data pipelines

Responsibilities

• Contributing to new technology investigations and complex solution design, supporting a culture of innovation by considering matters of security, scalability, and reliability, with a focus on building out our ETL processes
• Working with a modern data stack, coming up with well-designed technical solutions and robust code, and implementing data governance processes
• Working and professionally communicating with the customer’s team
• Taking responsibility for delivering major solution features
• Participating in the requirements gathering and clarification process, proposing optimal architecture strategies, and leading the data architecture implementation
• Developing core modules and functions, designing scalable and cost-effective solutions
• Performing code reviews, writing unit and integration tests
• Scaling the distributed system and infrastructure to the next level
• Building data platform using power of AWS cloud provider