Middle/Senior Big Data Engineer (Visual Platform)

  • Remote (Ukraine only)
  • Big Data

We are searching for a Big Data professional who knows how to organize raw data masterly and to build an accessible and reliable platform for this purpose.

You are likely to love this role if you are:

  • full of ideas, inspired to create robust and beautiful solutions
  • excited about dynamic projects with challenging tasks
  • initiative at choosing technologies and approaches in making a solution from the start

Customer

Our client is a UK-based web development company that has an ambitious goal to make Internet more human-friendly by improving visual representation of data within the listed structures in the Web. By building a data visualization platform that will process big volumes of user interaction data, the company improves user experience and usability of existing client sites.

Project

Our development team works on an efficient visual platform that dramatically improves browsing the web on any device without any plugins or responsive design. The project is in the active development phase, and the main technology stack is based on the WebGL/threejs technology on the frontend and Ruby on the backend.

As a Big Data Engineer, you will be a part of the data engineering team that will design and implement the Data Platform from scratch. Data Platform’s goal is to provide comprehensive analytics based on user interaction data from a client’s web sites like user heatmap click events, user navigation events, conversion events and other system events to improve visualization experience of the web sites.

The customer is extremely open to the professional opinion of the team, so you will have a chance to initiate changes, choose technologies and approaches.

Meet your team!
  • Responsibilities

    • Design and build Data Lake repository to store raw data ingested from different sources
    • Design Data Warehouse structures and relations using Snowflake
    • Implement ingestion pipelines to collect data from different sources using cloud-based solutions
    • Create data processing flows using Spark and cloud services to transform and store raw information from data sources
    • Apply Data Governance practices to meet regulatory requirements for data security and data quality
  • Requirements

    • Solid development experience with Java, Scala, or Python
    • Solid experience with Spark
    • Experience with designing or developing Data Lakes and Data Warehouses
    • Experience with NoSQL and RDBMS technologies
    • Experience with at least one of cloud providers like AWS, Google Cloud, or Azure

    Would be a plus:

    • Experience with designing or developing Data Lakes and Data Warehouses on AWS
    • Designing and implementing CI/CD practices
    • Experience with key-value databases
    • Experience with Grafana
    • Experience with AWS Big Data services like EMR, Glue
    • Experience with serverless solutions like AWS Lambda