freelance
Big Data Engineer (Python, Sql, Apache Spark) Reference number: 1385
Last update: 25-03-2020, 14:33
Region: Brussels
Sector: IT
ASAP
Work experience: 5 years
Job description
For our client based in Brussels we are looking for a candidates with in depth programming skills in Python and eager to learn within this new environment.
We need someone who is flexible and can operate in a frequently changing environment that is still under construction. Short time to market, short interval releases, frequent demo’s at the end of each sprint require these individuals to have a strong sense of urgency working towards a future proof solution. They are able to capture feedback on a frequent basis and incorporate this into the design of technical solutions. Specific knowledge of Big Data tools and technologies like Hadoop, Spark and OO development is required. They understand how to apply these tools and technologies to solve big data problems and to develop innovative big data solutions.

Goal of the project:

The deliverable of this project is a model lifecycle management tool. Data scientists need a tool that supports the creation, management and monitoring of datascientist models. The tool we will deliver thus contains a model factory that allows swift model creation. Besides that the tool will also support model management and model monitoring.
What you’ll do
  • Analysing, designing, developing, constructing and testing new and existing projects in Python
  • Developing and designing a framework with the focus on efficiency (starting from a prototype)
  • Incorporating enterprise standards into new software projects (CI/CD pipelines, unit and integration testing and versioning)
Profile
  • The Big Data Engineer role ensures that data pipelines are scalable, repeatable, and secure, and can serve multiple users. They help facilitate getting data from a variety of different sources, getting it in the right formats, assuring that it adheres to data quality standards, and assuring that downstream users can get that data quickly. This role usually functions as a core member of an agile team.
  • These professionals are responsible for the infrastructure that provides insight from raw data and handles diverse sources of data seamlessly. They enable big data and batch/real-time analytical solutions that leverage emerging technologies. Additional responsibilities include developing prototypes and proof of concepts for the selected solutions, and implementing complex big data projects with a focus on collecting, parsing, managing, analysing and visualizing large sets of data using multiple platforms. They understand how to apply technologies to solve big data problems and to develop innovative big data solutions.
  • The Big Data Engineer is a technical job that requires substantial expertise in a broad range of software development and programming fields. These professionals have knowledge of data analysis, end user requirements analysis, and business requirements analysis to develop a clear understanding of the business needs and to incorporate these needs into technical solutions.
  • They have a solid understanding of physical database design principles, and the system development life cycle. These individuals must work well in a team environment.
Required Skills
  • Big Data Framework / Hadoop: Cluster, HDFS, Yarn and MapReduce
  • Python
  • Apache Spark
  • Big Data Framework / Hadoop : Hive
  • SQL
  • Development methodologies :
    - Agile : Safe
  • Extra technical experience that can help :
    - Data science & tools: H20, scikit-learn, Jupyter notebooks
    - CI/CD tools: pybuilder, Anaconda, Jenkins, Nexus
Interested? Send us your resumé
To apply for this job, please complete the form below and join your resume. This instantly places your information into our database. Once we have received your information, we will be in touch by e-mail or phone. If you have not heard from us after 3 working days, please call us!

Thank you for your interest in working with Harvey Nash and we look forward to assisting you in your job search!

Only PDF, max. 10MB

Only PDF, max. 10MB