Big Data engineer develop, maintain, test and evaluate big data solutions within organisations. Most of the time he / she is also involved in the design of big data solutions, because of the experience they have with Hadoop based technologies such as MapReduce, Hive MongoDB or Cassandra.
A Big Data engineer builds large-scale data processing systems, is an expert in data warehousing solutions and should be able to work with the latest (NoSQL) database technologies.
A Big Data engineer should have sufficient experience in software engineering before the move can be made to the field of big data.
Experience with UNIX, object-oriented design, coding and testing patterns as well as experience in engineering (commercial or open source) software platforms and large-scale data infrastructures should be present.
Big Data engineer should also have the capability to architect highly scalable distributed systems, using different open source tools.
He or she should understand how algorithms work and have experience building high-performance algorithms.
The Capgemini Big Data engineer should embrace the challenge of dealing with petabyte or even exabytes of data on a daily basis.
He / She understands how to apply technologies to solve big data problems and to develop innovative big data solutions. In order to be able to do this, the Big Data engineer should have extensive knowledge in different programming or scripting languages like Java, Linux, C++, PHP, Ruby, Phyton and / or R.
Also expert knowledge should be present regarding different (NoSQL or RDBMS) databases such as MongoDB or Redis. Building data processing systems with Hadoop and Hive using Java or Python should be common knowledge to the big data technical consultant.
A Big Data engineer generally works on implementing complex big data projects with a focus on collecting, parsing, managing, analysing and visualizing large sets of data to turn information into insights using multiple platforms.
He or she should be able to decide on the needed hardware and software design needs and act according to the decisions. The Big Data engineer should be able to develop prototypes and proof of concepts for the selected solutions.
Additional qualifications should include :
Desired Skills & Experience