When you sign into LinkedIn and search for jobs as a data scientist, a jumbled list pops up: “Data Scientist”, “Data Scientist”, “Data Engineer”, “Senior Data Scientist”, “Data Engineer”, “Data Engineer”. On and on, the list continues. You are overcome with confusion; the job descriptions don’t offer a lot of assistance in telling these roles apart. They can’t be that different, can they? They both have data in the name – surely it doesn’t matter which position you apply to?
Well, yes and no. If just working with data is your goal in life, then either role will satisfy you. However, when it comes to what you will be doing with the data, these two positions are worlds apart. Data Scientists and Data Engineers will work together in almost all industries. In some smaller companies, where only a limited number of Data Engineers and Data Scientists can be hired, some of their tasks can overlap. This makes the differentiation between these two roles even more confusing.
In this post, we’ll be taking a look at the differences between data engineers and data scientists by discussing their different goals, mindsets, tools and backgrounds. Before we dive into the detail, however, it is important to understand the hierarchy of the Data Process.
Understanding the hierarchy of the Data Process
Companies that design products or services need valuable data. This data can consist of any facts, figures or statistics that provide information needed to understand their market, competitors, products, customer needs, and more.
Due to modern technologies, the world now has access to Big Data, which provides industries with access to a greater volume, velocity, and a variety of data. This means that the world can make significantly better decisions, much faster.
But Big Data requires procedures, organization, technologies, and, most importantly, people who can handle it. Depending on your goals, Data Engineers and Data Scientists will be essential for handling specific aspects of the process.
Figure 1: The Data Science Hierarchy of Needs -by Monica Rogati
The “Data Science Hierarchy of Needs” pyramid is an excellent representation of the processes required to handle Big Data in industry. From this hierarchy, it becomes easy to differentiate between the roles and responsibilities of Data Scientists and Data Engineers when it comes to handling data.
Data Engineers
Data Engineers have three main goals: to design, build and arrange data “pipelines” that support analytical dashboards and other data customers.
Data pipelines are collections of processing and analysis procedures that are applied to data for a specific purpose. They are useful in production projects, and can also be handy if one anticipates facing similar business challenges in the future. In Figure 2 the data pipeline process is illustrated as extracting data from the source, transforming it, and then loading it into the data warehouse. From there, the data can be accessed and reported on.
Data Engineers will use tools or programming languages such as SQL, Java, Scala, C++ or Python depending on their task.
These tasks include:
- Designing the big data infrastructure and preparing it to be analyzed.
- Building complex queries to create pipelines.
- Arranging any problems in the programmed system.
Data Engineers require an extensive knowledge and understanding of the received data as well as some programming knowledge to extract and transform the data into usable formats for Data Scientists. They usually have degrees in computer science or software engineering, and possess system creation skills.
Figure 2: Data Pipeline Process
Data Scientists
Data Scientists have four main goals: to analyse, test, create and present data models.
Data Scientists do extensive research to answer questions that have been posed. To answer these questions, which can either lead to conclusions or more questions, they have to deeply understand the data and analyze it to extract accurate information.
Data Scientists experiment with the data and perform statistical hypothesis tests and algorithms to determine how to accurately model the data. This can include advanced machine learning and artificial intelligence models.
Like Data Engineers, Data Scientists often use Python, but the way they use it is different. Where Data Engineers use Python to manage data pipelines, Data Scientists use Python and packages like Pandas, Scikit Learn and Tensor Flow to analyse data and build models. In addition, Data Scientists use advanced analytical tools such as R, SPSS, and Hadoop.
The tasks of Data Scientists include:
- Working on clean data
- Finding solutions with the data available
- Communicating analyses with the team
Figure 3: Data Science Structure
As a Data Scientist is more like a researcher, having a research-based background is beneficial. They require extensive knowledge in programming, statistical modelling, and analysis, as well as mathematics. Most Data Scientists have degrees in computer engineering, statistics, or mathematics, and those with Master’s degrees tend to have the upper hand in their field.
Conclusion
Data Science and Data Engineering are two completely different disciplines which manage different problem domains and require specific skills and approaches to deal with everyday problems. Data Engineering does not involve machine learning or statistical modeling, but Engineers must transform data so that Data Scientists can develop the machine learning models. Although Data Scientists can develop a core algorithm for analysing and visualising the data, they rely entirely on Data Engineers for their processed and enriched data requirements. Both areas offer numerous opportunities and work possibilities due to the increase in the amount of data available, and the advent of IoT and Big Data technologies.
Deidre is a Senior Technical Process Engineer at Convergenc3, a future-focused consulting company that helps clients across multiple industries to navigate the ever-innovating landscape of business and technology.
All opinions expressed are the author’s own.
