The rise of the data analytics engineer

1. What is a data analytics engineer?

A data analytics engineer is a specific data operative position within the data team - their role and job responsibilities lie at the intersection between analytics and engineering. 

2. What are the core responsibilities of a data analytics engineer?

The analytics engineer role comes with three broad areas of responsibility:

1. Construct data pipelines. Whether it is ETL or ELT, the analytics engineer is responsible for delivering clean data sets to the end-users. The end-users can either be data scientists, data analysts, or other non-technical stakeholders (e.g. for self-service analytics). Constructing data pipelines involves everything from extracting raw data from multiple sources, cleaning data (data transformations), and loading data into a data warehouse or database so it is accessible to end-users.
2. Build data models. The analytic engineer spends a considerable amount of time designing data models to optimize both the data warehousing aspects (e.g. query execution time) and the data consumption aspects (e.g. building such a snowflake model to optimize analytical queries). The process is a cross-function of architectural engineering design and analytic knowledge. 
3. Perform DataOps. From deploying data warehouses to managing the operational aspects of data pipelines, analytic engineers manage the DataOps cycle of data products. This includes monitoring the data quality throughout the system, orchestrating jobs, and validating metrics and other downstream results of data pipelines. 
   

Analytic engineers need to have a strong foundation in software engineering best practices: from being able to use multiple data-native programming languages and solutions (e.g. scripting in Python/SQL, or deploying models in dbt) to understanding the systemic need for version control, automation, and testing when running a DataOps.

So how exactly does an analytic engineer differ from a data engineer or a data analyst?

3. What is the difference between an analytics engineer and a data engineer, data analyst, or data scientist?

Before we understand the differences between the analytics engineer role and the other members of an analytics team, we need to understand the specifics of each role.

Data engineer:

• Takes care of data ingestion and integration for the entire company. Joining different and disparate data sources and real-time data streams together, using scripting or data tools such as Keboola, Stitch, Fivetran, etc. (check how they compare)
• Deploys, manages, and optimizes the data warehouse (Snowflake, BigQuery, Redshift, …).
• Performs DataOps, by orchestrating jobs and maintaining the cloud solution (AWS, Azure, GCP) or on-premise servers.
  

Data analyst:

• Focused on delivering insights through business intelligence and data analysis.
• Builds dashboards and data visualizations with data visualization tools such as Looker, Power BI, Tableau, or another BI tool.
• Spends a considerable amount of time translating between business questions for decision-making and the answers data can provide. 
  

Data scientist:

• Builds machine learning algorithms to predict or automate data-dependent tasks.
• Uses data science to answer hard questions that cannot be simply analyzed.
  

As you can see from the job descriptions, the analytics engineer works at the intersection of the data engineer and data analyst. So how does the analytics engineer role differ? 

The main point of difference is not the work done, but the intersection. Data engineers often have to answer to the entire company, so they think big-system big-picture, and do not dwell on the intricacies of particular data sets. Data analysts dive deep into analyses and often do not have the technical skill to build additional pipelines for further questions themselves - and need to rely on a data engineer.

So the analytic engineer fills the gap between big-picture and small-picture, by building custom (but durable) solutions to the analytics team. 

To use the metaphor from Dataform:

“Data engineers build the cupboard, they gather together the wood and the tools and put it together. The Analytics Engineers open the cupboard and start putting in the plates, mugs, bowls, and arrange them in a certain order. This could be arranging them into particular colours, shapes or sizes. Data analysts then go into the cupboard and they know where everything lives as it is arranged nicely. They can then grab the small blue mug they were looking for and go make a cup of tea!”

You can think of an analytical engineer as the role that can answer technically hard questions using data engineering as a tool, but understanding data analytics as the goal.

4. What are the career prospects for an analytics engineer?

The career path for future analytic engineers seems very promising.

According to Glassdoor, analytic engineer salaries show a high level of compensation, averaging at $102,000 per year, and ranging between $76,000 - $136,000. The particular compensation will depend on multiple factors, such as seniority, skillset (including data tool expertise), industry, and geographical location - to name just a few. 

Given the wide gap between demand for data experts and supply - there are 3x as many job vacancies as workers -, analytic engineers have a lot of options to choose from. When job hunting, just keep in mind that the analytics engineer jobs come under different job titles: 

• Data & analytics developer
• Analyst programmer
• BI analytics developer
• Modelling & Data Analysis Engineer
• Business Analytics Engineer
• Etc. 

5. How can you become a better analytics engineer?

Start by improving your data engineering skill set and knowledge of data tools and processes.

Keboola - the Data Platform as a Service platform - can help you with both:

1. Get certified as a data engineer. Learn the ropes end-to-end, so you can confidently drive engineering change within your organization.
2. Automate your data stack with Keboola. Keboola offers end-to-end data integrations and tooling, that allows you to connect your existing data tools into a centralized solution. So you can orchestrate, monitor, and automate all your (analytic) data pipelines and governance from a single solution. 

Contact our data engineers to learn how Keboola propelled their journey into becoming a better analytics engineer.