Do you want to build an ETL pipeline?

What is an ETL pipeline?

An ETL pipeline is a software architecture design that describes a specific type of data pipeline. 

The ETL pipeline flows data from source systems to target systems. In between, data is filtered, cleansed, aggregated, enriched, and transformed to be ready for BI tools and analytics.

The ETL pipeline can be broken down into three processes: extract, transform, and load. 

1. Extract

The extract phase extracts or collects data from raw data sources, such as APIs, sensors, marketing tools, and other sources of data.

The raw data can come in many structured and unstructured formats, such as JSON, XML, Excel, or CSV files, as well as entries from transaction databases. 

Once data is extracted, it is moved into the transform phase.

2. Transform

The transform stage cleanses, enriches, and alters data into a format and shape that is ready for consumption by users and (business intelligence) applications.

Data transformation includes:

1. Data cleaning: remove missing data, cut out corrupted data, recode disparate data to same standardized nomenclature, discard outliers, one-hot encode categorical data, deduplicate data …
2. Data enriching: join data across different sources, complement in-house data with attributes gathered from 3rd Party Apps specialized for enriching, compute calculated fields, …
3. Aggregations: compute aggregates of numerical data, cluster categorical data into groups, …
   

The output destination of the transformation stage is the staging database, where the cleansed, filtered, aggregated, and enriched data is pushed, before being loaded in the target destination.
‍

(read more about ETL transformation here or dive deeper into the subtleties of data cleaning here)

3. Load

Load is the third and final stage of the ETL pipeline. Data is loaded into the data storage of your choice. Be that a database (MySQL, PostgreSQL, etc.), data mart, data warehouse (Amazon Redshift, Snowflake, and Google BigQuery), or data lake (Amazon S3, Apache Hadoop, Microsoft Azure Data Lake Storage (ADLS), …).

We treat loading data separately, because the implementation details (the data engineering) of the loading phase depends heavily on the target data storage - from schema changes to typing coercion, a lot of engineering efforts go into loading data properly. 

The challenges of ETL pipelines

The ETL process might seem straightforward, but it hides many pesky challenges:

1. Changing data sources. Data sources change all the data. From migrating from one API version to another, changing payloads, outdated authentications to access data, and a million other minor inconveniences, the data will alter at its source. Solutions that extract data will have to be monitored for errors and constantly updated. 
2. Low transferability. Engineering a solution for a specific ETL pipeline does not transfer to a new environment. So even if your engineers spent weeks designing the data pipelines for marketing data extraction, the code will not transfer well to a new subject domain. 
3. Data quality risks. Assuring data quality can be hard. Data quality risks are mitigated by engineering another solution on top of your ETL pipelines to perform data validations - check for missing, corrupted, and other low-quality data. 
4. Monitoring. ETL pipelines need to be monitored for errors - from jobs not starting, to jobs not completing, monitoring is another feature you have to build on top of your ETL pipelines to assure they are working correctly. 
5. Scalability. A solution designed for 100 rows/second is drastically different from the solution that can handle 100 million rows/second. ETL pipelines do not scale well. As your organization grows and your data increases in volume, velocity, and variety, expect to perform data migration from legacy solutions to novel ones. 
   

How to build an ETL pipeline?

ETL pipelines are usually built using devoted ETL tools. ETL tools are preferred over custom scripting because of many advantages:

• ETL tools automate the ETL pipeline by default, thus saving you precious engineering hours.
• The vendor takes care of tooling maintenance.
• ETL tools take care of scaling.
• Data orchestration by design - no need to write
• Tooling comes with built-in data pipeline monitoring - so you get alerted when errors happen.
  

(check the best ETL tools on the market)

But tooling does not solve all of the problems. 

The main engineering tradeoff you need to decide upon is whether your ETL process needs to support real-time data or can be updated in batches.

Real-time data needs specialized technology (like Apache Kafka streams) or tooling (ETL tools supporting data streaming), while batch processing can extract, transform, and load data in batches at pre-defined (scheduled) time intervals. 

Real-time data offers you immediate business insights, but it also comes at the cost of increased engineering overhead and more challenging tooling or solutions.

Ultimately, the decision will depend on your business model - whether you will need real-time decision-making or not. 

Automate your ETL pipelines with Keboola

Keboola is a data integration platform that automates data operations end-to-end. 

Use it to build and deploy your ETL pipelines:

• Automate data extraction and loading with over 250 integrations between sources and targets (be it databases, data warehouses, data marts, or data lakes).
• Automate data cleaning and transformations with Transformations and Applications (bonus: it comes with data versioning).
• Scale seamlessly and automatically, without having to worry about increased data loads.
• Monitor your data pipelines out of the box, without having to rely on additional tooling. 
• Orchestrate your ETL jobs to run in real-time or in scheduled batches with a couple of clicks. 
  

Try it out. Keboola offers a no-questions-asked, always-free tier, so you can play around and build your pipelines with a couple of clicks.