How to choose your ETL tool

ETL tools help companies to streamline and enhance their data operations. They automate the repetitive tasks involved in extracting raw data from sources, transforming data into a consumable format and loading into data warehouses, where it is ready to be analyzed.

With so many offerings available to you, all of which do the heavy lifting ‘out of the box’, it is hard to discern which ETL tool is best suited to your needs.

You can always skip the reading and check out our FREE ETL calculator spreadsheet.

ETL tool evaluation #1: Core features

ETL tools offer several integrations, from 3rd party apps data extractors to data warehouse writers. 

When considering ETL tools, make sure to check their features to gauge whether or not they cover your technical needs:

1. Pipeline coverage. The best ETL tool will cover all of your current incoming data sources (3rd party apps, custom scrapers, CSV files…), destinations (Postgres, AWS S3…) and transformations. Meticulously document your technological needs and check them against the vendor’s offering. If any part of your sources, destinations or transformations needs cannot be met, you will have to fill the gap with additional in-house engineering or by purchasing multiple tools (and thus raising your overall costs).‍‍
2. ‍Data quality. Check exactly how the features work. Use the demo, trial or freemium version to ensure that your data does not get corrupted or changed in any way to make it unusable. Some ETL tools cut off UTM tags from marketing reports, which prevents you from tracking your advertising campaigns. Other vendors rename and abbreviate your columns when inserting data into your data warehouse, which complicates lineage tracing and analysis. Some open-source solutions even change your timestamp’s timezone when inserting into the database. ‍
3. Data granularity. Assess whether the data pipelines are built to the level of detail that you require. Weed out tools that cover your entire pipeline, but fail to offer the necessary granularity for running your operations. An example would be a tool that can export Salesforce weekly sellout information, but not daily reports.

ETL tool evaluation #2: Price

The price tag is often a decisive factor when weighing up different vendors, as it should be. You should budget for a tool that does not cut into your operational ability or dampen your scaling ambitions, but still enables your engineering value and insights to grow.

The right solution will automate your data pipeline and free up engineering hours, which can be redirected into more revenue-generating work. 

However, keep in mind that the vendor’s price is not the only cost you should consider. The Total Cost of Ownership (TCO) for ETL tools also includes:

1. Maintenance costs. If the ETL solution only covers part of your pipeline or you opt for an in-house engineered solution, you will have to shoulder the cost of fixing broken pipelines. From APIs changing endpoints to open source libraries deprecating functions, data pipeline operations require constant updates. That’s why companies tend to opt for vendors in the first place - they take over the maintenance responsibilities. 
2. Opportunity costs. The speed of implementing your data pipeline affects your opportunity costs. Think of it in terms of ‘time-to-insight’. The average in-house engineered ETL pipeline takes 9-12 months from idea to execution - that’s almost a year to gather business intelligence to boost your competitive edge. By picking a vendor solution, you have an operational insight-generating machine in weeks instead of months.
   

ETL tool evaluation #3: Security

Every vendor claims that “security is their number one priority”. But when data breaches have the potential to cause extensive, sometimes irreversible, damage to your company, you’d better check the validity behind those security claims!

You don’t have to be an expert in data infrastructure to do this, nor hire people who are. 

Check whether the vendor’s architecture covers the most important industry security standards and certifications, such as:

• GDPR compliance
• HIPAA compliant architecture
• SOC 2 and SOC 3 
• ISO 27001 Certification

ETL tool evaluation #4: Scaling

Your ETL tool needs to scale with you. As your company grows, you will produce and consume more data. The best ETL tool will adapt to your growth spurt without impacting your speed.

There are two ways to scale a company, and the best ETL tool will assist you in both of these:

1. Vertical scaling. Increasing the volume of your existing operations will result in more data. The right ETL solutions are resilient to scaling in data frequency and volume. If you are growing rapidly, take a look at the tool’s other use to see if they have high-profile cases. You do not want your ETL pipeline to degrade in quality at a time when you can profit from it the most.
2. Horizontal scaling. Going into new markets, adding new channels to your marketing stack and incorporating new resources all reflect horizontal growth. Some ETL tools will charge extra for each new data source, transformation or data warehouse. Be aware of the increased price tag, which may hinder your growth.
   

ETL tool evaluation #5: Flexibility

Flexibility provides long-term returns on tool investment:

1. Add your own sources. How customizable is the tool? Even if the tool doesn’t automatically have your data source out of the box, does it use open-sourced solutions, such as Singer, to speed up your addition? Adding experimental and novel sources can mean the difference between being an innovator or a lagger on the market. 
2. Unstructured data transformations. With the exponential growth of data globally, there has been a rise in unstructured data (long-form text, internet of things logs, etc.) as well. Companies that can incorporate unstructured data into their operations have a competitive advantage over others. However, not all tools are good at extracting unstructured data from their sources, and even fewer offer customized transformations which allow you to parse the data for business-valuable enrichments. Check the vendor’s offering around handling unstructured data pipelines.
3. Quick pipeline changes. Switching the destination to a different database, adding a new pipeline from scratch, and making general alterations to an existing pipeline are all in the domain of flexible tools. Evaluate tools based on how quickly they can adapt to changes. 

ETL tool evaluation #6: Future promise

The feature roadmap of ETL tools will not offer you immediate benefits, but it’s a good sign of things to come. Not all vendors are equally innovative, and you want to put your money where the growth will be.

For example, some tools already offer machine learning within their pipelines, while others are still adding yet another data source. Keeping one eye on the future of your toolbox can help you to plan your next strategic move.

ETL tool evaluation #7: Support

It’s not a question of if your ETL pipeline will break, it’s a question of when. Outages, disruptions, bugs, and hardware issues are as certain as the sun rising. It doesn’t even have to be the vendor’s fault! These issues can happen at any point in the data pipeline. Even tech giants like Google experience frequent issues with their reliability. 

When comparing ETL tools, check the extent of their support:

• Do they offer step-by-step onboarding?
• What is the availability of their maintenance support (24/7)?
• Is their online support or knowledge base well written and self-serving?
• Do they offer training for tool usage and troubleshooting?
• Does the tool have monitoring features and alerts ingrained within it?
  

Instead of trying to prevent all problems from ever happening, pick a tool that helps you to identify and fix the problems quickly once they arise.

ETL tool evaluation #8: Setup

Ease of setup and deployment is important. The easier the setup, the lower your opportunity costs will be on waiting for insights. 

When weighing up different ETL solutions, pay special attention to:

1. Speed of deployment. How quickly can you start using the tool after you have signed the contract?
2. Ease of use (for non-technical people). How accessible and intuitive is the tool for a wide range of users? You should also consider the solution’s user interfaces and user experience when playing around with the tool. Tool adoption is connected to its usability.
   

Overall, setup is not as important as the other criteria. A tool with a more complex setup will take 2 hours instead of 2 minutes. But inflexible tools and those with data quality issues may take 2 weeks to resolve.

ETL tool evaluation #9: Potential for transformation

The right ETL tool will not be limited to just mapping your current processes to a new infrastructure. It will either introduce better engineering practices for existing tasks (e.g. more extensive incoming data schema validation) or inspire you to head in new directions. Sometimes, the innovation is so vast that it transcends the role of ETL tools and the technology becomes a driver for change.

A great example of the latter is Keboola’s Scaffolds. These are end-to-end data pipeline recipes, which can be deployed in just a couple of clicks. Users can create novel pipelines without any engineering assistance. From experimenting with machine learning to running quick proof of concepts, Scaffolds speed up operational evaluations. Determine whether a pipeline can help your business grow before investing weeks into it.

The “Evaluate ETL Tool Calculator”

Choosing an ETL tool can be overwhelming. To make your decision easier, we’ve prepared a calculator which can help you to evaluate your ETL tool choices.

You can get it by opening the conversation with out chat bot- just start the bot and it'll take it from there.

Just copy the file to your local Google Drive and edit it to see the magic happen. 

Summary

When deciding on the best ETL tool for your company, judge it against the 9 criteria:

1. Core features: Does it cover all of your sources, destinations and transformation needs?
2. Price: Is the value that the tool provides higher than the cost of (not) implementing it?
3. Security: Will your data be secure with the ETL tool?
4. Scaling: Will the ETL tool help to scale or hinder your growth?
5. Flexibility: Can the tool adjust to your special requirements?
6. Future promise: Does the future roadmap of the vendor look beneficial to you in terms of using the tool?
7. Support: Does the ETL tool offer extensive support?
8. Setup: Is the setup smooth?
9. Potential for transformation: Will the tool open a new frontier for your company?

Keboola is much more than just an ETL tool

Keboola is not only satisfying the criteria but also surpassing it in many ways. It's an DataOps platform built for making your everyday easier - from automating simple tasks, to automating the entire data pipeline. It's a platform that'll grow with you and won't increase in its costs. It gives developers the creative freedom to create custom made apps within a safe environment, and with our full support.

It's the only tool you're ever need growing from a startup to an enterprise.

Here's how it surpasses the 9 criteria:

• Core features. The platform is architectured to be open by design, meaning that the core features cover all of our customers’ needs and can be adjusted to suit their specific business strategies with ease.
• Price. From as low as 2,500 USD/month, Keboola’s accessible pricing makes it easy to deploy enterprise-level solutions for growing customers.
• Security. Keboola relies on Amazon’s state of the art security… and then takes it a step further.
• Scaling. Keboola’s infrastructure is tried-and-tested, helping enterprise customers scale beyond expectations. What’s more, we do not charge you for new sources, databases or transformations. When you grow, we grow with you.
• Flexibility. The architecture is fully modular, allowing for plug-and-play flexibility. Keboola also relies on Singer for integrations. If your app is not among the many that come out of the box, creating a new component is simple.
• Future promise. Innovation is our drive. Data science, UX, and think-out-of-the-box solutions such as Scaffolds are what makes Keboola the player to look out for.
• Support. Onboarding, 24/7 support, extensive documentation - we will not let you deal with anything without our help.
• Setup. It’s so easy, we put it all into a tutorial.
• Potential for transformation. What sets us apart from our competitors is not the state-of-the-art ETL engineering that pumps Keboola’s blood. Why? Because Keboola is more than just an ETL tool. It is a suite of transformative technologies built on top of the ETL: scaffolds, data catalogs and digital sandboxes are just some of the solutions that we'd like you to explore.

Words are all good, but we’re ready to back them up. If the platform does not provide value within its trial period, then it’s probably not worth it.

Try Keboola for 2 weeks and see if it works for you. Free of charge - it’s on us.