Run your jobs faster with Keboola’s new feature: Dynamic Backend size

Data transformations are the backbone of smooth-running data operations. 

Transformations are used in data replication between databases, data migration from cloud to on-premise, and data cleaning (aggregations, outlier removal, deduplication …) aka all the good stuff that goes into extracting insights from data. But as any data professional can attest, transformation can also be a painful bottleneck. Think scripts that run for an entire day and finish just before the next scheduled job.

With the growth of data volumes as you scale and with the unexpected bursts of incoming data that strain your transformation resources, jobs can delay or even put to a stop ETL processes downstream. This is why Keboola launched a new feature: Dynamic Backend per job. 

The Dynamic Backend allows users to allocate more resources to speed up long-running jobs.

This feature is available only for non-PAYG projects with new transformations.

How does dynamic backend scaling work?

Pick a transformation or job that is a good candidate for backend re-sizing. 

Before you run the job or transformation, you can manually configure the size of the backend by picking from three options (SMALL, MEDIUM, and LARGE) in the transformation configuration pane.

With this release, the dynamic backend is for now limited only to the Snowflake DWH transformations.  

But make sure to check the “What’s coming next?” section below for a little teaser of what’s coming in the next few weeks.

How to determine which jobs would be best optimized by changing the size of the backend?

An excellent way to pick candidates for backend resizing is to navigate to “Jobs” in your navigation bar and pick the jobs with the longest “Run time” (penultimate column):

How do you access the dynamic backend scaling?

Dynamic backend resizing is automatically available for all subscribing users. Simply go to Transformations in the navigation window, choose a transformation you want to resize, and pick your candidate. 
As you will inspect the transformation, the dynamic backend resizing will be visible in the right-hand side configuration pane. Below you can see we picked a LARGE size for this job:

If you are on the pay-as-you-go (free) plan, you will not have the option to dynamically configure the backend size:

No worries, you can still access the feature. Simply reach out to us using the feedback button in the lower-right corner, and we will work together to find a solution.

What’s coming next?

Soon, we will  launch the dynamic backends also for any transformation job in Python or R.  So you can unlock the power of dynamic backend size on demand for all your transformations.

Wondering what other new features are brewing on our backlog? 
Make sure to check our Feature Roadmap to get a glimpse at everything we plan, are developing, or are already launching.