How do you load data to REST-like APIs when there is no dedicated data destination component?

How To

July 12, 2022

No items found.

Keboola can help you send and use your data outside the Keboola’s platform

Imagine you want to take data out of the Keboola’s ecosystem. But the unexpected happens - there is no devoted Writer component! You search through all the destination Writers - from Snowflake to Excel - but nothing works for you.

This is when the Generic Writer comes in handy.

The Generic Writer is a Keboola Component that allows you to send data to an API destination outside of Keboola. Even if there is no existing out-of-the-box Writer Component for your destination and with no need to write your own code.

You can deploy the Generic Writer for multiple use cases:

Webhook triggers
Notifications, e.g., Slack
Writing JSON data (UPDATES, etc.)
Sending CSV files as binary data (can be gzipped for performance)
Calling arbitrary endpoints with parameters defined on the input

E.g., DELETE api.com/[[user_id]] where user_id is a column in the input table

How does the Generic Writer work?

The Generic Writer allows you to send the data to a destination endpoint that is part of a REST API.

All you have to do is pick the input data from Keboola’s storage and specify the configuration file that authenticates your HTTP request and determines how data will be loaded to the destination.

The configuration file gives you multiple options:

HTTP method. All the traditional HTTP methods are supported: POST, PUT, UPDATE, DELETE, …
Granularity. You can send data in JSON or BINARY configurable chunks or iterate through the input data row by row and perform a HTTP request for each row.
Request content type. You can send the data as a JSON or BINARY payload, and even granularly specify the subtypes of the payload content (JSON encoded, gzipped binary, even an empty request).
Input data mapping and variables. The writer automatically converts your input data from its current data type (CSV file, Snowflake table, …) into the payload content type. And it gives you a lot of flexibility, you can specify the conversion data types mapping and use dynamic functions to generate new values from existing input data before the information is sent.
… and many more. The entire Writer works out-of-the-box with its basic configuration, but it can also be highly customized with custom parameters, headers, dynamic functions, retry configurations, etc. - check all the configuration options here.

#getsmarter

Oops! Something went wrong while submitting the form.

What are the benefits of using the Generic Writer?

Using the Generic Writer has multiple advantages:

Universal. No matter what your endpoint is, as long as it is designed as a REST API, you can send your data to any endpoint. The same goes for your input data - no matter what data type your input data is, you can send the data to your destination endpoint.
Speeds up production time. Instead of writing boilerplate code, you just specify the JSON configuration file and let Keboola do the heavy lifting.
Resilient at Scale. No need to discover the edge cases of your custom-brewed load scripts. Let Keboola do the heavy lifting, while you focus on more interesting initiatives. The Generic Writer allows you to configure retries to avoid throttling issues, chunk size, and content type for high throughput use cases, as well as multiple configuration options that take care of the necessary, but tedious details.
Secure. By default, Keboola encrypts all traffic at rest and in transit.
Allows Multiple Use Cases. You can use the Generic Writer to send all your data to a single endpoint. Or use the input data to dynamically change the outcome of your calls. By iterating through rows of your data, you can specify different behaviors. For example using a column as a parameter to your HTTP calls you can perform deletions on the endpoint (DELETE api.com/[[user_id]] where user_id is a column in the input table), use the Generic Writer to trigger a webhook, or even to send Slack notifications.
Fully automated. The Generic Writer is executed on the Keboola’s backend as a job. And therefore benefits from all the features of a job. It can be orchestrated, it is constantly monitored for errors and automatically alerts you when something goes wrong, it is versioned, it is sharable, and it can be scaled dynamically.

How can I try the Generic Writer myself?

Log into Keboola and head to Components. Use the search bar to find and select the Generic Writer destination.

The wizard will guide you through the Component configuration. But the basic idea is simple:

Select the data in your Storage you would like to send to the REST-like API.
With the documentation of your destination API at hand, you fill in the JSON configuration file that authenticates your HTTP request and specifies what CRUD operation your HTTP request will do. No worries, Keboola is so user-friendly, it already pre-fills the JSON configuration with a general template that you can adjust to your needs. Or you can refer to some existing real configuration examples
Run the component once or automate the job that will orchestrate the Generic Writer at regular intervals.

If you get lost, you can always refer to the extensive documentation or shoot us a message, we are always happy to help.

The Generic Writer is just one of the many features in Keboola’s platform. Learn more about how Keboola can help you automate your data operations with Keboola’s Data Academy or explore all the features Keboola has to offer.