How to load data to REST APIs

You've probably heard of Generic Extractor (Keboola's component for extracting data from virtually any API), and now get ready for Generic Writer. A tool which allows you to load data to REST APIs.

I’m no programmer, it's why I am not able to easily write custom components for Keboola. I remember when the first version of Generic Extractor was released in 2016. I was part of the Professional Services team back then and that event has brought tons of benefits to my work, as I’ve been able to deliver components for getting data from various sources without need of external help. Of course, custom components have their merits, like better UI or possibility to tune up the performance, and are able to get data from the APIs that Generic is not able to handle.

But Generic Extractor is still a great component to get data from the most of the REST APIs, to create data feed super fast and can be also transformed into the regular component later on - plenty of our components are based on Generic Extractor, like Zendesk, LinkedIn or Github.

Introducing Generic Writer

When I needed to send data to some APIs, everything became much more complicated and I had to ask a more skilled programmer to create a component. With the introduction of Generic Writer (built by David Esner, our consultant and creator of many Keboola Connection components), I am now able to load data by myself. Just like the Extractor, Generic Writer configuration is defined in JSON and you only need to add an Input Table with the request (all general methods are supported, like GET, PUT or DELETE).

Use case: Create and Item in Webflow Collection

Since its beta release a couple of weeks/months ago, I’ve created plenty of configurations which help me solve many use cases. I’d like to share one with you to show you how the new Writer works - creating an Item in Webflow Collection. Webflow is a tool we’re using to build our Website. Collection might be used as a page category or group. Item is a page.

Not that long ago, we’ve started a series of courses called Keboola Academy, where you can learn using Keboola Connection in a power user way and when you’re successful, you’ll receive a badge which is accessible on our website on your personal page as a proof. This is how your page looks like when you finish all of the tracks (of course I’m just a pretender and I have never finished the tracks, but looks good anyway…).

This example use case for the Writer is to create a page (like the one in the example) for a person which finished the course. Let's take a look at the API call which creates such page first:
‍

‍

Nothing unordinary - method, URI, headers, single url parameter, and body. You can find the API call description in Webflow API docs.

And this is the configuration of the Writer which does the same (except the values in the body): See the configuration.

To have a complete picture before describing the parts of the config, here’s the table we’re using as the Input:

‍

Let's take a look at some interesting parts of the configuration:

• path - you can see {{fields___cid}} in the URI. That’s a variable which is defined in the iteration_mode. By using that combination, each API call uses the value of the defined input table’s column. Such a column is not used in the body of the request itself.
• token_encoded - it uses a function (like you know from the Generic Extractor) to combine hashed value of the API Token (#token) and string “Bearer ”, to form secure value being used in the Authorization header.
• json_data_config - place where you define additional parameters to help the Writer determine how to work with the input table values.

• chunk_size - how many records should be sent in one batch.
• infer_types_for_unknown - if set to true, unspecified data types will be defined based on the values in the input table.
• delimiter - the value determines what delimiter should be used to define JSON objects of the request. The usual value would be a single underscore, but sometimes you have underscore in the parameter value in the API and you need to deal with that. Using double underscores will help you achieve that, as the first two underscores will be the delimiter and the third will be the actual underscore in the parm name (example: “fields___draft” will create “_draft” param in the “fields” object).

• column_types - here you can define data types being of the columns benign sent to the API if needed.
  

Overall, the configuration is not that difficult. And from my experience, once you create your first working configuration, the other will be much easier (I recommend just to copy the previous one and adjust it to your new needs :) ).

Start creating Generic Writer configurations by creating a free account.