Create custom functionalities in Keboola's Developer Portal

Every time you write another piece of code that picks up data from an FTP server, a small piece of you dies. As a developer in the data space, you know what we’re talking about. 80% of your time can be taken by building and improving the environment and tools, maintenance tasks, and pieces of functionality. That's simply too much time dedicated away from tackling more important issues. And the reality is that only a very small part of developers' time is occupied by the creative work, solving the real, specific problems.

On the other hand, you may be concerned about using too complex tools because they have the tendency to curtail your flexibility and ability to build whatever is needed. What if a massive part of those 80% was either automated, or just simply done for you in a way when you can rely on it, without concerns for security, fit to your deployment model, or reliability? And at the same time, you would have the flexibility to build your own functionality?

Welcome to Keboola Connection.

The best of both worlds

The way Keboola alleviates those concerns is the openness of the platform and the ease with which one can build their own components and functionalities for Keboola Connection (KBC). We cannot anticipate every need, and we cannot prioritize everything we do anticipate or know about. Opening the platform to customers, partners and individual developers enables anyone to bridge any functionality gap they may come across, as well as open the opportunity for developers to make their applications easily accessible by any Keboola customer.

As a concept, this is obviously nothing new. Apple didn’t build every single app, they have iOS and the AppStore. The rest is the community. The way Keboola implemented the principle truly expands the possibilities while keeping all the benefits of a solid, integrated platform.

Why you should consider building components for KBC

There are two main groups of solutions the community can develop - building components for KBC, or integrating with it via all the available API’s. This article will focus on the components, leaving the integration options for later time (if you are interested in that, check out the documentation).

Visiting our list of components, one can browse through all the publicly available components (there are many more unpublished) for inspiration. When developers build components for KBC, they usually solve for one of the following use cases:

• Integrations: Whether it is to pull data from a new source, write to a new destination or to integrate with an API for data enrichment, these components are the most commonly built. While Generic Extractor allows for code-less integration with most REST API’s, a dedicated component (Extractor in this case) offers more flexibility and options for optimizations and configuration.
• Business logic and models: Either proprietary or reusable models are often wrapped into applications rather than left to the transformations to be coded every time. That way they can be used across projects, there is more flexibility in languages and environments used (anything that can run in Docker can be turned into a component), and the code is obscured from a “regular” user.
• Data manipulation: Often, something needs to happen to the data on the fly. Whether it is conversion between file formats, compressions, quick text operations such as find and replace, adding headers to CSV files etc, processors are the way to go. They are attached to a component and modify data files either on input or output.

All tools, documentation and a support channel for developers are accessible through the Developer Portal. Anyone can create a developer account and start building right away. For applications to be published in the platform there is an approval process, where the Keboola team reviews the component, makes sure there is proper documentation and a support agreement in place, and sorts out monetization for the developer, if applicable. The common denominator is a quality check which preserves the expected user-experience consistency for the end-user of the platform.

‍

Start your developing journey today

What do you need to get started? Not much. As mentioned before, components run in Docker so you need to be familiar with it. Knowledge of Docker commands, while not strictly essential, is recommended. Understanding how to send API requests (via postman for example) also helps. Then just set up a git repository, select a language of your choice, check out the documentation and get going!

All that being said, of course, Keboola listens to feature requests from customers - there is a link directly in the UI for feature wishlist. We look forward to hearing from you!

If you want to see the Developer portal in action, join us at a webinar on May 7th, where one of our colleagues will give an introduction and answer any questions you may have.