Keboola Storage

Introduction to Keboola Storage

Keboola Storage is the heart of your Keboola data infrastructure, efficiently storing and managing tables and files. It is designed to simplify data management, providing a centralized location where your data is securely stored and easily accessible.

Keboola Storage supports various powerful backend solutions, including Snowflake and BigQuery, tailored to your organization's specific needs and preferences. By default, data is stored as STRING, but Keboola's Native Datatypes feature allows you to maintain original data types for enhanced accuracy and performance.

Organizing Data with Tables and Buckets

Within Keboola Storage, your data is neatly organized into buckets, much like folders on your computer. Each bucket can contain multiple tables, making it easy to group related data logically.

• Buckets: Containers that allow you to structure and categorize your data tables effectively.
• Tables: Individual units of data storage within buckets, storing structured data with clearly defined columns and rows.

Detailed Table Insights

Clicking into a table within Keboola Storage reveals detailed insights and metadata:

• Table ID & Name: Clearly identifies each table for easy management.
• Primary Key: If configured, easily view your primary key for efficient data indexing and lookup.
• Recent Configurations: Quickly review the latest updates or imports to your tables.
• Creation & Modification Dates: Track table history, including the date of creation, last import, and most recent changes.
• Row Count & Size: Immediately understand the volume and size of your data.

Track Every Action with Storage Jobs

The Jobs tab in Keboola Storage provides a comprehensive log of all actions performed on your storage tables. For example:

• Data imports and exports logged clearly with timestamps.
• Details on imported rows and column structure for quick verification and debugging.

Key Features That Enhance Data Management

Data Sample

The Data Sample feature enables you to preview sample rows from any table, providing immediate insights into table contents. Additionally, it includes basic filtering capabilities, allowing you to quickly verify and assess the relevance and quality of your data.

Restore

Keboola Storage's Restore option lets you revert a table to an earlier state, safeguarding against accidental data loss or unwanted changes. The platform typically retains seven days of historical data on Snowflake backends, enabling you to roll back to a previous version easily. Restoring creates a new table, preserving your existing data intact.

Snapshot

The Snapshot functionality provides an additional safety net, allowing you to create and store table snapshots indefinitely. Snapshots are especially valuable before making significant table alterations. If a change proves problematic, you can swiftly revert to the preserved snapshot, minimizing downtime and data loss. Additionally, snapshots can be exported for external use or backup.

Understanding Table Usage Across Transformations and Workspaces

The Usage tab provides transparency into how each table is utilized across your transformations and workspaces, enabling informed decisions and effective data governance. Quickly identify dependencies and connections, ensuring efficient workflow management.

Practical Examples and Use Cases

Use Case 1: Data Governance and Compliance

Organizations needing strong data governance rely on Keboola Storage’s robust tracking and auditing capabilities. The detailed logs in the Jobs section provide transparency for regulatory compliance, allowing businesses to demonstrate thorough oversight of data handling practices.

Use Case 2: Reducing Data Loss Risks

By routinely creating Snapshots before major table alterations, teams can significantly reduce the risk of data loss. For example, an analytics team preparing new customer segmentation models can snapshot their existing customer data tables before making substantial schema changes, ensuring quick recovery in case of errors.

Use Case 3: Efficient Debugging and Troubleshooting

The Storage Jobs log simplifies debugging and troubleshooting. For instance, if a recent data import appears incorrect or incomplete, a data engineer can quickly trace the import event details, pinpoint the exact issue, and resolve it swiftly.

Best Practices for Maximizing Keboola Storage Efficiency

2. Consistently Organize Buckets: Group related tables logically to streamline data management.
4. Regularly Use Snapshots: Adopt a proactive snapshot strategy before modifications to mitigate risks.
6. Monitor Usage Regularly: Regularly review table usage to understand dependencies and prevent unintended consequences from table modifications.
8. Leverage Native Datatypes: Maintain original data types for optimal performance and accuracy in data-driven workflows.

Further Resources and Documentation

Keboola provides comprehensive documentation and manuals to support users in maximizing the benefits and capabilities of Storage. Explore detailed guides, tutorials, and best practices through the linked resources provided in the Keboola presentation.

Conclusion

Keboola Storage is a versatile, powerful, and user-friendly solution for data storage and management. With advanced features like Data Sampling, Restore, Snapshots, and comprehensive logging, Keboola Storage simplifies data operations, enhances security, and optimizes workflows, empowering your organization to leverage data-driven insights with confidence and ease.