GlassFlow v2.2.0: Better Observability & Kafka → ClickHouse Pipeline Reliability

GlassFlow has released version 2.2.0, focused on strengthening the platform for real-time Kafka → ClickHouse pipelines. The update mainly improves observability, data type flexibility, and operational usability. These additions make GlassFlow more suitable for teams moving from experimentation toward continuous production workloads.

Better Observability with OpenTelemetry

A major component of this release is native OpenTelemetry (OTEL) integration.
GlassFlow can now publish pipeline-level metrics such as:

Records consumed from Kafka
Records processed
Records written to ClickHouse
Records redirected to DLQ

When running multiple replicas, metrics are tagged by service instance, making it easier to pinpoint performance issues. Logging is also more structured, helping identify errors without sifting through unformatted logs.

OTEL metrics integrate directly with Prometheus, allowing users to build dashboards in Grafana. This gives clearer insight into system health without additional instrumentation.

- job_name: 'glassflow-otel-collector'
  static_configs:
  - targets: ['glassflow-chart-otel-collector.glassflow.svc.cluster.local:9090']
  metrics_path: /metrics
  scrape_interval: 15s

Map Data Type Support

GlassFlow now supports Map and Array of Map fields within pipeline configurations.
This is especially useful when working with real-world JSON sources that include dynamic key-value or nested structures.

Map(String, String) – For simple key-value pairs
Array(Map(String, String)) – For arrays of maps
Automatic type conversion – Map values are automatically converted to strings for ClickHouse compatibility
Nested JSON support – Maps work seamlessly with nested JSON structures using dot notation

Dot-notation is supported for referencing values inside maps, and type conversions ensure compatibility with ClickHouse. This removes the need to flatten nested content before ingestion and simplifies schema management.

UI Enhancements for Operations

Pipeline detail pages now surface ClickHouse table statistics such as row counts and table size. This provides immediate confirmation that data is being written and growing as expected.

Additionally, pipeline configuration files can be downloaded directly from the interface. This helps with versioning, collaboration, and environment migration. The pipeline list view has also been refined to present relevant pipeline information more clearly.

These enhancements reduce the amount of switching between tools when validating ingestion or reviewing configurations.

More Reliable Null Handling

Earlier versions of GlassFlow could encounter type issues when null fields appeared. v2.2.0 improves how null values are interpreted and passed through the pipeline.
Records with problematic nulls can be routed to the Dead Letter Queue instead of halting processing. This allows pipelines to continue operating even when upstream data is inconsistent.

Conclusion

GlassFlow v2.2.0 brings practical improvements that strengthen day-to-day streaming operations. The update provides:

Clearer visibility into pipeline behavior
Better handling of complex data structures
More operational information through the UI
Increased resilience when encountering nulls

These changes support more predictable and maintainable Kafka → ClickHouse pipelines, making GlassFlow a stronger option for production workloads.

Exploring ClickHouse for Your Analytics?

At Quantrail, we’ve spent time understanding the challenges teams face with real-time analytics. That’s why we provide hands-on support around ClickHouse – from smooth deployments and guided migrations to ongoing service agreements that keep systems reliable.

At Quantrail, we see these updates in action when helping teams adopt ClickHouse for demanding analytics pipelines. For example, we recently shared how a customer achieved bare-metal level performance with ClickHouse in production – you can read the full story here Success Story: Quantrail Baremetal

Whether you’re moving from another database or simply want to get the best out of ClickHouse, our goal is to make the process straightforward and sustainable. If this resonates, we’d be happy to share how others are approaching it and explore what could work for you.

Contact

Quantrail Data