ClickHouse processor configuration

The memory-service process clickhouse command accepts CLI flags or environment variables. Use the toggle to switch between formats. CLI flags take precedence when both forms configure the same setting.

Tip: Your format preference is saved across visits.

This page covers the analytics processor. See Service configuration for the Memory Service server.

Memory Service connection

The client must have the Memory Service admin role. A second processor with the same client ID cannot acquire the checkpoint lease.

Export behavior

The exporter ID separates independent analytics datasets that share ClickHouse tables. It participates in row keys, deterministic event IDs, purges, projection registration, and metrics. It defaults to the processor client ID. Keep it stable for the lifetime of a dataset; changing it starts a separate ClickHouse row namespace.

All resource kinds and features are enabled by default. Repeat --disable for multiple CLI selectors. Separate multiple MEMORY_SERVICE_CLICKHOUSE_DISABLE selectors with commas.

The supported feature selectors are:

For example, this setting keeps current-state rows but omits all lifecycle events:

MEMORY_SERVICE_CLICKHOUSE_DISABLE=*:lifecycle_events

The processor rejects unknown resources, unknown features, and unsupported combinations such as entries:lineage. A bare resource selector also disables all features for that resource. The common processor schema remains present, so tables for disabled outputs can remain empty. Rows written before an output was disabled remain until a purge or TTL removes them.

Disabling an output can reuse the current checkpoint. Enabling lifecycle, lineage, or a full resource after events were skipped requires a checkpoint and ClickHouse data reset or a new exporter ID. To enable projections again, set a new projection replay ID so the processor scans current entries and memories.

Managed and record-only purge modes continue to consume hard-delete notifications for disabled resources. These notifications remove rows written before the output was disabled without writing new lifecycle or current-state rows. Conversation hard deletes also remove dependent entry analytics. External purge mode subscribes only to enabled resources. At least one resource kind must remain enabled.

Tail-only mode deliberately omits retained history and current-state backfill. Use it only for disposable development data when exporting new events is sufficient. Readiness remains false so monitoring continues to report the durability gap.

Checkpoints

ClickHouse connection

Schema manage mode enables a 1,000-block insert deduplication window on exporter output and registered projection tables where deduplication is disabled. It preserves existing positive windows and stored data. validate mode rejects a zero window without applying DDL. Both modes need permission to read the server’s default MergeTree settings. For the bundled analytics user, run this grant when upgrading an existing deployment:

GRANT SELECT ON system.merge_tree_settings TO memory_service_analytics;

Insert tokens protect retries within one processor run. After a restart, new batches have new tokens; use canonical views to deduplicate replayed events.

Native TCP commonly uses port 9000 or TLS port 9440. HTTP commonly uses port 8123 or HTTPS port 8443. The repository’s Docker Compose stack maps the ClickHouse native port to localhost:9002 because MinIO uses host port 9000. Containers in that stack continue to use clickhouse:9000.

Configure either the direct password or the password file. The processor rejects configurations that set both.

Payload and metadata

--metadata-key selects the exact top-level conversation metadata keys that the processor writes to ClickHouse. The admin event stream still sends the complete conversation resource to the processor. Unlisted metadata is discarded before the processor creates ClickHouse rows. This setting does not select entry content or memory values.

The processor stores Memory Service source identifiers directly so an analytics result contains the identifiers needed to retrieve the original resource through the Memory Service API. Treat ClickHouse readers as trusted to see those identifiers. Payload mode and metadata allowlists still control whether content and user-defined metadata cross the analytics boundary.

Projections

Projection paths are valid only in projected or full mode. A path can identify one YAML file or a directory. For a directory, the processor reads the *.yaml and *.yml files directly inside it; it does not scan subdirectories. YAML documents with another kind are ignored, and at least one kind: AnalyticsProjection resource must be found across all configured paths.

Repeat --projection-path to load multiple paths from the CLI. For MEMORY_SERVICE_CLICKHOUSE_PROJECTION_PATH, separate multiple paths with commas. See Create analytics projections for the manifest format.

To check projection paths before you start the processor, run memory-service process clickhouse-projections with the same --projection-path values or MEMORY_SERVICE_CLICKHOUSE_PROJECTION_PATH. It validates the manifests and prints each table’s chain, combined columns, and which manifest’s logic projects each content type or kind. It doesn’t connect to ClickHouse. Add --format json for machine-readable output.

Batching and retries

batch-bytes is the aggregate flush target. A single encoded record may exceed that target up to max-record-bytes, in which case the processor writes it as a one-record batch.

The checkpoint advances only after ClickHouse acknowledges every table write in the frozen batch. If the process stops after writing only part of a batch, replay writes the source events again. Stable event and resource keys keep canonical query results logically idempotent.

Retention and deletion

In managed retention mode, the processor owns table TTL expressions and coordinates the database-wide policy through retention_policy. A zero duration leaves that table class without a TTL. When tombstone retention is positive, generic active-row retention must also be positive and cannot exceed tombstone retention. Projection retention follows the same rule when projections are configured.

In external retention mode, the processor does not register a retention owner, validate TTL expressions, or change TTL DDL. Configure all retention durations as 0; the processor rejects nonzero durations in this mode. Existing TTL expressions remain unchanged. Your ClickHouse administration must replace or remove them and retain deletion tombstones for at least as long as older active rows.

Purge modes handle hard-delete events differently:

External purge must remove matching generic, typed, lifecycle, projection, projection-failure, and deletion-fence rows. Remove each deletion fence only after its dependent rows. Switching from record-only or external back to managed requires a reset or a new exporter ID because the event checkpoint may have passed unprocessed hard deletes.

deletion_fences remain active in every purge mode. The built-in current-state views depend on them to prevent an older active row from becoming visible after a tombstone expires. The common schema also retains empty purge and retention tables so schema validation stays consistent.

Health

For runtime behavior and failure recovery, see Operate the ClickHouse processor.