> ## Documentation Index
> Fetch the complete documentation index at: https://www.edgee.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Custom Observability Destinations
> Forward hosted-gateway usage logs to your own observability endpoint, in addition to Edgee's internal pipeline.
Send a copy of your usage logs to an observability endpoint or datastore you control, alongside
Edgee's own pipeline.
Available on hosted gateways only.
☁️ Self-hosted deployments configure their own OTLP usage-export target directly.
See [Kubernetes & Helm](/docs/self-hosting/kubernetes-helm#common-values).
## Supported protocols
| Protocol | Transport | Status |
| ---------- | -------------------------------------------------------- | ------ |
| OTLP | gRPC or HTTP | ✅ |
| S3 | Batched NDJSON | ✅ |
| ClickHouse | Native protocol, batched `INSERT ... FORMAT JSONEachRow` | ✅ |
| BigQuery | Streaming inserts | ✅ |
| Snowflake | SQL REST API, batched INSERT | ✅ |
## Setup
From **Org settings → Integrations** (Infrastructure group), add one or more destinations. Pick a
destination type, then fill in its fields.
### OTLP
| Field | Purpose |
| -------- | ------------------------------------------------------------- |
| Name | Label shown in the destinations table |
| Protocol | gRPC or HTTP |
| URL | Your endpoint |
| Headers | Custom key/value pairs, e.g. a bearer token for auth |
| Active | Toggles forwarding on or off without deleting the destination |
### S3
| Field | Purpose |
| --------------------------------- | --------------------------------------------------------------------------------- |
| Name | Label shown in the destinations table |
| Bucket | Destination bucket name |
| Region | Bucket region |
| Endpoint | Only needed for an S3-compatible provider other than AWS (e.g. MinIO, R2, Wasabi) |
| Key prefix | Optional path prefix applied to every object written |
| Access key ID / Secret access key | Static credentials with write access to the bucket |
| Active | Toggles forwarding on or off without deleting the destination |
### ClickHouse
| Field | Purpose |
| ------------------- | --------------------------------------------------------------- |
| Name | Label shown in the destinations table |
| Host | Your ClickHouse server hostname |
| Port | Optional; defaults to 9440 (TLS) or 9000 (no TLS) if left blank |
| Database | Destination database name |
| Table | Destination table name (must already exist) |
| Username / Password | Credentials with insert access to the table |
| Secure | Toggles TLS for the connection (on by default) |
| Active | Toggles forwarding on or off without deleting the destination |
The table must already exist before you add the destination: Edgee never creates or migrates
schema on your instance. Every row is sent via `INSERT ... FORMAT JSONEachRow`, so the table needs a
column for each field Edgee sends, with matching names and compatible types.
```sql theme={"dark"}
CREATE TABLE IF NOT EXISTS your_database.usage_logs
(
timestamp DateTime64(9),
request_id String DEFAULT '',
provider String,
model String,
input_tokens UInt32,
output_tokens UInt32,
cached_input_tokens UInt32,
cache_creation_input_tokens UInt32 DEFAULT 0,
reasoning_output_tokens UInt32,
input_cost UInt64,
output_cost UInt64,
cached_input_cost UInt64,
cache_creation_input_cost UInt64 DEFAULT 0,
reasoning_output_cost UInt64,
edgee_tools_cost UInt64 DEFAULT 0,
total_cost UInt64,
duration_ms UInt128,
key_uuid String,
organization_uuid String,
billable Bool DEFAULT true,
has_edgee_tools Bool DEFAULT false,
edgee_tools_count UInt32 DEFAULT 0,
tags Array(String) DEFAULT [],
dc_name String DEFAULT '',
ttft_ms UInt64 DEFAULT 0,
network_latency_ms UInt64 DEFAULT 0,
error Bool DEFAULT false,
error_type String DEFAULT '',
error_message String DEFAULT '',
streaming Bool DEFAULT true,
compression_enabled Bool DEFAULT false,
brevity_enabled Bool DEFAULT false,
brevity_rate Float64 DEFAULT 0,
compression_time_ms Nullable(UInt64),
compression_rate Nullable(Float64),
uncompressed_input_tokens Nullable(UInt32),
compressed_input_tokens Nullable(UInt32),
compression_error Nullable(String),
compression_fell_back_to_original Bool DEFAULT false,
token_cost_savings UInt64 DEFAULT 0,
debug_enabled Bool DEFAULT false,
session_id String DEFAULT '',
compression_tool_stats JSON,
uncompressed_tools_tokens Nullable(UInt32),
compressed_tools_tokens Nullable(UInt32),
user_id String DEFAULT '',
squad_id String DEFAULT '',
mcp_surface_stats JSON,
optimization_stats JSON,
is_byok Bool DEFAULT false,
is_api_billing Bool DEFAULT false,
is_subscription_billing Bool DEFAULT false,
is_fallback Bool DEFAULT false,
is_reroute Bool DEFAULT false,
is_plan_fallback Bool DEFAULT false
)
ENGINE = MergeTree()
PARTITION BY toYYYYMMDD(timestamp)
ORDER BY (organization_uuid, key_uuid, timestamp, provider, model)
COMMENT 'Edgee Usage Logs v1';
```
This is the exact schema Edgee's own internal ClickHouse pipeline uses, so you can copy it as-is.
Adapt the `ENGINE`/`PARTITION BY`/`ORDER BY` clauses to your own retention and query patterns as
long as every column above stays present.
Usage logs are batched in memory per destination and flushed every 60 seconds, or sooner if the
buffer grows past 8 MB. Each flush is one write, not one per log record. For S3, that write is one
NDJSON object with one usage-log row per line; for ClickHouse, one `INSERT ... FORMAT JSONEachRow`
into your configured table.
S3 object keys are laid out as `/org=/date=/hour=/.jsonl`,
the partitioning layout most S3-based query engines (Athena, BigQuery external tables, DuckDB)
expect out of the box.
### BigQuery
| Field | Purpose |
| ------------------- | ----------------------------------------------------------------------------------------- |
| Name | Label shown in the destinations table |
| Project ID | GCP project that owns the destination dataset |
| Dataset ID | BigQuery dataset the usage rows are written into |
| Table ID | Optional; defaults to `usage_logs` when left blank |
| Service account key | A GCP service-account key (JSON) with the BigQuery Data Editor role on the target dataset |
| Active | Toggles forwarding on or off without deleting the destination |
Usage rows are buffered in memory per destination and written through BigQuery streaming inserts
in batches, every 60 seconds or sooner once 5,000 rows accumulate. Each row carries the same
fields Edgee's own ClickHouse pipeline stores. The destination table is created automatically on
first delivery if it does not exist yet, provided the service account has the BigQuery Table Data
Editor role; a pre-created table works too.
### Snowflake
| Field | Purpose |
| --------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| Name | Label shown in the destinations table |
| Account | Snowflake account identifier, e.g. `myorg-myaccount` |
| Token | A [Programmatic Access Token](https://docs.snowflake.com/en/user-guide/programmatic-access-tokens) issued for a user with write access |
| Database | Optional; falls back to the token user's default database |
| Schema | Optional; falls back to the token user's default schema |
| Table | Optional; defaults to `EDGEE_USAGE_LOGS_V1` |
| Warehouse | Optional; falls back to the token user's default warehouse |
| Active | Toggles forwarding on or off without deleting the destination |
Before saving, generate a Programmatic Access Token for the user this destination writes as
(`ALTER USER ADD PROGRAMMATIC ACCESS TOKEN `; the secret is shown once, so capture it
immediately), and attach a network policy to that user. Snowflake refuses PAT authentication
without one. A "Test Connection" that fails with `Network policy is required` is failing on this
check. An allow-all policy satisfies the requirement:
```sql theme={"dark"}
CREATE NETWORK POLICY edgee_pat ALLOWED_IP_LIST = ('0.0.0.0/0');
ALTER USER SET NETWORK_POLICY = edgee_pat;
```
Narrow `ALLOWED_IP_LIST` once you know which source IPs the traffic should come from.
`ALLOWED_IP_LIST` takes IPv4 addresses only. If your account enforces IPv6, add the IPv6
counterpart with `SET ALLOWED_IP6_LIST = ('::/0')`. Both statements need a role holding the
network-policy privileges, typically `ACCOUNTADMIN`.
Rows are delivered over Snowflake's SQL REST API rather than a driver connection, so there is no
username field: the token resolves its own user. If the target table doesn't already exist, Edgee
creates it automatically the first time it flushes, with this shape: one typed column per
usage-log field, the same fields the OTLP and S3 destinations carry.
```sql theme={"dark"}
CREATE TABLE IF NOT EXISTS EDGEE_USAGE_LOGS_V1 (
timestamp TIMESTAMP_NTZ,
request_id STRING,
provider STRING,
model STRING,
streaming BOOLEAN,
input_tokens NUMBER,
output_tokens NUMBER,
cached_input_tokens NUMBER,
cache_creation_input_tokens NUMBER,
reasoning_output_tokens NUMBER,
input_cost NUMBER,
output_cost NUMBER,
cached_input_cost NUMBER,
cache_creation_input_cost NUMBER,
reasoning_output_cost NUMBER,
edgee_tools_cost NUMBER,
total_cost NUMBER,
key_uuid STRING,
organization_uuid STRING,
billable BOOLEAN,
is_byok BOOLEAN,
is_api_billing BOOLEAN,
is_subscription_billing BOOLEAN,
is_fallback BOOLEAN,
is_reroute BOOLEAN,
is_plan_fallback BOOLEAN,
has_edgee_tools BOOLEAN,
edgee_tools_count NUMBER,
tags VARIANT,
dc_name STRING,
network_latency_ms NUMBER,
ttft_ms NUMBER,
duration_ms NUMBER,
debug_enabled BOOLEAN,
compression_enabled BOOLEAN,
brevity_enabled BOOLEAN,
brevity_rate FLOAT,
compression_time_ms NUMBER,
compression_rate FLOAT,
uncompressed_input_tokens NUMBER,
compressed_input_tokens NUMBER,
compression_error STRING,
compression_fell_back_to_original BOOLEAN,
token_cost_savings NUMBER,
compression_tool_stats VARIANT,
uncompressed_tools_tokens NUMBER,
compressed_tools_tokens NUMBER,
mcp_surface_stats VARIANT,
optimization_stats VARIANT,
error BOOLEAN,
error_type STRING,
error_message STRING,
session_id STRING,
user_id STRING,
squad_id STRING
);
```
Four columns (`tags`, `compression_tool_stats`, `mcp_surface_stats`, `optimization_stats`) are
`VARIANT` rather than a scalar type, since each holds a list or a nested object, the standard
Snowflake choice for that shape. Query them with
Snowflake's [dot/bracket notation](https://docs.snowflake.com/en/user-guide/querying-semistructured)
or `FLATTEN`, e.g. `SELECT tags[0]::string FROM EDGEE_USAGE_LOGS_V1`. Every other column is a plain
scalar you can `SELECT` directly, e.g. `SELECT model, total_cost FROM EDGEE_USAGE_LOGS_V1`.
The `_V1` suffix is a schema version. It applies only to the auto-created default table, not to
the name you pass in the **Table** field. If a future change needs a different column shape, it
ships as `EDGEE_USAGE_LOGS_V2` rather than altering `_V1` in place, so existing rows and any views
or pipes built on top of this table keep working unchanged. If you'd rather pre-create the table
yourself (e.g. under a role that doesn't grant `CREATE TABLE`), run the statement above against
your own table name and pass it in the **Table** field.
Usage logs are batched in memory per destination and flushed as one multi-row `INSERT` every 60
seconds, or sooner once 1,000 rows are buffered, delivered in chunks of 50 rows per statement.
Larger chunks would put too many bound parameters in a single statement, since every usage-log
field is its own column.
You can configure more than one destination. Each active one gets its own copy of the logs.
## Test before saving
Run two checks against the form before saving:
* **Test Connection**: checks reachability only (a TCP/TLS handshake for OTLP, a bucket `HEAD` for
S3, a ping for ClickHouse, a dataset lookup for BigQuery, a `SELECT 1` probe for Snowflake).
* **Send Log**: sends one real log record through the full path, in the destination's protocol, so
you can confirm your endpoint or bucket actually receives it.
## Security
Secret values (OTLP header values, an S3 destination's secret access key, a ClickHouse
destination's password, a BigQuery destination's service account key, a Snowflake destination's
Programmatic Access Token) are encrypted at rest and masked (`********`) on every read, the same
handling Edgee uses for [BYOK](/docs/features/byok) provider keys. Plaintext values are never returned
once saved. Editing a destination requires re-entering its secret, since saving fully replaces the
stored config.
## Delivery semantics
* Usage logs keep reaching Edgee's internal pipeline unchanged, regardless of what you configure here.
* Forwarding is additive and asynchronous: a slow or failing destination never blocks or delays
internal delivery, and never affects your other destinations.
* The feature is entirely opt-in: no destination configured means no behavior change and no extra calls.