Observability

Self-hosted Lettuce can export traces and structured logs to any OpenTelemetry-compatible backend over OTLP/gRPC. The hosted SaaS doesn’t use this path — it’s intended for operators who already run an observability stack and want Lettuce signals in the same place as the rest of their services. When OTEL_EXPORTER_OTLP_ENDPOINT is unset, the OTel subsystem is a hard no-op: no SDK code loads, no providers are replaced, zero overhead per request. The same single env var flips both traces and logs on — see Logs configuration below if you want to opt out of just one half of the pipeline.

Install the optional extra

The opentelemetry-* packages are gated behind an optional extra so your image only pulls them if you actually want them:

pip install 'lettuce[otel]'

In a Docker build:

RUN pip install '.[otel]'

Configure the exporter

The standard OpenTelemetry environment variables are honoured — set the endpoint and (optionally) the service name, then restart the service and worker:

Env var	Required	Notes
`OTEL_EXPORTER_OTLP_ENDPOINT`	yes	gRPC endpoint of your OTLP collector. Example: `http://otel-collector:4317`. Unset = OTel disabled.
`OTEL_SERVICE_NAME`	no	Defaults to `lettuce`. Override to differentiate multiple installs.
`OTEL_EXPORTER_OTLP_HEADERS`	depends	Backend-specific auth headers. Honeycomb wants `x-honeycomb-team=...`, New Relic wants `api-key=...`.
`OTEL_RESOURCE_ATTRIBUTES`	no	Free-form `key=value,key2=value2` attached to every span. Useful for `deployment.environment=prod`.

Any of the other OTEL_* vars defined by the OpenTelemetry spec also work — they’re read by the SDK directly, Lettuce doesn’t re-parse them.

What Lettuce emits

Traces (LET-25) and structured logs (LET-40). Metrics beyond the auto-instrumentation defaults are still a roadmap item.

Traces

HTTP server spans — every inbound request to the API / MCP endpoint, with http.method, http.route, http.status_code.
httpx client spans — outbound calls to GitHub, GitLab, Bitbucket, Stripe, OAuth providers. Nested under the inbound request that triggered them.
psycopg SQL spans — every database statement with sanitised query text. The single most useful signal for diagnosing API latency.
Worker job.<kind> spans — one span per claimed background job (job.clone_index, job.embed). Tagged with lettuce.job.id, lettuce.repo.id, lettuce.account.id.

Logs

Every stdlib logging record (INFO and above by default) is shipped over the same OTLP/gRPC endpoint as a LogRecord. Records emitted inside an HTTP request span or a background-job span automatically carry that span’s trace_id and span_id as attributes — which is what makes log lines clickable from Tempo back to Loki / Datadog Logs / New Relic Logs. Resource attributes (service.name, service.version) are shared with the trace pipeline, so the same Grafana / Datadog filter narrows both signals at once.

Logs configuration

Logs ship alongside traces over the same OTEL_EXPORTER_OTLP_ENDPOINT, under a separate OTel signal type (logs) so a collector can fan them out to a different backend than traces if you want (typically: traces → Tempo, logs → Loki).

Env var	Required	Notes
`OTEL_EXPORTER_OTLP_ENDPOINT`	yes	Same endpoint as traces. Unset = OTel disabled entirely.
`OTEL_LOGS_EXPORTER`	no	Set to `none` to disable log shipping while keeping traces (e.g. you already ship logs through promtail or a Datadog agent and don’t want a parallel pipeline). Default = OTLP.
`OTEL_LOG_LEVEL`	no	Minimum stdlib log level to ship. Default `INFO`. `DEBUG` is shippable but very chatty over gRPC.

Trace-log correlation

opentelemetry-instrumentation-logging patches the stdlib log record factory so every record emitted inside an active span picks up the current trace_id and span_id as record attributes. When the OTel Collector forwards those records to Loki, they land as the traceID / spanID labels — which the Grafana “Logs to traces” data-link feature turns into a clickable jump from a Loki log line back to the originating span in Tempo, and vice versa. In Datadog Logs / New Relic Logs the same attributes are surfaced under their respective trace-correlation panels with no extra setup.

Worked example: Loki via OTel Collector

The standard Grafana stack pipeline: Lettuce → OTel Collector → Loki (logs) + Tempo (traces). Lettuce only needs to know about the collector — fan-out is the collector’s job. otel-collector.yaml:

receivers:
  otlp:
    protocols:
      grpc:

exporters:
  otlphttp/tempo:
    endpoint: http://tempo:4318
  otlphttp/loki:
    endpoint: http://loki:3100/otlp

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlphttp/tempo]
    logs:
      receivers: [otlp]
      exporters: [otlphttp/loki]

In Grafana, configure your Loki data source’s “Derived fields” to extract traceID and link it to the Tempo data source. Click the link on any log line and you land on the originating trace.

Worked example: Grafana Tempo + Loki

Point Lettuce at an OpenTelemetry Collector that fans out to Tempo (traces) and Loki (logs). One env var, both signals. docker-compose.yml excerpt:

services:
  lettuce-api:
    image: lettuce/cloud:latest
    environment:
      LETTUCE_SELF_HOSTED: "1"
      OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector:4317"
      OTEL_SERVICE_NAME: "lettuce-api"
      OTEL_RESOURCE_ATTRIBUTES: "deployment.environment=prod"
    depends_on: [otel-collector]

  lettuce-worker:
    image: lettuce/cloud:latest
    command: codewaze-worker
    environment:
      LETTUCE_SELF_HOSTED: "1"
      OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector:4317"
      OTEL_SERVICE_NAME: "lettuce-worker"
    depends_on: [otel-collector]

  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otel-collector.yaml"]
    volumes:
      - ./otel-collector.yaml:/etc/otel-collector.yaml

  tempo:
    image: grafana/tempo:latest
    command: ["-config.file=/etc/tempo.yaml"]

See the Logs configuration section above for the matching otel-collector.yaml that fans out to both Tempo and Loki. Once everything’s up, find a recent slow request in Grafana’s Tempo search; you’ll see the inbound HTTP span at the top, the httpx call to GitHub nested under it, and any SQL statements that ran while serving it. Click any log line tied to that request in Loki and Grafana jumps you straight to the corresponding span in Tempo — that’s the trace_id / span_id correlation working end-to-end.

Other backends

The endpoint is the only thing that changes:

Datadog Agent (OTLP receiver enabled) — http://datadog-agent:4317.
Honeycomb — https://api.honeycomb.io:443 + set OTEL_EXPORTER_OTLP_HEADERS=x-honeycomb-team=<your-key>.
New Relic — https://otlp.nr-data.net:4317 + set OTEL_EXPORTER_OTLP_HEADERS=api-key=<your-key>.

Troubleshooting

Nothing showing up in your backend: confirm the endpoint reachable from inside the container (nc -vz otel-collector 4317). The exporter buffers and retries silently — if the collector is unreachable the worker won’t crash, but you also won’t see traces or logs.
OTEL_EXPORTER_OTLP_ENDPOINT is set but the [otel] extra isn't installed in the logs: rebuild the image with pip install '.[otel]'. The SaaS Dockerfile deliberately omits this extra.
Traces show up but logs don’t: check that OTEL_LOGS_EXPORTER isn’t set to none, and that your collector has a logs pipeline configured (the trace pipeline alone will silently drop log records).
Logs show up but no traceID label in Loki: confirm the log line was actually emitted inside a request or job span — log lines from startup code, scheduled timers etc. legitimately have no trace context. If a log emitted inside a request span also has no traceID, check that opentelemetry-instrumentation-logging is installed (it ships with the [otel] extra by default).
Bad scheme / hostname: the service logs OTLP exporter init failed (or OTel logs setup failed for the logs half) and continues without that signal. Lettuce will never refuse to boot because of an observability-config mistake.

Roadmap

Metrics beyond the auto-instrumentation defaults (active job count, queue depth, embedding throughput) are still a roadmap item — file against the Lettuce issue tracker if that’s blocking.

Documentation Index

​Observability

​Install the optional extra

​Configure the exporter

​What Lettuce emits

​Traces

​Logs

​Logs configuration

​Trace-log correlation

​Worked example: Loki via OTel Collector

​Worked example: Grafana Tempo + Loki

​Other backends

​Troubleshooting

​Roadmap