> ## Documentation Index
> Fetch the complete documentation index at: https://recipe.uselettuce.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability (OpenTelemetry)

> Pipe Lettuce traces and structured logs into your existing Datadog, Grafana, New Relic, or Honeycomb stack via OTLP.

# Observability

Self-hosted Lettuce can export **traces** and **structured logs** to
any OpenTelemetry-compatible backend over OTLP/gRPC. The hosted SaaS
doesn't use this path — it's intended for operators who already run
an observability stack and want Lettuce signals in the same place as
the rest of their services.

When `OTEL_EXPORTER_OTLP_ENDPOINT` is unset, the OTel subsystem is a
hard no-op: no SDK code loads, no providers are replaced, zero
overhead per request. The same single env var flips both traces and
logs on — see [Logs configuration](#logs-configuration) below if you
want to opt out of just one half of the pipeline.

## Install the optional extra

The `opentelemetry-*` packages are gated behind an optional extra so
your image only pulls them if you actually want them:

```bash theme={null}
pip install 'lettuce[otel]'
```

In a Docker build:

```dockerfile theme={null}
RUN pip install '.[otel]'
```

## Configure the exporter

The standard OpenTelemetry environment variables are honoured — set
the endpoint and (optionally) the service name, then restart the
service and worker:

| Env var                       | Required | Notes                                                                                                 |
| ----------------------------- | -------- | ----------------------------------------------------------------------------------------------------- |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | yes      | gRPC endpoint of your OTLP collector. Example: `http://otel-collector:4317`. Unset = OTel disabled.   |
| `OTEL_SERVICE_NAME`           | no       | Defaults to `lettuce`. Override to differentiate multiple installs.                                   |
| `OTEL_EXPORTER_OTLP_HEADERS`  | depends  | Backend-specific auth headers. Honeycomb wants `x-honeycomb-team=...`, New Relic wants `api-key=...`. |
| `OTEL_RESOURCE_ATTRIBUTES`    | no       | Free-form `key=value,key2=value2` attached to every span. Useful for `deployment.environment=prod`.   |

Any of the other `OTEL_*` vars defined by the OpenTelemetry spec also
work — they're read by the SDK directly, Lettuce doesn't re-parse
them.

## What Lettuce emits

Traces (LET-25) and structured logs (LET-40). Metrics beyond the
auto-instrumentation defaults are still a roadmap item.

### Traces

* **HTTP server spans** — every inbound request to the API / MCP
  endpoint, with `http.method`, `http.route`, `http.status_code`.
* **`httpx` client spans** — outbound calls to GitHub, GitLab,
  Bitbucket, Stripe, OAuth providers. Nested under the inbound
  request that triggered them.
* **`psycopg` SQL spans** — every database statement with sanitised
  query text. The single most useful signal for diagnosing API
  latency.
* **Worker `job.<kind>` spans** — one span per claimed background
  job (`job.clone_index`, `job.embed`). Tagged with `lettuce.job.id`,
  `lettuce.repo.id`, `lettuce.account.id`.

### Logs

Every stdlib `logging` record (INFO and above by default) is shipped
over the same OTLP/gRPC endpoint as a `LogRecord`. Records emitted
inside an HTTP request span or a background-job span automatically
carry that span's `trace_id` and `span_id` as attributes — which is
what makes log lines clickable from Tempo back to Loki / Datadog
Logs / New Relic Logs. Resource attributes (`service.name`,
`service.version`) are shared with the trace pipeline, so the same
Grafana / Datadog filter narrows both signals at once.

## Logs configuration

Logs ship alongside traces over the same `OTEL_EXPORTER_OTLP_ENDPOINT`,
under a separate OTel signal type (`logs`) so a collector can fan
them out to a different backend than traces if you want (typically:
traces → Tempo, logs → Loki).

| Env var                       | Required | Notes                                                                                                                                                                           |
| ----------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | yes      | Same endpoint as traces. Unset = OTel disabled entirely.                                                                                                                        |
| `OTEL_LOGS_EXPORTER`          | no       | Set to `none` to disable log shipping while keeping traces (e.g. you already ship logs through promtail or a Datadog agent and don't want a parallel pipeline). Default = OTLP. |
| `OTEL_LOG_LEVEL`              | no       | Minimum stdlib log level to ship. Default `INFO`. `DEBUG` is shippable but very chatty over gRPC.                                                                               |

### Trace-log correlation

`opentelemetry-instrumentation-logging` patches the stdlib log
record factory so every record emitted inside an active span picks
up the current `trace_id` and `span_id` as record attributes. When
the OTel Collector forwards those records to Loki, they land as the
`traceID` / `spanID` labels — which the Grafana "Logs to traces"
data-link feature turns into a clickable jump from a Loki log line
back to the originating span in Tempo, and vice versa.

In Datadog Logs / New Relic Logs the same attributes are surfaced
under their respective trace-correlation panels with no extra setup.

### Worked example: Loki via OTel Collector

The standard Grafana stack pipeline: Lettuce → OTel Collector →
Loki (logs) + Tempo (traces). Lettuce only needs to know about the
collector — fan-out is the collector's job.

`otel-collector.yaml`:

```yaml theme={null}
receivers:
  otlp:
    protocols:
      grpc:

exporters:
  otlphttp/tempo:
    endpoint: http://tempo:4318
  otlphttp/loki:
    endpoint: http://loki:3100/otlp

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlphttp/tempo]
    logs:
      receivers: [otlp]
      exporters: [otlphttp/loki]
```

In Grafana, configure your Loki data source's "Derived fields" to
extract `traceID` and link it to the Tempo data source. Click the
link on any log line and you land on the originating trace.

## Worked example: Grafana Tempo + Loki

Point Lettuce at an OpenTelemetry Collector that fans out to Tempo
(traces) and Loki (logs). One env var, both signals.

`docker-compose.yml` excerpt:

```yaml theme={null}
services:
  lettuce-api:
    image: lettuce/cloud:latest
    environment:
      LETTUCE_SELF_HOSTED: "1"
      OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector:4317"
      OTEL_SERVICE_NAME: "lettuce-api"
      OTEL_RESOURCE_ATTRIBUTES: "deployment.environment=prod"
    depends_on: [otel-collector]

  lettuce-worker:
    image: lettuce/cloud:latest
    command: codewaze-worker
    environment:
      LETTUCE_SELF_HOSTED: "1"
      OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector:4317"
      OTEL_SERVICE_NAME: "lettuce-worker"
    depends_on: [otel-collector]

  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otel-collector.yaml"]
    volumes:
      - ./otel-collector.yaml:/etc/otel-collector.yaml

  tempo:
    image: grafana/tempo:latest
    command: ["-config.file=/etc/tempo.yaml"]
```

See the [Logs configuration](#logs-configuration) section above for
the matching `otel-collector.yaml` that fans out to both Tempo and
Loki.

Once everything's up, find a recent slow request in Grafana's Tempo
search; you'll see the inbound HTTP span at the top, the `httpx` call
to GitHub nested under it, and any SQL statements that ran while
serving it. Click any log line tied to that request in Loki and
Grafana jumps you straight to the corresponding span in Tempo —
that's the `trace_id` / `span_id` correlation working end-to-end.

## Other backends

The endpoint is the only thing that changes:

* **Datadog Agent (OTLP receiver enabled)** — `http://datadog-agent:4317`.
* **Honeycomb** — `https://api.honeycomb.io:443` + set
  `OTEL_EXPORTER_OTLP_HEADERS=x-honeycomb-team=<your-key>`.
* **New Relic** — `https://otlp.nr-data.net:4317` + set
  `OTEL_EXPORTER_OTLP_HEADERS=api-key=<your-key>`.

## Troubleshooting

* **Nothing showing up in your backend**: confirm the endpoint
  reachable from inside the container (`nc -vz otel-collector 4317`).
  The exporter buffers and retries silently — if the collector is
  unreachable the worker won't crash, but you also won't see traces
  or logs.
* **`OTEL_EXPORTER_OTLP_ENDPOINT is set but the [otel] extra isn't
  installed`** in the logs: rebuild the image with
  `pip install '.[otel]'`. The SaaS Dockerfile deliberately omits
  this extra.
* **Traces show up but logs don't**: check that `OTEL_LOGS_EXPORTER`
  isn't set to `none`, and that your collector has a `logs` pipeline
  configured (the trace pipeline alone will silently drop log
  records).
* **Logs show up but no `traceID` label in Loki**: confirm the log
  line was actually emitted *inside* a request or job span — log
  lines from startup code, scheduled timers etc. legitimately have
  no trace context. If a log emitted inside a request span also has
  no `traceID`, check that `opentelemetry-instrumentation-logging`
  is installed (it ships with the `[otel]` extra by default).
* **Bad scheme / hostname**: the service logs `OTLP exporter init
  failed` (or `OTel logs setup failed` for the logs half) and
  continues without that signal. Lettuce will never refuse to boot
  because of an observability-config mistake.

## Roadmap

Metrics beyond the auto-instrumentation defaults (active job count,
queue depth, embedding throughput) are still a roadmap item — file
against the Lettuce issue tracker if that's blocking.