> ## Documentation Index
> Fetch the complete documentation index at: https://recipe.uselettuce.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Self-hosted install

> Run Lettuce inside your own infrastructure with Docker Compose or Kubernetes (Helm).

# Self-hosted Lettuce — install guide

This page is the runbook a Lettuce operator follows to stand up their own
instance. It is opinionated: when there is a sane default, we pick it.
When the v1 product is missing a piece, we call that out explicitly rather
than pretend it's there.

If you're evaluating self-hosted as a concept, start with
[RFC-001](./RFC-001-self-hosted-offering.md) for the design decisions and
[RFC-002](./RFC-002-compilation-toolchain.md) for how the binaries are
built. This page assumes you've already decided to install.

<Note>
  **Status:** v0 install path. The Docker Compose flow is fully working end
  to end; the Helm chart is a skeleton (see §4). If you hit something this
  guide doesn't cover, write to **[founders@uselettuce.dev](mailto:founders@uselettuce.dev)** — we'll help
  you finish the install and feed the gap back into this page.
</Note>

## 0. Quick start with `lettuce-init`

If you'd rather not hand-edit YAML, the `lettuce-init` CLI (LET-31)
turns a license key into a working install layout in one command:

```bash theme={null}
# Validate the license file sales sent you.
lettuce-init validate --license-key-file ~/lettuce.key

# Docker Compose: writes .env / docker-compose.yml / public.pem / README.md.
# Add `--compiled-worker` to use the Nuitka-compiled worker image
# (LET-34) — requires ghcr.io/uselettuce/worker-compiled to exist.
lettuce-init compose \
  --license-key-file ~/lettuce.key \
  --out-dir ./lettuce \
  --version v1.0.0 \
  --domain https://lettuce.acme.com

# Kubernetes / Helm: writes my-values.yaml + README.md.
lettuce-init helm \
  --license-key-file ~/lettuce.key \
  --out-dir ./lettuce-values \
  --version v1.0.0 \
  --domain https://lettuce.acme.com
```

What you get:

* `.env` pre-wired with your license, your domain, the right
  `LETTUCE_TELEMETRY_ENABLED=true` (LET-27 caught the naming: it is
  `*_ENABLED`, not `*_DISABLED`), and freshly-minted 32-byte URL-safe
  random secrets for `POSTGRES_PASSWORD`, `CODEWAZE_ADMIN_TOKEN`,
  `JWT_SIGNING_KEY`, and `MINIO_ROOT_PASSWORD`. The CLI refuses to
  overwrite the `.env` without `--force` so a re-run doesn't silently
  rotate secrets out from under a running install.
* `docker-compose.yml` copied from this repo with image tags pinned
  to your `--version`.
* `public.pem` (the bundled dev key by default — swap for your
  per-customer production key before `docker compose up`).
* `README.md` with the exact next-step commands, including the real
  first-admin flow (`POST /api/admin/invites`).

<Note>
  **Honest caveat — image pulls don't work yet.** LET-17 wired the
  GitHub Actions pipeline that *will* publish per-service images to
  `ghcr.io/uselettuce/<svc>` on tag push, but the `uselettuce` GitHub
  org doesn't exist publicly yet. Until then `docker compose pull`
  will fail with `unauthorized` or `manifest not found`. The
  generated README explains the workaround (private registry token
  or local builds). Watch this space — once the org is up the same
  generated layout will pull cleanly without changes.
</Note>

The rest of this page is the long-form runbook for operators who
prefer to do it by hand or who need to deviate from the defaults the
CLI picks. Start at §1 for prerequisites.

## 1. Prerequisites

### Host

Pick one of:

* **Single Linux host with Docker Compose.** Docker Engine 24+ and the
  `docker compose` plugin. Suitable for a pilot, a single-team install,
  or a non-production demo.
* **Kubernetes cluster.** 1.27+ with Helm 3.12+. Suitable for production.

### Sizing (per service node)

| Resource | Minimum              | Recommended |
| -------- | -------------------- | ----------- |
| CPU      | 4 vCPU               | 8 vCPU      |
| RAM      | 8 GB                 | 16 GB       |
| Storage  | 100 GB block storage | 250 GB SSD  |

The 100 GB floor is **shared between Postgres data and the MinIO bucket**
that holds cloned source + graph databases. Repos with a million-plus
lines of code or a long history can push past that — provision with a
2× headroom.

### Data plane

* **Postgres 15+** if you're bringing your own. The Compose stack ships a
  Postgres 16 container; the Helm chart expects a `DATABASE_URL` you
  point at RDS / Cloud SQL / Supabase / a bare instance.
* **No Redis required.** Background jobs run on Postgres `LISTEN/NOTIFY`
  (see `src/codewaze/service/jobs.py`), so the worker and API share one
  database and that's it.
* **Object storage** — S3-compatible. The Compose stack ships MinIO; the
  Helm chart expects you to point `CODEWAZE_S3_*` at AWS S3, Cloudflare
  R2, Backblaze B2, or your own MinIO/Ceph deployment.

### Network

Lettuce needs a reachable **HTTPS** hostname. OAuth + GitHub-App callback
URLs come back over HTTPS, so plaintext-only deployments aren't
supported. Terminate TLS in front of the `web` service with whichever
ingress you already use:

* Caddy / Traefik / nginx-ingress with **Let's Encrypt** is the cheap
  default.
* `cert-manager` + ClusterIssuer on Kubernetes is the obvious choice if
  you already run it.

### GitHub integration

You need permission to register a GitHub App on the GitHub instance
Lettuce will index — either:

* **github.com** (free / Pro / Team / Enterprise Cloud), or
* a **GitHub Enterprise Server** instance you operate (LET-20 — see §5).

GitLab self-hosted is supported via `GITLAB_HOST` (LET-21 — see §5).
Bitbucket Data Center is still on the roadmap (LET-15.6) — not yet
in v1.

### License key

Self-hosted Lettuce is license-gated. The worker refuses to boot without
one when `LETTUCE_SELF_HOSTED=1`. Get yours by writing to
**[sales@uselettuce.dev](mailto:sales@uselettuce.dev)** or via the contact form on
[uselettuce.dev/self-hosted](https://uselettuce.dev/self-hosted). The
team will issue a signed Ed25519 JWT with your seat/repo cap and
expiry baked in. Expect a turnaround of one business day.

<Note>
  **Pricing tiers are not yet published.** Per RFC-001 §7, the founder
  hasn't finalised whether the floor is Team-5 or Team-25, or whether
  billing is per-seat / per-LOC / hybrid. For pilots today: contact sales,
  and we'll quote against your team size + repo count.
</Note>

## 2. License & telemetry — read this first

The license and telemetry models cause more confusion than any single
piece of YAML, so we cover them up front.

### What the license does

When `LETTUCE_SELF_HOSTED=1`, the worker and API call `gate_boot_or_raise`
on startup (`src/codewaze/service/license.py`). That:

1. Reads `LETTUCE_LICENSE_KEY` (a JWT) and `LETTUCE_LICENSE_PUBLIC_KEY`
   (the Lettuce Ed25519 public key — bundled in the runtime image at
   `/var/lib/lettuce/public.pem`, default works).
2. Verifies the JWT signature against the public key.
3. Verifies the JWT isn't past its `exp` claim.
4. Writes a cache of the last-good claims to
   `/var/cache/lettuce/license.json` (override with
   `LETTUCE_LICENSE_CACHE_PATH`).

If verification fails for any reason — missing env, bad signature,
expired claim — the service exits. There is no "trial mode" and no
permissive fallback. **A misconfigured worker will not start.** This is
deliberate: an undocumented zero-license install is worse than a noisy
failure.

License claims also carry tier / seats / repo caps that are consumed by
the entitlements layer (`src/codewaze/service/entitlements.py`) when
billing-gated features run. Hitting a cap surfaces in the UI as a
"License limit reached" badge, not a crash.

### What the telemetry receiver sees

Self-hosted instances POST a daily heartbeat to
`https://telemetry.uselettuce.dev/v1/self-hosted/telemetry`. The payload
is **exactly** these fields:

```json theme={null}
{
  "instance_id": "f1d8…-uuid",
  "license_id": "lic_…",
  "version": "v1.0.0",
  "repo_count": 47,
  "user_count": 12,
  "active_users_30d": 9,
  "ts": "2026-05-31T07:00:00Z"
}
```

That's the whole list. No repo names, no source, no user emails, no
queries, no graph contents — see `src/codewaze/service/telemetry.py`
(`parse_ping`) for the schema, and `tests/test_self_hosted_telemetry.py`
for the contract tests that pin it.

Telemetry is **on by default** unless your license's
`telemetry_required` claim is `false` — that's the regulated-customer
escape hatch. The Compose stack and the Helm chart both expose:

| Env var                     | Default                                         | Meaning                                                                                                               |
| --------------------------- | ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| `LETTUCE_TELEMETRY_ENABLED` | `true`                                          | Operator-side switch. Set to `false` (also accepts `0`, `no`, `off`) to suppress emission.[^telemetry-disabled-alias] |
| `LETTUCE_TELEMETRY_URL`     | `https://telemetry.uselettuce.dev/v1/heartbeat` | Override the receiver (rarely useful unless you proxy it).                                                            |

[^telemetry-disabled-alias]: The legacy `LETTUCE_TELEMETRY_DISABLED=1` knob is still honoured as a deprecated alias for one release for backward compatibility; the worker logs a WARN naming the new var the first time it observes the old one set.

<Note>
  **Operator opt-out is only honoured when the license allows it.** If your
  license was issued with `telemetry_required=true` and you flip
  `LETTUCE_TELEMETRY_ENABLED=false`, the worker logs the contradiction and
  keeps pinging. Customers who need an offline-only key should request one
  explicitly from sales — see RFC-001 §3.
</Note>

<Note>
  **Sender path is shipping incrementally.** LET-30 landed the **receiver**
  endpoint + ops dashboard. The instance-side daily sender lands in
  LET-30.1 — until then `LETTUCE_TELEMETRY_ENABLED` is wired through env
  but the daily POST is gated by that follow-up.
</Note>

## 3. Install — Docker Compose

This is the easy path. One host, one `docker compose up`, \~3 minutes from
clone to a working stack.

<Note>
  **Repo URL is a placeholder.** Until we cut the public install artifact
  (RFC-001 §10 — `lettuce/install` repo + signed tarballs), clone from the
  internal mirror your sales contact shares with you. The path the rest of
  this guide assumes is `coze/self-hosted/`.
</Note>

<Steps>
  <Step title="Clone the install repo">
    ```bash theme={null}
    git clone https://github.com/aviram-wba/coze.git    # placeholder — sales will share the real URL
    cd coze/self-hosted
    ```
  </Step>

  <Step title="Write the .env file">
    ```bash theme={null}
    cp .env.example .env
    $EDITOR .env
    ```

    At minimum fill in:

    ```ini theme={null}
    # Image version. Pin to a release tag in production.
    LETTUCE_VERSION=v1.0.0

    # License JWT issued by sales.
    LETTUCE_LICENSE_KEY=eyJ0eXAi...

    # Admin token gates /admin/*. openssl rand -hex 16.
    CODEWAZE_ADMIN_TOKEN=...

    # Public URLs Lettuce will redirect to on OAuth + GitHub install.
    # LETTUCE_WEB_URL must be HTTPS in production.
    LETTUCE_WEB_URL=https://lettuce.acme.com
    LETTUCE_MCP_URL=https://mcp.lettuce.acme.com   # optional split; defaults to LETTUCE_WEB_URL/mcp

    # Postgres password if you're using the bundled DB.
    POSTGRES_PASSWORD=...

    # MinIO root creds for the bundled object store.
    MINIO_ROOT_USER=lettuce
    MINIO_ROOT_PASSWORD=...
    ```

    The image already bundles the Ed25519 public key at
    `/var/lib/lettuce/public.pem`, so the default works out of the box.
    Override `LETTUCE_LICENSE_PUBLIC_KEY` only if you're testing with a
    custom keypair (see `self-hosted/README.md`).

    For GitHub integration, also fill the four `GITHUB_APP_*` vars. If
    you're targeting GitHub Enterprise Server, set `GITHUB_HOST` to your
    GHES hostname (and `NEXT_PUBLIC_GITHUB_HOST` for the web bundle —
    see §5).
  </Step>

  <Step title="Set LETTUCE_SELF_HOSTED">
    The bundled `docker-compose.yml` already sets `LETTUCE_SELF_HOSTED=true`
    on `api` and `worker`. Don't unset it — entitlements + license gating
    are wired off this single flag.
  </Step>

  <Step title="Start the stack">
    ```bash theme={null}
    docker compose up -d
    ```

    Compose pulls `ghcr.io/uselettuce/{api,worker,web}:${LETTUCE_VERSION}`
    (LET-17). First boot takes \~30s while Postgres comes up and the
    idempotent schema applies.
  </Step>

  <Step title="Verify health">
    ```bash theme={null}
    docker compose ps                             # api / worker / web should be 'healthy' or 'running'
    curl -fsS https://lettuce.acme.com/healthz    # expect {"status":"ok"}
    docker compose logs api | grep "license verified"   # confirms LET-29 boot gate passed
    ```

    If the worker is restart-looping, `docker compose logs worker` will
    show one of the `LicenseError` subclasses (see §11).
  </Step>

  <Step title="Create the first user">
    The first-admin workflow runs **through the API**, not a dedicated
    `lettuce admin create` command (no such command exists today — see
    the caveat in the closing notes). Mint an invite with your admin
    token, then redeem it through the web signup form:

    ```bash theme={null}
    # Mint an invite code (admin only)
    curl -fsS -X POST https://lettuce.acme.com/api/admin/invites \
      -H "Authorization: Bearer $CODEWAZE_ADMIN_TOKEN" \
      -H "Content-Type: application/json" \
      -d '{"note": "first admin"}'
    # → returns {"code": "INV-...", ...}

    # Open the web app and sign up with that code:
    open https://lettuce.acme.com/signup
    ```

    The first signup creates an account; subsequent signups against the
    same invite code join it as teammates. Generate per-user invites
    afterwards from the **Team** page in the dashboard.
  </Step>
</Steps>

## 4. Install — Kubernetes (Helm)

```bash theme={null}
helm upgrade --install lettuce ./self-hosted/helm/lettuce \
  -n lettuce --create-namespace \
  -f my-values.yaml
```

Minimum `my-values.yaml`:

```yaml theme={null}
image:
  registry: ghcr.io/uselettuce
  tag: "v1.0.0"

license:
  # Either inline (dev only)…
  key: "eyJ0eXAi..."
  # …or reference a Secret with a `license-key` data field (recommended).
  existingSecret: lettuce-license

# Bring your own Postgres. Don't run Postgres in-cluster for prod.
postgres:
  external:
    url: "postgresql://lettuce:...@db.acme.local:5432/lettuce"

# S3-compatible object storage. AWS, R2, on-prem MinIO — all fine.
s3:
  external:
    bucket: lettuce-graphs
    endpointUrl: ""                       # leave empty for AWS S3
    region: us-east-1
    accessKeyExistingSecret: lettuce-s3   # Secret with AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY

adminToken:
  existingSecret: lettuce-admin           # Secret with `admin-token`

telemetry:
  enabled: true
  url: https://telemetry.uselettuce.dev/v1/heartbeat
```

<Note>
  **The Helm chart is a v0 skeleton.** Read this before you push it at
  prod. Tracked under LET-15.1 / LET-15.x:

  * **No Ingress template.** You have to bring your own Ingress (or
    Gateway API resource) and point it at the `web` and `api` Services.
    TLS termination is on you — `cert-manager` + a ClusterIssuer is the
    obvious answer.
  * **No PVCs.** The `web` and `api` Deployments don't mount persistent
    storage. Postgres is expected externally; MinIO isn't templated. Set
    `postgres.external.url` + `s3.external.*` to real-managed services.
  * **No HPAs.** Single replica per service by default. Scale manually
    via `helm upgrade --set api.replicas=N`.
  * **No probes.** Liveness/readiness probes are not wired. Pods come up
    on container start, not on `/healthz` actually returning 200 — add
    probes via a values override if your cluster needs them.
  * **No NetworkPolicies / ServiceMonitors.** Add your own if your
    cluster requires either.

  These gaps are deliberate for the v0 chart — see RFC-001 §5 and
  `self-hosted/helm/lettuce/values.yaml`. Until the chart graduates,
  expect to maintain a thin overlay (Kustomize or a wrapper chart) that
  adds the missing pieces.
</Note>

## 5. Register your GitHub App

Lettuce talks to GitHub via a **customer-registered** GitHub App — we
do not run a centrally-managed App against your repos (RFC-001 §6).
You register the App on the GitHub instance you want to index, then
paste its credentials into Lettuce.

### github.com

1. Go to
   [github.com/settings/apps/new](https://github.com/settings/apps/new)
   (for an org App, use
   `https://github.com/organizations/<org>/settings/apps/new`).

2. **Homepage URL:** `${LETTUCE_WEB_URL}`.

3. **Callback URL:** `${LETTUCE_WEB_URL}/api/v1/integrations/github/oauth/callback`.

4. **Webhook URL:** leave the App-level webhook empty. Lettuce
   registers a **per-repo** webhook at add-repo time, pointing at
   `${LETTUCE_WEB_URL}/v1/github/webhook/{account_id}` — that's why
   the App needs `Webhooks: Read & Write`.

5. **Permissions:**

   | Scope                      | Access       |
   | -------------------------- | ------------ |
   | Repository — Contents      | Read         |
   | Repository — Metadata      | Read         |
   | Repository — Webhooks      | Read & Write |
   | Repository — Pull requests | Read         |
   | Repository — Issues        | Read         |

6. **Subscribe to events:** `Push`, `Pull request`, `Issues`.

7. **Where can this App be installed:** "Only on this account" is the
   safer default.

8. After creation, generate a private key (PEM) and copy:

   * App ID
   * App slug (the URL segment)
   * OAuth client ID + client secret
   * the PEM contents

   …into the four `GITHUB_APP_*` env vars on `api` + `worker`:

   ```ini theme={null}
   GITHUB_APP_ID=12345
   GITHUB_APP_NAME=lettuce-acme
   GITHUB_APP_CLIENT_ID=Iv1.abc...
   GITHUB_APP_CLIENT_SECRET=...
   GITHUB_APP_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----"
   ```

   Both pods need them — see §11's "App is not configured" footgun.

### GitHub Enterprise Server (LET-20)

Same shape, different host. On your GHES instance, register the App at
`https://<your-ghes>/settings/apps/new` with the same permissions and
events as above, then set:

```ini theme={null}
GITHUB_HOST=github.acme.com
NEXT_PUBLIC_GITHUB_HOST=github.acme.com   # baked into the web bundle at build time
GITHUB_APP_ID=...
GITHUB_APP_NAME=...
GITHUB_APP_CLIENT_ID=...
GITHUB_APP_CLIENT_SECRET=...
GITHUB_APP_PRIVATE_KEY=...
```

`NEXT_PUBLIC_GITHUB_HOST` is a **build-time** variable — if you build
the `web` image yourself, set it as a build arg. The published
`ghcr.io/uselettuce/web` image is baked for `github.com`; GHES customers
who can't rebuild should contact sales for a per-instance build.

Webhook URL on the App is still empty; Lettuce uses the per-repo hook
at `${LETTUCE_WEB_URL}/v1/github/webhook/{account_id}`.

Full GHES runbook lives in
[RFC-001 §10](./RFC-001-self-hosted-offering.md#10-github-enterprise-server-ghes-support-let-20).

### GitLab self-hosted (LET-21)

If you run a self-managed GitLab instance, point Lettuce at it with:

```ini theme={null}
GITLAB_HOST=gitlab.acme.com
# Historical knob, kept as a legacy fallback; GITLAB_HOST wins when both are set.
GITLAB_BASE_URL=https://gitlab.acme.com

GITLAB_APP_CLIENT_ID=...
GITLAB_APP_CLIENT_SECRET=...
```

Register the OAuth application against your GitLab instance under
**Admin → Applications** (system-wide) or **User settings →
Applications** (user-scoped). The callback URL is
`${LETTUCE_WEB_URL}/v1/integrations/gitlab/callback`. The required
scope is `api` (project read + webhook management).

Bitbucket Data Center is on the roadmap (LET-15.6) — not in v1.

## 6. Auth: bring your own IdP (LET-18)

Default is Supabase, which works fine if you don't care. If you have a
corporate IdP — Okta, Auth0, Azure AD, Keycloak, anything that speaks
OIDC — switch to the generic backend:

```ini theme={null}
LETTUCE_AUTH_PROVIDER=oidc

# Either point at the issuer (we derive JWKS from .well-known/openid-configuration)…
OIDC_ISSUER_URL=https://login.acme.com

# …or give us the JWKS URL directly.
OIDC_JWKS_URL=https://login.acme.com/.well-known/jwks.json

# Required. The audience claim the JWT must carry.
OIDC_AUDIENCE=lettuce

# Optional. Default 300s.
OIDC_JWKS_CACHE_TTL_SECONDS=300
```

The backend re-verifies every inbound user JWT against the JWKS, with
cache + `kid`-based refresh on cache miss. See
`src/codewaze/service/auth/oidc.py` for the spec.

<Note>
  The **web UI** still expects Supabase login flows today. The OIDC
  backend lets your API accept tokens minted by your IdP (useful for
  agents and machine-to-machine), but the human-facing sign-in form is
  not yet wired to OIDC. Tracked separately — file against LET-15.2 if
  this blocks you.
</Note>

## 7. Secrets — bring your own secret store (LET-23)

By default Lettuce reads secrets straight from env vars
(`LETTUCE_SECRET_STORE=env`). That's fine if your orchestrator already
injects secrets at runtime (Kubernetes Secrets, Compose `env_file`,
SOPS-encrypted overlays, etc).

If you'd rather pull secrets at boot from HashiCorp Vault, switch:

```ini theme={null}
LETTUCE_SECRET_STORE=vault

VAULT_ADDR=https://vault.acme.local
LETTUCE_VAULT_MOUNT=secret                # KV v2 mount

# Auth — one of:
VAULT_TOKEN=hvs.…
# or AppRole:
VAULT_ROLE_ID=...
VAULT_SECRET_ID=...
```

Lettuce expects each secret to live at `<mount>/data/lettuce/<name>` —
the secret store reads keys lazily by their env-var name.

**AWS Secrets Manager** (LET-32) is also wired:

```ini theme={null}
LETTUCE_SECRET_STORE=aws

# Either set the explicit override…
LETTUCE_AWS_SECRETS_REGION=us-east-1
# …or rely on AWS_REGION + the default boto3 credential chain
# (env, shared profile, IAM role on EC2/ECS/EKS).
```

Each secret-name lookup hits `secretsmanager.GetSecretValue` with the
literal env-var name as the `SecretId`. Rotate by updating the value
in Secrets Manager — Lettuce reads through on every fetch (no
process-level cache).

**GCP Secret Manager** is still pending as LET-33; the `aws`-style
switch (`LETTUCE_SECRET_STORE=gcp`) will land with that issue.

## 8. Observability (LET-25)

Lettuce ships OpenTelemetry tracing for HTTP, outbound httpx, and
`psycopg` SQL spans plus a per-job `job.<kind>` span on the worker.
Point at any OTLP-compatible backend:

```ini theme={null}
OTEL_EXPORTER_OTLP_ENDPOINT=https://tempo.acme.com:4317
OTEL_SERVICE_NAME=lettuce-api
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod
```

When the endpoint is unset, the OTel subsystem is a hard no-op — no
SDK loads, zero overhead. Full reference + worked Grafana/Tempo example
in [Observability](./observability.mdx).

## 9. Supply chain (LET-37)

Every image published to `ghcr.io/uselettuce/*` on a tag build is
signed with **cosign keyless** (Sigstore OIDC — no private keys held
by Lettuce), ships an **SPDX-JSON SBOM** produced by `syft`, and is
scanned by `grype`. This is what unlocks the SOC 2 supply-chain
control, FedRAMP SR-3, SLSA, and most CISO-checklist questions.

<Note>
  **Not yet exercised end-to-end.** The signing / SBOM / scan steps live
  in `.github/workflows/{compiled-image,publish-images}.yml` but
  `ghcr.io/uselettuce` does not exist yet (per the LET-17 note); the
  first real run is on the next tag push after the org is created.
</Note>

### Verify the image signature

Install [cosign](https://docs.sigstore.dev/cosign/installation/) (`brew
install cosign` or grab the static binary), then verify any image
Lettuce publishes:

```bash theme={null}
cosign verify \
  --certificate-identity-regexp '^https://github\.com/aviram-wba/coze/' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  ghcr.io/uselettuce/api:vX.Y.Z
```

Substitute `api` for `worker`, `web`, `api-compiled`, `worker-compiled`,
or `mcp-compiled` (LET-35 — the stdio MCP server + local-dev CLI image,
intended for the developer's machine, not the Lettuce server). A successful
verify prints the Rekor public-log entry plus the workflow identity
that signed it; a failure exits non-zero. The
`--certificate-identity-regexp` pins the signer to a GitHub Actions
workflow under `aviram-wba/coze` — change that prefix if you fork.

### Fetch the SBOM

Two equivalent paths — pick whichever your tooling prefers:

1. **Workflow artifact.** Each tag build uploads `sbom-<svc>.spdx.json`
   as an `actions/upload-artifact` artifact named
   `sbom-<svc>-<version>` (90-day retention). Grab it from the run
   page on GitHub, or with `gh run download`.
2. **Cosign attestation.** The same SBOM is attested to the image
   under the SPDX predicate type, so it lives next to the image in
   the registry:

   ```bash theme={null}
   cosign download attestation \
     --predicate-type https://spdx.dev/Document \
     ghcr.io/uselettuce/api:vX.Y.Z \
     | jq -r '.payload | @base64d | fromjson | .predicate' \
     > api.spdx.json
   ```

   `cosign verify-attestation` against the same image digest will
   prove the SBOM was signed by the same workflow identity that
   signed the image.

### Vulnerability scan results

`grype` runs on every tag build with `severity-cutoff: high`, and its
SARIF is uploaded to GitHub code-scanning — findings land in the
**Security → Code scanning alerts** tab of the published repo, tagged
`grype-<svc>`. The scan is **non-blocking on purpose**: a brand-new
CVE disclosed five minutes before a release should not stop the
release, only inform whether to ship + patch or roll back. Operators
who want to gate their own deploys can fail their pipeline on the
`grype` SARIF directly.

## 10. Upgrade

### Rolling (Kubernetes)

```bash theme={null}
helm upgrade lettuce ./self-hosted/helm/lettuce \
  -n lettuce -f my-values.yaml \
  --set image.tag=v1.1.0
```

The Deployments rolling-restart their pods. In-flight worker jobs run
to completion before SIGTERM (the worker's poll loop wraps each claim
in a graceful-shutdown context).

### Blue/green (Compose)

```bash theme={null}
# Spin a parallel stack against the same Postgres / MinIO.
cp -r self-hosted self-hosted-next
sed -i 's/LETTUCE_VERSION=v1.0.0/LETTUCE_VERSION=v1.1.0/' self-hosted-next/.env
docker compose -f self-hosted-next/docker-compose.yml -p lettuce-next up -d

# Verify on the new stack, then flip the ingress to point at lettuce-next.
# Once the old stack has drained:
docker compose -f self-hosted/docker-compose.yml down
```

### Migrations

The API auto-applies the idempotent `SCHEMA: list[str]` block from
`src/codewaze/service/db.py` at boot. New releases extend that list —
the boot path runs each statement under `IF NOT EXISTS` / `ALTER … IF
NOT EXISTS` guards, so re-running is safe.

Destructive migrations are not in v0. If a future release ever requires
one (column drop, type narrowing, etc), the release notes will call it
out and ship a separate `lettuce-migrate` step. Until then: upgrades
are pure image-tag bumps.

## 11. Troubleshooting

<Note>
  **`GitHub App is not configured on this Lettuce instance`** (HTTP 503
  from `/v1/integrations/github/...`, also surfaces during repo-add)

  Your worker or api pod is missing one of `GITHUB_APP_ID`,
  `GITHUB_APP_NAME`, `GITHUB_APP_PRIVATE_KEY`, `GITHUB_APP_CLIENT_ID`,
  or `GITHUB_APP_CLIENT_SECRET`. The boot path logs the exact missing
  vars (see `config.github_app_env_or_warn`).

  Most common cause: the env was set on the `api` pod but not the
  `worker` pod (or vice versa). Both need all four. This is what LET-8
  re-opened to make actionable — the error message is loud on purpose.
</Note>

<Note>
  **`License expired` / `LicenseExpired`** at boot

  Your JWT's `exp` claim is in the past. Get a renewal from sales. The
  cache file at `/var/cache/lettuce/license.json` is never trusted past
  expiry — there's no way to ride out a lapsed license.
</Note>

<Note>
  **`LETTUCE_LICENSE_KEY is required when LETTUCE_SELF_HOSTED=1`**

  Your license env vars didn't propagate to the worker. Verify with:

  ```bash theme={null}
  docker compose exec worker env | grep LETTUCE_LICENSE      # Compose
  kubectl -n lettuce exec deploy/lettuce-worker -- env | grep LETTUCE_LICENSE   # k8s
  ```

  Compose users: the `.env` file must sit next to `docker-compose.yml`
  and you have to start with `docker compose --env-file …` if you put it
  elsewhere. Helm users: check `existingSecret` resolves and the Secret
  has the right keys.
</Note>

<Note>
  **`No telemetry receipt verified in 7 days`** in the worker logs

  Your instance can't reach `https://telemetry.uselettuce.dev`. Check
  outbound firewall rules — egress on 443 to that host is required
  unless your license is offline-capable (`telemetry_required=false`).
  Run `curl -fsS https://telemetry.uselettuce.dev/healthz` from inside
  the worker pod to confirm DNS + TLS work end-to-end.
</Note>

<Note>
  **Webhook works on add-repo but Lettuce never sees a push**

  Two common causes:

  1. Your GitHub App is missing `Webhooks: Read & Write` — Lettuce
     silently logs the registration failure and falls back to the
     "manual webhook" card in the repo page. Re-grant the permission
     on the App, reinstall on the org, then click **Retry webhook
     setup** on the repo page.
  2. `${LETTUCE_WEB_URL}` isn't reachable from GitHub's outbound IPs.
     This is GHES customers whose Lettuce sits inside a VPN — either
     put a reverse-proxy on the public internet or stop registering
     webhooks and clone on a schedule instead (not yet first-class — file
     feedback).
</Note>

## 12. FAQ

**Can I run Lettuce fully air-gapped?**
Not in v1. RFC-001 §1 explicitly defers air-gap to a paid SKU once a
defense / finance customer signs. Telemetry + license checks both
expect outbound HTTPS. If "no outbound" is a hard requirement, talk to
sales — we'll quote an Enterprise Air-gap pilot.

**Can I clone from a private GitLab self-hosted?**
Yes — LET-21 landed. Set `GITLAB_HOST=gitlab.acme.com` and register an
OAuth app on your GitLab instance (see §5).

**Can I clone from Bitbucket Data Center?**
Same answer — LET-15.6, on the roadmap.

**Do I need Redis?**
No. Lettuce uses Postgres `LISTEN/NOTIFY` for job queuing. One database
serves both api and worker.

**Can I run multiple `api` replicas?**
Yes — they're stateless behind Postgres + S3. Compose users override
`docker compose up -d --scale api=N`; Helm users set `api.replicas`.
The worker also scales horizontally — each claims its own job under a
row-level lock.

**What's the disaster-recovery story?**
Postgres backups + S3 versioning. The `CODEWAZE_DATA_DIR` volume on
each pod is a local scratch cache; everything durable lives in
Postgres + S3.

**What's the SLA?**
TBD per RFC-001 §7.3. Today: best-effort business-hours email at the
Team tier, named CSM + business-hours email at Enterprise. 24×7 paid
support is a roadmap item — not yet sold.

**What's the pricing?**
TBD per RFC-001 §7.1 — sales quotes against your team size and repo
count. Per-seat tiers with a bundled repo cap is the working
recommendation, but the founder hasn't finalised.

**Does the telemetry receiver work today?**
The **receiver** (LET-30) ships and is live. The instance-side
**sender** (LET-30.1) is in flight — until it lands, the daily POST
isn't emitted even with `LETTUCE_TELEMETRY_ENABLED=true`. License
verification still gates boot.

**Where do I file bugs / feature requests?**
Email **[founders@uselettuce.dev](mailto:founders@uselettuce.dev)** with your `instance_id` (visible on
the admin status page, also in the telemetry payload) and a `docker
compose logs` / `kubectl logs` snippet. Linear-tracked issue tracker
goes public in v1.1.