Proxmox-Coolify-Manager/docs/AGENTS-coolify-apps.md

# AGENTS — Building apps that deploy cleanly on this Coolify instance

> **Audience:** LLMs and code agents that are *developing* an application or service
> intended to run on this specific self-hosted Coolify instance.
>
> **Purpose:** This is the single source of truth for *compatibility*. Follow it while
> writing the app (Dockerfile, `docker-compose.yaml`, env handling, ports) so the first
> deploy works instead of going through the trial-and-error captured in the `docs/`
> issues. For *operating* an already-deployed app, see [`../CLAUDE.md`](../CLAUDE.md)
> and the runbooks under [`runbooks/`](runbooks/).

This guide is portable: copy it into the new app's repo (e.g. as its own `AGENTS.md`)
so the agent building that app has the constraints on hand.

---

## 1. The deployment environment (you cannot change this)

```
Internet
  → Cloudflare Tunnel (dashboard-managed, *.urieljareth.org)
    → Traefik  (container: coolify-proxy, terminates TLS, routes by Host header)
      → your app container  (Docker, inside LXC 102)
```

- Everything runs as Docker containers **inside Proxmox LXC `102`** on host
  `192.168.0.200`. There is no direct Docker or LAN access — operators reach it only
  via `ssh [email protected] → pct exec 102 -- docker ...`.
- The public edge is a **dashboard-managed Cloudflare Tunnel**; ingress rules live in
  the Cloudflare Zero Trust dashboard, **not** in files on the host.
- TLS is terminated by **Traefik** (`coolify-proxy`), which also obtains Let's Encrypt
  certificates automatically via **DNS challenge**.
- Coolify manages the lifecycle (build, deploy, env, domains) and writes a generated
  compose to `/data/coolify/applications/<UUID>/docker-compose.yaml`.

Source of truth for topology: [`proxmox-inventory.md`](proxmox-inventory.md) and
[`runbooks/cloudflare-tunnel.md`](runbooks/cloudflare-tunnel.md).

---

## 2. Hard rules (non-negotiable)

Each rule lists **what**, **why**, and **how to verify**.

### 2.1 Networking — reach siblings by Docker service name, never `localhost`

- **Rule:** When your app and its database/cache run as sibling containers on the same
  Docker network, the app must connect using the **Docker service name**, not
  `localhost`/`127.0.0.1`.
- **Why:** `localhost` resolves to the app's own container, not the DB. This is the
  single most common "cannot connect to database" failure on this host
  (see [`../CLAUDE.md`](../CLAUDE.md) and [`runbooks/nextcloud.md`](runbooks/nextcloud.md),
  where Nextcloud's `dbhost` must be `nextcloud-db`).
- **Verify:**
  ```powershell
  .\scripts\Invoke-ProxmoxSsh.ps1 -Command "pct exec 102 -- docker exec <app> getent hosts <db-service-name>"
  ```
  It must return the DB container's IP.

### 2.2 Networking — join the external `coolify` network to reach shared services

- **Rule:** If the app must reach services managed separately by Coolify (a shared
  Postgres/Redis, or another app), declare the global `coolify` network as external and
  attach your service to it:
  ```yaml
  networks:
    coolify:
      name: coolify
      external: true
  ```
- **Why:** Without this, service-name DNS to those resources fails. This is the fix
  documented for Baserow ([`runbooks/baserow.md`](runbooks/baserow.md)), which resolves
  its external Postgres/Redis only through the `coolify` network.
- **Verify:** same `getent hosts` check against the target service name.

### 2.3 Ports — do not publish 80/443; let Traefik route

- **Rule:** **Never** publish host ports `80` or `443` in your compose (no
  `ports: ["80:80"]` / `["443:443"]`). Your app listens on its own internal port; Traefik
  connects to it and handles the public 80/443.
- **Why:** `80`/`443` on the host belong to `coolify-proxy`. Publishing them collides and
  the app fails to start — exactly the Baserow startup failure in
  [`runbooks/baserow.md`](runbooks/baserow.md).
- **Also:** Coolify defaults a new app's exposed port to `3000`. If your app actually
  listens elsewhere (e.g. `80` for nginx, `8080`, etc.), the exposed port **must** match
  the real listening port or Traefik routes to a dead port (502). Set it correctly in the
  Coolify UI/API before the first deploy (`ports_exposes`).

### 2.4 Domains & TLS — `*.urieljareth.org`, DNS challenge, set the FQDN early

- **Rule:** Public hostname pattern is `https://<name>.urieljareth.org`. Set the app's
  **FQDN before the first deploy** — do not rely on Coolify's auto-generated UUID
  subdomain (it has no working DNS route and produces 502s).
- **Why:** TLS is issued by Traefik via **DNS challenge** (Cloudflare API token), so it
  works regardless of Cloudflare's "Always Use HTTPS". Do **not** assume HTTP-01 challenge
  — that path is intentionally not used here
  (see [`issue-coolify-static-app-deploy.md`](issue-coolify-static-app-deploy.md)).
- **Verify:**
  ```powershell
  curl.exe -k -sSI https://<name>.urieljareth.org/
  ```

### 2.5 Secrets — env vars only, never in the repo or image

- **Rule:** All credentials (DB passwords, API tokens, keys) come from **environment
  variables** injected by Coolify. Never hardcode them in source, Dockerfile, compose, or
  committed files, and never bake them into the image.
- **Why:** Repo policy forbids secrets in the tree ([`../CLAUDE.md`](../CLAUDE.md),
  [`runbooks/seguridad.md`](runbooks/seguridad.md)); baked secrets also leak through image
  layers.
- **Verify:** the app boots with secrets supplied only as env vars; the image contains no
  credential values.

---

## 3. Build / deploy options

Coolify can deploy your app in several ways. Pick the simplest that fits, and design the
repo accordingly. (API endpoints are under `/api/v1/...`; reference files in
[`../coolify_skill/references/ops/`](../coolify_skill/references/ops/).)

| Type | When to use | Coolify create endpoint |
|------|-------------|-------------------------|
| Public Git + Nixpacks | Standard app, language auto-detected (Node, Python, Go…) | `POST /applications/public` |
| Static / SPA | Pure HTML/CSS/JS or built front-end (React/Vue) | `POST /applications/public` with `is_static: true` (and `is_spa: true` for SPAs), `static_image` for the serving image |
| Dockerfile | Custom build, no separate registry | `POST /applications/dockerfile` |
| Prebuilt image | Image already in a registry (ghcr.io, Docker Hub) | `POST /applications/dockerimage` |
| Private Git | Private repo via deploy key / GitHub App | `POST /applications/private-deploy-key` or `/private-github-app` |
| Multi-container stack | App + its own DB/cache defined together | `POST /services` (compose). **Note:** `/applications/dockercompose` is deprecated — use `/services`. |

For static sites, prefer `is_static: true` with a known `static_image` (e.g. nginx) and
remember the listening port is then `80` (§2.3).

---

## 4. App configuration reference

Fields your app must expose or that you'll set in Coolify. Schemas come from
[`../coolify_skill/references/ops/`](../coolify_skill/references/ops/) (`healthcheck.md`,
`create-*.md`, `create-env-by-application-uuid.md`).

### Environment variables (`POST /applications/{uuid}/envs`)

| Field | Meaning |
|-------|---------|
| `key` / `value` | The variable |
| `is_literal` | Do not interpolate `$`-style references in the value |
| `is_multiline` | Value spans multiple lines |
| `is_preview` | Applies to preview deployments only |
| `is_shown_once` | Hide in UI after first reveal (for secrets) |

Bulk update: `PATCH /applications/{uuid}/envs/bulk`.

### Ports

| Field | Meaning |
|-------|---------|
| `ports_exposes` | Port the app actually listens on (Traefik targets this). Must match reality — default is `3000`. |
| `ports_mappings` | Optional `host:container` mapping. Avoid `80`/`443` (§2.3). |

### Healthcheck (strongly recommended)

| Field | Notes |
|-------|-------|
| `health_check_enabled` | Turn it on |
| `health_check_path` | e.g. `/health` (expose a cheap, dependency-light endpoint) |
| `health_check_port` | Defaults to the exposed port |
| `health_check_method` / `health_check_return_code` | e.g. `GET` / `200` |
| `health_check_scheme` | `http` between Traefik and the container (TLS is terminated upstream) |
| `health_check_interval` / `_timeout` / `_retries` / `_start_period` | Tune for slow boots |

Design the app to serve a healthcheck endpoint that does **not** depend on the DB being
reachable at boot, so a slow DB doesn't flap the container.

### Resource limits & deployment hooks (optional)

- Limits: `limits_memory`, `limits_memory_swap`, `limits_cpus`, `limits_cpuset`,
  `limits_cpu_shares`.
- Hooks: `pre_deployment_command` / `post_deployment_command` (+ their `_container`),
  e.g. run DB migrations post-deploy. Keep these idempotent.

---

## 5. Volumes / persistence

- Persistence is modeled in **docker-compose** with **named volumes**. Anonymous volumes
  and container-filesystem writes are lost on redeploy.
- Name every volume that must survive a redeploy, and mount it at the app's data path.
- The compose Coolify actually runs is generated at
  `/data/coolify/applications/<UUID>/docker-compose.yaml` — inspect it after deploy to
  confirm your volumes and network landed as intended.

---

## 6. Reference docker-compose snippet

Minimal compose that satisfies every hard rule: app + sibling DB by service name, joined
to the external `coolify` network, **no published 80/443**, a healthcheck, and a named
volume. Secrets arrive as env vars (no literal values here).

```yaml
services:
  app:
    image: your-app:latest          # or build: .
    environment:
      # Reach the DB by SERVICE NAME, never localhost (§2.1)
      DB_HOST: app-db
      DB_PORT: "5432"
      DB_NAME: ${DB_NAME}
      DB_USER: ${DB_USER}
      DB_PASSWORD: ${DB_PASSWORD}    # injected by Coolify env, not hardcoded (§2.5)
      APP_PUBLIC_URL: https://yourapp.urieljareth.org
    # No `ports:` block — Traefik routes to the internal port (§2.3)
    expose:
      - "8080"                        # the port the app listens on -> set ports_exposes=8080
    healthcheck:
      test: ["CMD", "curl", "-fsS", "http://localhost:8080/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 20s
    networks:
      - coolify                       # only if it must reach shared Coolify services (§2.2)
    depends_on:
      - app-db

  app-db:
    image: postgres:16
    environment:
      POSTGRES_DB: ${DB_NAME}
      POSTGRES_USER: ${DB_USER}
      POSTGRES_PASSWORD: ${DB_PASSWORD}
    volumes:
      - app-db-data:/var/lib/postgresql/data   # named volume survives redeploys (§5)

volumes:
  app-db-data: {}

networks:
  coolify:
    name: coolify
    external: true
```

---

## 7. Pre-deploy checklist

- [ ] App reaches its DB/cache by **Docker service name**, not `localhost` (§2.1)
- [ ] If it needs shared Coolify services, compose declares **`coolify` network, `external: true`** (§2.2)
- [ ] Compose does **not** publish host ports `80`/`443` (§2.3)
- [ ] `ports_exposes` in Coolify matches the app's **real listening port** (not the `3000` default) (§2.3)
- [ ] **FQDN set** (`https://<name>.urieljareth.org`) before the first deploy — no auto UUID subdomain (§2.4)
- [ ] TLS assumed via **DNS challenge** — no HTTP-01 / `.well-known` dependency in the app (§2.4)
- [ ] All secrets injected as **env vars**, none committed or baked into the image (§2.5)
- [ ] **Healthcheck** endpoint defined and DB-independent at boot (§4)
- [ ] Persistent data uses **named volumes** (§5)

---

## 8. Post-deploy verification

Run from the repo root (PowerShell). These reuse existing scripts — no new tooling.

```powershell
# 1. Public endpoint responds (TLS issued, Traefik routing correct)
curl.exe -k -sSI https://<name>.urieljareth.org/

# 2. Sibling/shared DNS resolves from inside the app container
.\scripts\Invoke-ProxmoxSsh.ps1 -Command "pct exec 102 -- docker exec <app> getent hosts <db-service-name>"

# 3. Container status overview
.\coolify_skill\scripts\Get-CoolifyDockerStatus.ps1 -All

# 4. Certificate / routing errors at the proxy
.\scripts\Invoke-ProxmoxSsh.ps1 -Command "pct exec 102 -- docker logs coolify-proxy --tail 50 2>&1 | grep -Ei 'error|certificate|acme|<name>'"

# 5. Inspect the generated compose actually in use
.\scripts\Invoke-ProxmoxSsh.ps1 -Command "pct exec 102 -- cat /data/coolify/applications/<UUID>/docker-compose.yaml"
```

---

## 9. Known pitfalls (symptom → cause → fix)

| Symptom | Root cause | Fix | Source |
|---------|-----------|-----|--------|
| App's domain returns **502 Bad Gateway** | `ports_exposes` doesn't match the real listening port (often `3000` vs `80`) | Set `ports_exposes` to the actual port before deploy | [issue-coolify-static-app-deploy.md](issue-coolify-static-app-deploy.md) |
| 502 / no certificate on the app domain | Traefik couldn't get a cert via HTTP challenge | Environment uses **DNS challenge**; ensure FQDN is set and don't depend on HTTP-01 | [issue-coolify-static-app-deploy.md](issue-coolify-static-app-deploy.md) |
| App reachable only at an ugly **UUID subdomain** | FQDN not set, Coolify auto-generated it | Set `fqdn` to `https://<name>.urieljareth.org` before first deploy | [issue-coolify-static-app-deploy.md](issue-coolify-static-app-deploy.md) |
| App **won't start**, port conflict | Compose publishes `80`/`443` (owned by `coolify-proxy`) | Remove host port publishing; let Traefik route | [runbooks/baserow.md](runbooks/baserow.md) |
| App **can't reach its DB** | Used `localhost`, or DB is on a different network | Use the DB **service name**; for shared services join the `coolify` network | [runbooks/nextcloud.md](runbooks/nextcloud.md), [runbooks/baserow.md](runbooks/baserow.md) |
| Coolify **UI blank** when opening the app page | Cloudflare tunnel route order — `/app/*` captured `/application/...` | Operator fix: `/project/*` route must precede `/app/*` in the dashboard | [issue-coolify-static-app-deploy.md](issue-coolify-static-app-deploy.md) |
| **WebSocket / terminal** drops, `tls: first record does not look like a TLS handshake` | Tunnel routes for ports `6001`/`6002` set to `https://` | Operator fix: those routes must be `http://` | [ISSUE_cloudflare-tunnel_routing_websocket-tls-handshake.md](ISSUE_cloudflare-tunnel_routing_websocket-tls-handshake.md) |

> The last two are *operator/infrastructure* fixes (Cloudflare dashboard), not things the
> app developer changes — listed here so an agent recognizes the symptom and points the
> operator to the right runbook instead of debugging the app.

---

## 10. See also

- [`../CLAUDE.md`](../CLAUDE.md) — repo architecture, SSH path, operating rules
- [`proxmox-inventory.md`](proxmox-inventory.md) — verified topology
- [`runbooks/cloudflare-tunnel.md`](runbooks/cloudflare-tunnel.md) — tunnel routes
- [`runbooks/coolify-docker.md`](runbooks/coolify-docker.md) — Docker ops inside LXC 102
- [`runbooks/nextcloud.md`](runbooks/nextcloud.md), [`runbooks/baserow.md`](runbooks/baserow.md) — real app patterns
- [`../coolify_skill/references/`](../coolify_skill/references/) — Coolify API operation reference (search with `rg`, open one `ops/*.md`)