Proxmox-Coolify-Manager/deploy_skill/references/coolify-4.1.2-notes.md

# Coolify v4.1.2 on this instance — hard realities (learned the hard way)

> Written while deploying `cotizador` (Next.js + FastAPI + Postgres, Docker Compose
> build pack, private repo). These are behaviors of the **specific** instance at
> `https://coolify.urieljareth.org`, verified empirically with a root-team token.
> Re-check with `GET /api/v1/version` — if it changed, re-verify the API surface.

## 1. The REST API surface is PARTIAL — `/applications/*` is 404

With a valid **root-team** token (`GET /teams/current` → "Root Team"):

| Endpoint | Result |
|----------|--------|
| `GET /version`, `GET /teams/current` | 200 |
| `GET /projects`, `POST /projects` | 200 / 201 |
| `GET /projects/{uuid}/environments` | 200 |
| `GET /servers` | 200 |
| `POST /security/keys` (create private key) | 201 |
| `GET /resources` | 200 — **the only reliable app/service/db inventory** |
| `GET /services`, `POST /services`, `/services/{uuid}/envs`, `/start` | work |
| `GET /deploy?uuid=<uuid>[&force=true]` | 200 — **triggers a deploy of any existing resource by uuid** |
| `GET /deployments/{deployment_uuid}` | 200 — status + logs (logs = JSON string) |
| **`/applications` (list)** | **404** |
| **`/applications/{uuid}` (get / PATCH)** | **404** |
| **`/applications/{uuid}/envs` (list/create)** | **404** |
| **`/applications/{uuid}/logs`** | **404** |
| **`POST /applications/public`** | **404** |
| **`POST /applications/dockerfile`** | **404** |
| **`POST /applications/private-deploy-key`** | **404** |
| **`POST /applications/private-github-app`** | **404** |
| **`POST /applications/dockercompose`** | **404** (deprecated upstream anyway) |
| `GET /github-apps` | 404 |

**Consequence:** you CANNOT create OR configure a **git-based application that builds
from source** (Dockerfile or Docker Compose build pack) via the API here. The skill's
`New-CoolifyApplication.ps1` (POST /applications/public) will 404 on this instance.
`POST /services` only accepts `docker_compose_raw` with **no git repo** → only good for
**prebuilt-image** stacks, not `build:`-from-source composes.

→ For git+build apps you MUST use the **web UI** (drive it with Playwright). Application
config (build pack, env vars, domains) is UI-only. Only the **deploy trigger** is API-able
(`GET /deploy?uuid=`), and reads via `/resources` + `/deployments/{uuid}`.

## 2. Inventory & status via `/resources`
`GET /resources` returns every application/service/database with `uuid`, `name`,
`status` (e.g. `running:healthy`, `exited:unhealthy`), `fqdn`, `build_pack`,
`git_repository`, `docker_compose_location`, `docker_compose_raw`, `docker_compose`
(parsed), `source_type`/`source_id`, `environment_id`, `destination`. Map an app's
`environment_id` to a project by scanning `/projects/{uuid}/environments`.

Build the UI URL as:
`https://coolify.urieljareth.org/project/{project_uuid}/environment/{env_uuid}/application/{app_uuid}`
(+ `/environment-variables`, `/logs`, `/deployment`, `/source`, `/danger`, …)

## 3. Driving the UI with Playwright (Livewire app)
- Login: `/login`, fill `input[type=email]` + `input[type=password]`, submit; save
  `storageState` and reuse it (see `scripts/coolify-ui/coolify-login.mjs`).
- Import Playwright from the manager repo when the script lives elsewhere:
  `createRequire('<manager>/package.json')` then `require('playwright')`.
- Dismiss recurring popups: buttons "Accept and Close", "Acknowledge & Disable This Popup",
  "Maybe next time".
- **Build Pack** is a `<select wire:model="buildPack">`; robust locator
  `select:has(option[value="dockercompose"])`; `selectOption('dockercompose')`.
- **After switching build pack to Docker Compose you MUST click "Reload Compose File"**
  so Coolify fetches+parses+persists the compose from the repo. Otherwise the deploy
  crashes: `Symfony\Component\Yaml\Yaml::parse(): Argument #1 must be of type string,
  null given` (the parsed `docker_compose` field is null). Verify via `/resources`:
  `docker_compose` must be non-null after reload.
- **Env vars**: Environment Variables tab → "Developer view" → fill the big textarea with
  `KEY=VALUE` lines → click **"Save All Environment Variables"** (NOT a generic "Save").
  Re-saving replaces the whole set, so always send the full list.
- **Pin a compose service's public domain** with a magic env var:
  `SERVICE_FQDN_<SERVICE>_<PORT>=https://host` (e.g. `SERVICE_FQDN_WEB_3000=https://cotizador.urieljareth.org`).
  The compose should also declare a bare `- SERVICE_FQDN_WEB_3000` in that service's env.
- **Deploy**: prefer API `GET /deploy?uuid=&force=true` (works even though app CRUD is 404),
  or the UI "Deploy" button.

## 4. Deploy pitfalls seen
- **UTF-8 BOM / mojibake in the compose file breaks Coolify's YAML parser** → `docker_compose`
  stays null → `Yaml::parse(null)` crash. Keep compose files clean UTF-8, **no BOM**.
  Check: `head -c3 file | xxd` must NOT be `ef bb bf`.
- **Don't queue multiple deploys.** Triggering a new deploy while one is building **cancels**
  the in-progress one → the build dies with `exit code 255` + "Gracefully shutting down build
  container". Trigger ONE deploy and let it finish. Check `/deployments` (running list) is empty
  before triggering.
- `force=true` adds `--no-cache --pull` (slow full rebuild); omit `force` to reuse BuildKit
  cache and cut build time / cancellation risk.
- Coolify's `/resources` `status` can lag reality (showed `running:healthy` while Traefik
  returned 503). Always confirm with an actual HTTP request to the FQDN + a browser render
  (`Test-ServiceOnline.ps1`).

## 5. PowerShell 5.1 gotcha (secret payloads → API)
`Get-Content -Raw` returns a string decorated with ETS NoteProperties; `ConvertTo-Json` then
serializes it as `{"value":"..."}`, so Coolify rejects e.g. `private_key` ("must be a string").
FIX: read with `[System.IO.File]::ReadAllText($path)`.

## 6. Deploy-key path (kept as fallback / for instances where the API is complete)
1. `ssh-keygen -t ed25519`.
2. `POST /security/keys` with `private_key` via `ReadAllText`.
3. Add the public key as a GitHub deploy key: `POST /repos/{owner}/{repo}/keys` `{title,key,read_only:true}`.
4. `POST /applications/private-deploy-key` with `build_pack=dockercompose`,
   `docker_compose_location`, `docker_compose_domains`, `ports_exposes`, `private_key_uuid`,
   `[email protected]:owner/repo.git`.  ← **404 on v4.1.2**; use the UI instead.

## 7. Verified insights — full-stack app + self-hosted Supabase (E3 Manager, 2026-07)

Deployed a Next.js 16 app + Node worker + a full self-hosted Supabase (Auth, PostgREST,
Realtime, Storage, Kong, Studio) end-to-end. New, reusable facts:

### 7.1 `POST /services` request contract (corrects `New-CoolifyService.ps1`)
- `docker_compose_raw` **must be base64-encoded** → else `422 {"docker_compose_raw":"should be base64 encoded"}`.
- **Do NOT send `type` together with `docker_compose_raw`** → else `422 "You cannot provide both service type and docker_compose_raw"`. For a custom compose service, OMIT `type` entirely (the script's `type="one-click-service"` is wrong for this path).
- Minimal working body: `{ name, project_uuid, environment_name, server_uuid, docker_compose_raw:<base64>, instant_deploy:false }`. Returns `{ uuid, domains }`. (`urls` optional; pin domains with your own Traefik labels or `SERVICE_FQDN_*`.)

### 7.2 Locally-built images → Coolify's deploy `pull`s and fails
- Coolify's service deploy runs `docker compose pull` → `pull access denied ... repository does not exist` for a local-only image tag. **Don't use Coolify's Deploy button/`/start` for local images.**
- Instead build the image on the server (clone repo + `docker build -t <tag>`), then `docker compose up -d` **manually** in `/data/coolify/services/<uuid>/` (default pull policy skips pull when the image exists locally). Same pattern as `Deploy-SoloLeveling.ps1`.
- Coolify's normalized on-disk compose (from your base64 raw) **preserves** your `image`, inlined `environment`, custom `labels` (incl. Traefik) and networks — but **renames containers to `<service>-<uuid>`**. Cross-container refs must use the *other* service's real name (unchanged), not the renamed one.
- The service dir + its per-service external network `<uuid>` are created only on Coolify's own deploy. For a first manual `up`, run `docker network create --attachable <uuid>` first (else `network <uuid> declared as external, but could not be found`). Put app containers on the external `coolify` network to reach other stacks (Traefik `coolify-proxy` is already on it).

### 7.3 `Invoke-GitHubApi.ps1` BOM bug
- It writes the JSON body with `Set-Content -Encoding utf8` → PS 5.1 prepends a **BOM** → GitHub `400 "Problems parsing JSON"`. Workaround: POST bodies via `[IO.File]::WriteAllText($tmp,$json)` (no BOM) + `curl.exe --data @tmp`, or fix the helper to use `WriteAllText`.

### 7.4 Self-hosted Supabase for apps that need more than Postgres
- Clone official `supabase/supabase` `docker/volumes` (sparse) for kong.yml + kong-entrypoint.sh + db init SQL; base the compose on the **master** `docker-compose.yml` (kong 3.9.1 uses a Lua entrypoint — keep the matching kong-entrypoint.sh). Trim `functions`/`supavisor`/`analytics`/`vector` if unneeded. Legacy symmetric keys (JWT_SECRET + anon/service_role JWTs) still work; leave publishable/secret empty.
- Route Kong (and Mailpit/Studio) via Traefik **labels + `coolify` network**; the wildcard `*.urieljareth.org` (Cloudflare→Traefik) means new subdomains 503 until a router matches, then work — no DNS change needed.
- **`supabase/postgres` init exceeds the compose healthcheck window** (many internal migrations, no `start_period`) → `db` flaps `unhealthy` and dependents abort. Just re-run `docker compose up -d` once `db` reports healthy.
- Apply the app's own SQL migrations **after** Auth/Storage have initialized their schemas (they need `auth.users`/`storage.buckets`); run as `postgres` via `docker exec -i supabase-db psql` (bypasses `request.jwt.claims`-gated triggers).

### 7.5 `@supabase/ssr` cookie-name gotcha (SSR apps)
- The session cookie name is `sb-<ref>-auth-token` where `ref = new URL(supabaseUrl).hostname.split('.')[0]`. If the **browser** uses the public URL (`demosupabase…` → `sb-demosupabase-…`) and the **server** uses a different internal URL (`http://kong:8000` → `sb-kong-…`), the middleware can't find the session and every login bounces back to `/login` (auth returns 200, but no redirect). Fix: server must use a URL whose first hostname label matches the browser's (simplest: use the same public URL server-side; or give the internal host a network alias equal to that first label).

### 7.6 Remote shell = dash, not bash
- `pct exec 102 -- sh -c '…'` runs under **dash**. No `${PIPESTATUS[0]}`, `[[ ]]`, arrays → "Bad substitution". A build can succeed while the wrapper script exits non-zero purely on a trailing bashism — check the actual artifact (image), not just the exit code.

### 7.7 Invoking these scripts HEADLESSLY (agent/CI) — never wrap them in `2>&1` (verified 2026-07, Solo Leveling re-deploy)
Every deploy script here sets `$ErrorActionPreference = "Stop"` and shells out to native
`git`/`ssh`/`docker` (via `Invoke-ProxmoxSsh.ps1`). Under an **outer** stderr redirect —
`& .\Deploy-*.ps1 2>&1 | …` or `*>&1` — PowerShell **5.1** turns each native-stderr line into a
`NativeCommandError` ErrorRecord; with `Stop` in effect the **first** one becomes terminating and
kills the script on its first line of benign progress (e.g. git `Cloning into 'repo'...`).
Symptom: exit 1 with the error anchored at `& ssh @sshArgs` in `ProxmoxAgent.ps1`, right after the
first `git`/`ssh` progress line — **before any real work fails**.
- **Fix:** invoke the script **plain** — `& .\Deploy-SoloLeveling.ps1` (no `2>&1`/`*>&1`, no
  merging pipe). The harness/terminal already captures the process's stderr at the OS level, which
  does **not** create ErrorRecords. Same rule for `Publish-ProjectToCoolify.ps1`,
  `New-CoolifyService.ps1`, `Invoke-CoolifyRollback.ps1`, `Test-ServiceOnline.ps1`.
- Need the tail in a file? Redirect **stdout only** (`1>`) or wrap with `Start-Transcript`/
  `Tee-Object` on the success stream — never `2>&1` around a native-calling, `Stop`-mode script.
- Note the scripts' *internal* `git push 2>&1 | ForEach-Object {…}` is fine (scoped to one
  statement); the trap is the **outer** redirect the caller adds.

### 7.8 build-on-server re-deploy: verification + rollback caveats (Solo Leveling, `Deploy-SoloLeveling.ps1`)
- **Confirm the shipped commit, not just health.** The build step echoes `HEAD: <sha> <subject>`
  from the fresh clone — gate on it matching your intended commit, and confirm the app container
  shows **`Recreated`** (not reused) in `docker compose up -d` output. Health 200 alone only proves
  *something* is up (and Coolify's `/resources` status lags reality — see §4). Close with
  `Test-ServiceOnline.ps1` (HTTP + real Chromium render) as the definition of done.
- **No instant image rollback on this path.** The build tags only `…/solo-leveling:latest` and
  overwrites it every deploy, so there is no previous image to `docker` back to. Rollback =
  re-run the build-on-server deploy with the repo checked out at the older SHA (slow: full rebuild).
  If instant rollback matters, tag per-commit too (`:<sha>` alongside `:latest`) so you can retag
  `:latest` to a prior digest and `docker compose up -d`. (The skill's `Invoke-CoolifyRollback.ps1`
  assumes the git-build path, which is `/applications/*` = 404 here — it doesn't apply to build-on-server.)