Proxmox-Coolify-Manager/deploy_skill/references/coolify-4.1.2-notes.md

Coolify v4.1.2 on this instance — hard realities (learned the hard way)

Written while deploying cotizador (Next.js + FastAPI + Postgres, Docker Compose build pack, private repo). These are behaviors of the specific instance at https://coolify.urieljareth.org, verified empirically with a root-team token. Re-check with GET /api/v1/version — if it changed, re-verify the API surface.

1. The REST API surface is PARTIAL — /applications/* is 404

With a valid root-team token (GET /teams/current → "Root Team"):

Endpoint	Result
GET /version, GET /teams/current	200
GET /projects, POST /projects	200 / 201
GET /projects/{uuid}/environments	200
GET /servers	200
POST /security/keys (create private key)	201
GET /resources	200 — the only reliable app/service/db inventory
GET /services, POST /services, /services/{uuid}/envs, /start	work
GET /deploy?uuid=<uuid>[&force=true]	200 — triggers a deploy of any existing resource by uuid
GET /deployments/{deployment_uuid}	200 — status + logs (logs = JSON string)
/applications (list)	404
/applications/{uuid} (get / PATCH)	404
/applications/{uuid}/envs (list/create)	404
/applications/{uuid}/logs	404
POST /applications/public	404
POST /applications/dockerfile	404
POST /applications/private-deploy-key	404
POST /applications/private-github-app	404
POST /applications/dockercompose	404 (deprecated upstream anyway)
GET /github-apps	404

Consequence: you CANNOT create OR configure a git-based application that builds from source (Dockerfile or Docker Compose build pack) via the API here. The skill's New-CoolifyApplication.ps1 (POST /applications/public) will 404 on this instance. POST /services only accepts docker_compose_raw with no git repo → only good for prebuilt-image stacks, not build:-from-source composes.

→ For git+build apps you MUST use the web UI (drive it with Playwright). Application config (build pack, env vars, domains) is UI-only. Only the deploy trigger is API-able (GET /deploy?uuid=), and reads via /resources + /deployments/{uuid}.

2. Inventory & status via /resources

GET /resources returns every application/service/database with uuid, name, status (e.g. running:healthy, exited:unhealthy), fqdn, build_pack, git_repository, docker_compose_location, docker_compose_raw, docker_compose (parsed), source_type/source_id, environment_id, destination. Map an app's environment_id to a project by scanning /projects/{uuid}/environments.

Build the UI URL as: https://coolify.urieljareth.org/project/{project_uuid}/environment/{env_uuid}/application/{app_uuid} (+ /environment-variables, /logs, /deployment, /source, /danger, …)

3. Driving the UI with Playwright (Livewire app)

• Login: /login, fill input[type=email] + input[type=password], submit; save storageState and reuse it (see scripts/coolify-ui/coolify-login.mjs).
• Import Playwright from the manager repo when the script lives elsewhere: createRequire('<manager>/package.json') then require('playwright').
• Dismiss recurring popups: buttons "Accept and Close", "Acknowledge & Disable This Popup", "Maybe next time".
• Build Pack is a <select wire:model="buildPack">; robust locator select:has(option[value="dockercompose"]); selectOption('dockercompose').
• After switching build pack to Docker Compose you MUST click "Reload Compose File" so Coolify fetches+parses+persists the compose from the repo. Otherwise the deploy crashes: Symfony\Component\Yaml\Yaml::parse(): Argument #1 must be of type string, null given (the parsed docker_compose field is null). Verify via /resources: docker_compose must be non-null after reload.
• Env vars: Environment Variables tab → "Developer view" → fill the big textarea with KEY=VALUE lines → click "Save All Environment Variables" (NOT a generic "Save"). Re-saving replaces the whole set, so always send the full list.
• Pin a compose service's public domain with a magic env var: SERVICE_FQDN_<SERVICE>_<PORT>=https://host (e.g. SERVICE_FQDN_WEB_3000=https://cotizador.urieljareth.org). The compose should also declare a bare - SERVICE_FQDN_WEB_3000 in that service's env.
• Deploy: prefer API GET /deploy?uuid=&force=true (works even though app CRUD is 404), or the UI "Deploy" button.

4. Deploy pitfalls seen

• UTF-8 BOM / mojibake in the compose file breaks Coolify's YAML parser → docker_compose stays null → Yaml::parse(null) crash. Keep compose files clean UTF-8, no BOM. Check: head -c3 file | xxd must NOT be ef bb bf.
• Don't queue multiple deploys. Triggering a new deploy while one is building cancels the in-progress one → the build dies with exit code 255 + "Gracefully shutting down build container". Trigger ONE deploy and let it finish. Check /deployments (running list) is empty before triggering.
• force=true adds --no-cache --pull (slow full rebuild); omit force to reuse BuildKit cache and cut build time / cancellation risk.
• Coolify's /resources status can lag reality (showed running:healthy while Traefik returned 503). Always confirm with an actual HTTP request to the FQDN + a browser render (Test-ServiceOnline.ps1).

5. PowerShell 5.1 gotcha (secret payloads → API)

Get-Content -Raw returns a string decorated with ETS NoteProperties; ConvertTo-Json then serializes it as {"value":"..."}, so Coolify rejects e.g. private_key ("must be a string"). FIX: read with [System.IO.File]::ReadAllText($path).

6. Deploy-key path (kept as fallback / for instances where the API is complete)

1. ssh-keygen -t ed25519.
2. POST /security/keys with private_key via ReadAllText.
3. Add the public key as a GitHub deploy key: POST /repos/{owner}/{repo}/keys {title,key,read_only:true}.
4. POST /applications/private-deploy-key with build_pack=dockercompose, docker_compose_location, docker_compose_domains, ports_exposes, private_key_uuid, [email protected]:owner/repo.git. ← 404 on v4.1.2; use the UI instead.

7. Verified insights — full-stack app + self-hosted Supabase (E3 Manager, 2026-07)

Deployed a Next.js 16 app + Node worker + a full self-hosted Supabase (Auth, PostgREST, Realtime, Storage, Kong, Studio) end-to-end. New, reusable facts:

7.1 POST /services request contract (corrects New-CoolifyService.ps1)

• docker_compose_raw must be base64-encoded → else 422 {"docker_compose_raw":"should be base64 encoded"}.
• Do NOT send type together with docker_compose_raw → else 422 "You cannot provide both service type and docker_compose_raw". For a custom compose service, OMIT type entirely (the script's type="one-click-service" is wrong for this path).
• Minimal working body: { name, project_uuid, environment_name, server_uuid, docker_compose_raw:<base64>, instant_deploy:false }. Returns { uuid, domains }. (urls optional; pin domains with your own Traefik labels or SERVICE_FQDN_*.)

7.2 Locally-built images → Coolify's deploy pulls and fails

• Coolify's service deploy runs docker compose pull → pull access denied ... repository does not exist for a local-only image tag. Don't use Coolify's Deploy button//start for local images.
• Instead build the image on the server (clone repo + docker build -t <tag>), then docker compose up -d manually in /data/coolify/services/<uuid>/ (default pull policy skips pull when the image exists locally). Same pattern as Deploy-SoloLeveling.ps1.
• Coolify's normalized on-disk compose (from your base64 raw) preserves your image, inlined environment, custom labels (incl. Traefik) and networks — but renames containers to <service>-<uuid>. Cross-container refs must use the other service's real name (unchanged), not the renamed one.
• The service dir + its per-service external network <uuid> are created only on Coolify's own deploy. For a first manual up, run docker network create --attachable <uuid> first (else network <uuid> declared as external, but could not be found). Put app containers on the external coolify network to reach other stacks (Traefik coolify-proxy is already on it).

7.3 Invoke-GitHubApi.ps1 BOM bug

• It writes the JSON body with Set-Content -Encoding utf8 → PS 5.1 prepends a BOM → GitHub 400 "Problems parsing JSON". Workaround: POST bodies via [IO.File]::WriteAllText($tmp,$json) (no BOM) + curl.exe --data @tmp, or fix the helper to use WriteAllText.

7.4 Self-hosted Supabase for apps that need more than Postgres

• Clone official supabase/supabase docker/volumes (sparse) for kong.yml + kong-entrypoint.sh + db init SQL; base the compose on the master docker-compose.yml (kong 3.9.1 uses a Lua entrypoint — keep the matching kong-entrypoint.sh). Trim functions/supavisor/analytics/vector if unneeded. Legacy symmetric keys (JWT_SECRET + anon/service_role JWTs) still work; leave publishable/secret empty.
• Route Kong (and Mailpit/Studio) via Traefik labels + coolify network; the wildcard *.urieljareth.org (Cloudflare→Traefik) means new subdomains 503 until a router matches, then work — no DNS change needed.
• supabase/postgres init exceeds the compose healthcheck window (many internal migrations, no start_period) → db flaps unhealthy and dependents abort. Just re-run docker compose up -d once db reports healthy.
• Apply the app's own SQL migrations after Auth/Storage have initialized their schemas (they need auth.users/storage.buckets); run as postgres via docker exec -i supabase-db psql (bypasses request.jwt.claims-gated triggers).

7.5 @supabase/ssr cookie-name gotcha (SSR apps)

• The session cookie name is sb-<ref>-auth-token where ref = new URL(supabaseUrl).hostname.split('.')[0]. If the browser uses the public URL (demosupabase… → sb-demosupabase-…) and the server uses a different internal URL (http://kong:8000 → sb-kong-…), the middleware can't find the session and every login bounces back to /login (auth returns 200, but no redirect). Fix: server must use a URL whose first hostname label matches the browser's (simplest: use the same public URL server-side; or give the internal host a network alias equal to that first label).

7.6 Remote shell = dash, not bash

• pct exec 102 -- sh -c '…' runs under dash. No ${PIPESTATUS[0]}, [[ ]], arrays → "Bad substitution". A build can succeed while the wrapper script exits non-zero purely on a trailing bashism — check the actual artifact (image), not just the exit code.

7.7 Invoking these scripts HEADLESSLY (agent/CI) — never wrap them in 2>&1 (verified 2026-07, Solo Leveling re-deploy)

Every deploy script here sets $ErrorActionPreference = "Stop" and shells out to native git/ssh/docker (via Invoke-ProxmoxSsh.ps1). Under an outer stderr redirect — & .\Deploy-*.ps1 2>&1 | … or *>&1 — PowerShell 5.1 turns each native-stderr line into a NativeCommandError ErrorRecord; with Stop in effect the first one becomes terminating and kills the script on its first line of benign progress (e.g. git Cloning into 'repo'...). Symptom: exit 1 with the error anchored at & ssh @sshArgs in ProxmoxAgent.ps1, right after the first git/ssh progress line — before any real work fails.

• Fix: invoke the script plain — & .\Deploy-SoloLeveling.ps1 (no 2>&1/*>&1, no merging pipe). The harness/terminal already captures the process's stderr at the OS level, which does not create ErrorRecords. Same rule for Publish-ProjectToCoolify.ps1, New-CoolifyService.ps1, Invoke-CoolifyRollback.ps1, Test-ServiceOnline.ps1.
• Need the tail in a file? Redirect stdout only (1>) or wrap with Start-Transcript/ Tee-Object on the success stream — never 2>&1 around a native-calling, Stop-mode script.
• Note the scripts' internal git push 2>&1 | ForEach-Object {…} is fine (scoped to one statement); the trap is the outer redirect the caller adds.

7.8 build-on-server re-deploy: verification + rollback caveats (Solo Leveling, Deploy-SoloLeveling.ps1)

• Confirm the shipped commit, not just health. The build step echoes HEAD: <sha> <subject> from the fresh clone — gate on it matching your intended commit, and confirm the app container shows Recreated (not reused) in docker compose up -d output. Health 200 alone only proves something is up (and Coolify's /resources status lags reality — see §4). Close with Test-ServiceOnline.ps1 (HTTP + real Chromium render) as the definition of done.
• No instant image rollback on this path. The build tags only …/solo-leveling:latest and overwrites it every deploy, so there is no previous image to docker back to. Rollback = re-run the build-on-server deploy with the repo checked out at the older SHA (slow: full rebuild). If instant rollback matters, tag per-commit too (:<sha> alongside :latest) so you can retag :latest to a prior digest and docker compose up -d. (The skill's Invoke-CoolifyRollback.ps1 assumes the git-build path, which is /applications/* = 404 here — it doesn't apply to build-on-server.)