Proxmox-Coolify-Manager/deploy_skill/references/coolify-4.1.2-notes.md

Coolify v4.1.2 on this instance — hard realities (learned the hard way)

Written while deploying cotizador (Next.js + FastAPI + Postgres, Docker Compose build pack, private repo). These are behaviors of the specific instance at https://coolify.urieljareth.org, verified empirically with a root-team token. Re-check with GET /api/v1/version — if it changed, re-verify the API surface.

1. The REST API surface is PARTIAL — /applications/* is 404

CORRECCIÓN (2026-08-29, §11): ese 404 lo imponía Cloudflare en el hostname público, no Coolify. Contra el origen (http://192.168.0.117:8000/api/v1) el namespace completo responde 200 y el openapi.yaml del contenedor declara toda la superficie. Esta sección y la tabla se conservan como registro histórico del diagnóstico de 2026-08-07.

With a valid root-team token (GET /teams/current → "Root Team"):

Endpoint	Result
GET /version, GET /teams/current	200
GET /projects, POST /projects	200 / 201
GET /projects/{uuid}/environments	200
GET /servers	200
POST /security/keys (create private key)	201
GET /resources	200 — the only reliable app/service/db inventory
GET /services, POST /services, /services/{uuid}/envs, /start	work
GET /deploy?uuid=<uuid>[&force=true]	200 — triggers a deploy of any existing resource by uuid
GET /deployments/{deployment_uuid}	200 — status + logs (logs = JSON string)
/applications (list)	404
/applications/{uuid} (get / PATCH)	404
/applications/{uuid}/envs (list/create)	404
/applications/{uuid}/logs	404
POST /applications/public	404
POST /applications/dockerfile	404
POST /applications/private-deploy-key	404
POST /applications/private-github-app	404
POST /applications/dockercompose	404 (deprecated upstream anyway)
GET /github-apps	404

Consequence: you CANNOT create OR configure a git-based application that builds from source (Dockerfile or Docker Compose build pack) via the API here. The skill's New-CoolifyApplication.ps1 (POST /applications/public) will 404 on this instance. POST /services only accepts docker_compose_raw with no git repo → only good for prebuilt-image stacks, not build:-from-source composes.

→ For git+build apps you MUST use the web UI (drive it with Playwright). Application config (build pack, env vars, domains) is UI-only. Only the deploy trigger is API-able (GET /deploy?uuid=), and reads via /resources + /deployments/{uuid}.

2. Inventory & status via /resources

GET /resources returns every application/service/database with uuid, name, status (e.g. running:healthy, exited:unhealthy), fqdn, build_pack, git_repository, docker_compose_location, docker_compose_raw, docker_compose (parsed), source_type/source_id, environment_id, destination. Map an app's environment_id to a project by scanning /projects/{uuid}/environments.

Build the UI URL as: https://coolify.urieljareth.org/project/{project_uuid}/environment/{env_uuid}/application/{app_uuid} (+ /environment-variables, /logs, /deployment, /source, /danger, …)

3. Driving the UI with Playwright (Livewire app)

• Login: /login, fill input[type=email] + input[type=password], submit; save storageState and reuse it (see scripts/coolify-ui/coolify-login.mjs).
• Import Playwright from the manager repo when the script lives elsewhere: createRequire('<manager>/package.json') then require('playwright').
• Dismiss recurring popups: buttons "Accept and Close", "Acknowledge & Disable This Popup", "Maybe next time".
• Build Pack is a <select wire:model="buildPack">; robust locator select:has(option[value="dockercompose"]); selectOption('dockercompose').
• After switching build pack to Docker Compose you MUST click "Reload Compose File" so Coolify fetches+parses+persists the compose from the repo. Otherwise the deploy crashes: Symfony\Component\Yaml\Yaml::parse(): Argument #1 must be of type string, null given (the parsed docker_compose field is null). Verify via /resources: docker_compose must be non-null after reload.
• Env vars: Environment Variables tab → "Developer view" → fill the big textarea with KEY=VALUE lines → click "Save All Environment Variables" (NOT a generic "Save"). Re-saving replaces the whole set, so always send the full list.
• Pin a compose service's public domain with a magic env var: SERVICE_FQDN_<SERVICE>_<PORT>=https://host (e.g. SERVICE_FQDN_WEB_3000=https://cotizador.urieljareth.org). The compose should also declare a bare - SERVICE_FQDN_WEB_3000 in that service's env.
• Deploy: prefer API GET /deploy?uuid=&force=true (works even though app CRUD is 404), or the UI "Deploy" button.

4. Deploy pitfalls seen

• UTF-8 BOM / mojibake in the compose file breaks Coolify's YAML parser → docker_compose stays null → Yaml::parse(null) crash. Keep compose files clean UTF-8, no BOM. Check: head -c3 file | xxd must NOT be ef bb bf.
• Don't queue multiple deploys. Triggering a new deploy while one is building cancels the in-progress one → the build dies with exit code 255 + "Gracefully shutting down build container". Trigger ONE deploy and let it finish. Check /deployments (running list) is empty before triggering.
• force=true adds --no-cache --pull (slow full rebuild); omit force to reuse BuildKit cache and cut build time / cancellation risk.
• Coolify's /resources status can lag reality (showed running:healthy while Traefik returned 503). Always confirm with an actual HTTP request to the FQDN + a browser render (Test-ServiceOnline.ps1).

5. PowerShell 5.1 gotcha (secret payloads → API)

Get-Content -Raw returns a string decorated with ETS NoteProperties; ConvertTo-Json then serializes it as {"value":"..."}, so Coolify rejects e.g. private_key ("must be a string"). FIX: read with [System.IO.File]::ReadAllText($path).

6. Deploy-key path (kept as fallback / for instances where the API is complete)

1. ssh-keygen -t ed25519.
2. POST /security/keys with private_key via ReadAllText.
3. Add the public key as a GitHub deploy key: POST /repos/{owner}/{repo}/keys {title,key,read_only:true}.
4. POST /applications/private-deploy-key with build_pack=dockercompose, docker_compose_location, docker_compose_domains, ports_exposes, private_key_uuid, [email protected]:owner/repo.git. ← 404 on v4.1.2; use the UI instead.

7. Verified insights — full-stack app + self-hosted Supabase (E3 Manager, 2026-07)

Deployed a Next.js 16 app + Node worker + a full self-hosted Supabase (Auth, PostgREST, Realtime, Storage, Kong, Studio) end-to-end. New, reusable facts:

7.1 POST /services request contract (corrects New-CoolifyService.ps1)

• docker_compose_raw must be base64-encoded → else 422 {"docker_compose_raw":"should be base64 encoded"}.
• Do NOT send type together with docker_compose_raw → else 422 "You cannot provide both service type and docker_compose_raw". For a custom compose service, OMIT type entirely (the script's type="one-click-service" is wrong for this path).
• Minimal working body: { name, project_uuid, environment_name, server_uuid, docker_compose_raw:<base64>, instant_deploy:false }. Returns { uuid, domains }. (urls optional; pin domains with your own Traefik labels or SERVICE_FQDN_*.)

7.2 Locally-built images → Coolify's deploy pulls and fails

• Coolify's service deploy runs docker compose pull → pull access denied ... repository does not exist for a local-only image tag. Don't use Coolify's Deploy button//start for local images.
• Instead build the image on the server (clone repo + docker build -t <tag>), then docker compose up -d manually in /data/coolify/services/<uuid>/ (default pull policy skips pull when the image exists locally). Same pattern as scripts/apps/Deploy-SoloLeveling.ps1.
• Coolify's normalized on-disk compose (from your base64 raw) preserves your image, inlined environment, custom labels (incl. Traefik) and networks — but renames containers to <service>-<uuid>. Cross-container refs must use the other service's real name (unchanged), not the renamed one.
• The service dir + its per-service external network <uuid> are created only on Coolify's own deploy. For a first manual up, run docker network create --attachable <uuid> first (else network <uuid> declared as external, but could not be found). Put app containers on the external coolify network to reach other stacks (Traefik coolify-proxy is already on it).

7.3 Invoke-GitHubApi.ps1 BOM bug

• It writes the JSON body with Set-Content -Encoding utf8 → PS 5.1 prepends a BOM → GitHub 400 "Problems parsing JSON". Workaround: POST bodies via [IO.File]::WriteAllText($tmp,$json) (no BOM) + curl.exe --data @tmp, or fix the helper to use WriteAllText.

7.4 Self-hosted Supabase for apps that need more than Postgres

• Clone official supabase/supabase docker/volumes (sparse) for kong.yml + kong-entrypoint.sh + db init SQL; base the compose on the master docker-compose.yml (kong 3.9.1 uses a Lua entrypoint — keep the matching kong-entrypoint.sh). Trim functions/supavisor/analytics/vector if unneeded. Legacy symmetric keys (JWT_SECRET + anon/service_role JWTs) still work; leave publishable/secret empty.
• Route Kong (and Mailpit/Studio) via Traefik labels + coolify network; the wildcard *.urieljareth.org (Cloudflare→Traefik) means new subdomains 503 until a router matches, then work — no DNS change needed.
• supabase/postgres init exceeds the compose healthcheck window (many internal migrations, no start_period) → db flaps unhealthy and dependents abort. Just re-run docker compose up -d once db reports healthy.
• Apply the app's own SQL migrations after Auth/Storage have initialized their schemas (they need auth.users/storage.buckets); run as postgres via docker exec -i supabase-db psql (bypasses request.jwt.claims-gated triggers).

7.5 @supabase/ssr cookie-name gotcha (SSR apps)

• The session cookie name is sb-<ref>-auth-token where ref = new URL(supabaseUrl).hostname.split('.')[0]. If the browser uses the public URL (demosupabase… → sb-demosupabase-…) and the server uses a different internal URL (http://kong:8000 → sb-kong-…), the middleware can't find the session and every login bounces back to /login (auth returns 200, but no redirect). Fix: server must use a URL whose first hostname label matches the browser's (simplest: use the same public URL server-side; or give the internal host a network alias equal to that first label).

7.6 Remote shell = dash, not bash

• pct exec 102 -- sh -c '…' runs under dash. No ${PIPESTATUS[0]}, [[ ]], arrays → "Bad substitution". A build can succeed while the wrapper script exits non-zero purely on a trailing bashism — check the actual artifact (image), not just the exit code.

7.7 Invoking these scripts HEADLESSLY (agent/CI) — never wrap them in 2>&1 (verified 2026-07, Solo Leveling re-deploy)

Every deploy script here sets $ErrorActionPreference = "Stop" and shells out to native git/ssh/docker (via Invoke-ProxmoxSsh.ps1). Under an outer stderr redirect — & .\Deploy-*.ps1 2>&1 | … or *>&1 — PowerShell 5.1 turns each native-stderr line into a NativeCommandError ErrorRecord; with Stop in effect the first one becomes terminating and kills the script on its first line of benign progress (e.g. git Cloning into 'repo'...). Symptom: exit 1 with the error anchored at & ssh @sshArgs in ProxmoxAgent.ps1, right after the first git/ssh progress line — before any real work fails.

• Fix: invoke the script plain — & .\scripts\apps\Deploy-SoloLeveling.ps1 (no 2>&1/*>&1, no merging pipe). The harness/terminal already captures the process's stderr at the OS level, which does not create ErrorRecords. Same rule for Publish-ProjectToCoolify.ps1, New-CoolifyService.ps1, Invoke-CoolifyRollback.ps1, Test-ServiceOnline.ps1.
• Need the tail in a file? Redirect stdout only (1>) or wrap with Start-Transcript/ Tee-Object on the success stream — never 2>&1 around a native-calling, Stop-mode script.
• Note the scripts' internal git push 2>&1 | ForEach-Object {…} is fine (scoped to one statement); the trap is the outer redirect the caller adds.

7.8 build-on-server re-deploy: verification + rollback caveats (Solo Leveling, scripts/apps/Deploy-SoloLeveling.ps1)

• Confirm the shipped commit, not just health. The build step echoes HEAD: <sha> <subject> from the fresh clone — gate on it matching your intended commit, and confirm the app container shows Recreated (not reused) in docker compose up -d output. Health 200 alone only proves something is up (and Coolify's /resources status lags reality — see §4). Close with Test-ServiceOnline.ps1 (HTTP + real Chromium render) as the definition of done.
• No instant image rollback on this path. The build tags only …/solo-leveling:latest and overwrites it every deploy, so there is no previous image to docker back to. Rollback = re-run the build-on-server deploy with the repo checked out at the older SHA (slow: full rebuild). If instant rollback matters, tag per-commit too (:<sha> alongside :latest) so you can retag :latest to a prior digest and docker compose up -d. (The skill's Invoke-CoolifyRollback.ps1 assumes the git-build path, which is /applications/* = 404 here — it doesn't apply to build-on-server.)

8. Creating git-based apps via direct DB INSERT (when UI credentials are unavailable)

Verified 2026-07-27 deploying AgendaMax (Node 22 + Vite + Express + SQLite, Dockerfile build pack, GitHub source). This is the third path, alongside §3 (Playwright UI) and §6 (API, 404 here). Use it when you have SSH+DB access but NOT the Coolify web UI email/password.

8.1 The SSH → pct → docker exec → psql chain

local machine
  → ssh [email protected] -i keys\proxmox_ed25519
    → pct exec 102 -- docker exec coolify-db psql -U coolify -d coolify -c "SQL_HERE"

Credentials (all local/private, no secrets leave the host):

• Proxmox host: [email protected]
• SSH key: keys\proxmox_ed25519
• Coolify LXC: 102
• Coolify DB container: coolify-db (PostgreSQL 15, user coolify, db coolify, no password — container-internal)
• Coolify app container: coolify (image ghcr.io/coollabsio/coolify:4.1.2)
• Coolify API token: in .env.local.ps1 → $env:COOLIFY_TOKEN
• GitHub PAT: in .env.local.ps1 → $env:GITHUB_TOKEN (scope repo, account urieljarethbusiness-cpu)

8.2 PowerShell quoting nightmare — the scp+sh workaround

Problem: Passing SQL (with single quotes, backslashes in PHP namespaces like App\Models\GithubApp, or {{pr_id}} braces) through the chain PowerShell → ssh → pct → docker → psql is quoting hell. Nested '…' inside "…" inside "…" breaks at every level.

Solution: Write a .sh script locally → scp to the Proxmox host → ssh … sh /tmp/script.sh.

# 1. Write the script locally with heredoc-safe content
# 2. scp it to the host
scp -i $SSH_KEY script.sh root@192.168.0.200:/tmp/script.sh
# 3. Execute remotely
ssh -i $SSH_KEY root@192.168.0.200 "sh /tmp/script.sh"

Never try to pass complex SQL inline through ssh … "pct exec … psql … -c '…'" from PowerShell — the nested quoting will eat hours. Always scp a script.

8.3 Step-by-step: create a Dockerfile-based app from a GitHub repo

Prerequisites:

• The repo must be on GitHub (urieljarethbusiness-cpu/<name>) — Coolify's GitHub App (source_id=3, installation_id=121211999) is already configured and can access all repos under that account.
• Push the code first: git push github main.
• Get the GitHub repo numeric ID: GET https://api.github.com/repos/<owner>/<repo> → .id.

Step 1 — Find the environment_id:

SELECT id FROM environments WHERE uuid = '<env_uuid>';
-- e.g. for project "tools" production: returns 9

Step 2 — INSERT into applications: every field with a NOT NULL DEFAULT in the schema must be set explicitly (the INSERT doesn't fire Laravel's model events, so defaults from migrations are the DB column defaults, not Eloquent $casts/boot logic).

Key fields for a Dockerfile app:

INSERT INTO applications (
  uuid, name, git_repository, git_branch, git_commit_sha,
  build_pack,              -- 'dockerfile'
  dockerfile_location,     -- '/Dockerfile'
  ports_exposes,           -- '3000' (your app's port)
  health_check_path, health_check_port, health_check_host,
  health_check_method, health_check_return_code, health_check_scheme,
  health_check_interval, health_check_timeout, health_check_retries, health_check_start_period,
  health_check_enabled,    -- false (simpler; enable later via UI if needed)
  limits_memory, limits_memory_swap, limits_memory_swappiness, limits_memory_reservation,
  limits_cpus, limits_cpu_shares,
  status,                  -- 'exited'
  preview_url_template,    -- '{{pr_id}}.{{domain}}'
  fqdn,                    -- 'https://<name>.urieljareth.org'
  repository_project_id,   -- numeric GitHub repo ID (from GitHub API)
  source_type,             -- 'App\Models\GithubApp'
  source_id,               -- 3 (the GitHub App, see SELECT id FROM github_apps)
  destination_type,        -- 'App\Models\StandaloneDocker'
  destination_id,          -- 0 (the localhost Docker, see SELECT id FROM standalone_dockers)
  environment_id,          -- from Step 1
  base_directory,          -- '/'
  static_image,            -- 'nginx:alpine' (unused for dockerfile pack, but NOT NULL)
  created_at, updated_at   -- NOW()
) VALUES (...);

Step 3 — INSERT into application_settings (CRITICAL — skip this and deploys crash):

production.ERROR: Attempt to read property "disable_build_cache" on null
  at ApplicationDeploymentJob.php:214

Coolify's deployment job reads $application->settings->disable_build_cache in its constructor. Without a settings row, settings is null and the job dies instantly — the deployment stays "in_progress" forever and blocks the queue (see §8.4).

INSERT INTO application_settings (
  application_id,  -- the id from Step 2's RETURNING
  is_static, is_git_submodules_enabled, is_git_lfs_enabled,
  is_auto_deploy_enabled, is_force_https_enabled, is_debug_enabled,
  is_preview_deployments_enabled, is_log_drain_enabled, is_gpu_enabled,
  is_swarm_only_worker_nodes, is_raw_compose_deployment_enabled,
  is_build_server_enabled, is_consistent_container_name_enabled,
  is_gzip_enabled, is_stripprefix_enabled,
  is_container_label_escape_enabled, is_container_label_readonly_enabled,
  disable_build_cache, is_spa, is_git_shallow_clone_enabled,
  is_pr_deployments_public_enabled, use_build_secrets, inject_build_args_to_dockerfile,
  docker_images_to_keep,
  created_at, updated_at
) VALUES (
  <app_id>, false, true, true, true, true, false, false, false, false,
  true, false, false, false, true, true, true, true, false, false, true,
  false, false, true, 2, NOW(), NOW()
);

Step 4 — Trigger deploy via API (works even though app CRUD is 404):

. .\.env.local.ps1
$h = @{ Authorization = "Bearer $env:COOLIFY_TOKEN" }
Invoke-RestMethod "$env:COOLIFY_API_URL/deploy?uuid=<app_uuid>&force=true" -Headers $h
# Returns: { deployments: [{ deployment_uuid: "..." }] }

Monitor: GET /deployments/<deployment_uuid> → status goes queued → in_progress → finished.

8.4 Stuck deployment queue — how to unblock

When a deployment fails (e.g. missing application_settings), the queue row stays in_progress forever and blocks ALL subsequent deploys of that app.

Diagnose:

SELECT id, status, created_at FROM application_deployment_queues
  WHERE application_id = '<app_id>' ORDER BY id DESC LIMIT 5;

Fix:

-- Mark the stuck job as failed
UPDATE application_deployment_queues SET status = 'failed', updated_at = NOW()
  WHERE id = <stuck_id>;
-- Delete ALL queue entries for the app and start clean
DELETE FROM application_deployment_queues WHERE application_id = '<app_id>';

Then restart the Coolify queue worker (so it picks up new jobs):

pct exec 102 -- docker exec coolify php artisan queue:restart

Wait 5 seconds, then trigger a fresh deploy via the API.

8.5 Reference: existing GitHub-source apps on this instance (verified 2026-07-27)

id	name	repo	build_pack	source_id
43	baserow	baserow/baserow	dockercompose	3
46	estación-de-documentos	urieljarethbusiness-cpu/Estación-de-Documentos	dockercompose	3
47	cotizador	urieljarethbusiness-cpu/cotizador	dockercompose	3
48	firecrawl	firecrawl/firecrawl	dockercompose	3
50	audio-a-texto	urieljarethbusiness-cpu/Audio-a-Texto	dockerfile	3
51	agendamax	urieljarethbusiness-cpu/agendamax	dockerfile	3

GitHub App: id=3, uuid=miakw8c0vthrtweroh9kzy1t, app_id=3266541, installation_id=121211999. Public GitHub source: id=0, uuid=yyq0od5j3xkdf8twc28n6coh. Standalone Docker (destination): id=0, uuid=jxlxd62k0d8owl6orgkjj1ob, network=coolify. Server: id=0, uuid=l10mdaago0z605pga93gl6cz, name=localhost.

8.6 Gitea repos as build source — not directly supported via this path

Coolify's GitHub App source only works with GitHub.com repos. For a Gitea-hosted repo:

• Option A (recommended): mirror to GitHub (git remote add github …; git push github main), then create the Coolify app from the GitHub repo. This is what we did for AgendaMax — the primary repo stays on Gitea, GitHub is just a deploy mirror.
• Option B: create a "Private repository (deploy key)" app via the web UI — set git_repository to the Gitea SSH URL and private_key_id to an SSH key with Gitea access. Gitea SSH is on port 22222 (mapped from container 22). This path needs UI credentials and hasn't been tested on this instance.

Re-verified 2026-07-31 (prompt-gallery-e3): Option A still works, and the GitHub App (source_id=3) reaches a freshly created private repo under urieljarethbusiness-cpu with no extra configuration — the installation covers all repos on the account, so there is nothing to click between POST /user/repos and the first deploy.

8.7 Env vars are Laravel-encrypted — raw SQL INSERT kills the deploy

Verified 2026-07-31 deploying prompt-gallery-e3 (Next.js 16 + node:sqlite). §8.3 creates the app but says nothing about env vars; this is the trap.

environment_variables.value carries Laravel's encrypted cast. Insert a plaintext value with psql and the app row looks perfect, but every deploy dies after cloning and reading the Dockerfile, with no hint about which field is at fault:

Deployment failed: The payload is invalid.
Error type: Illuminate\Contracts\Encryption\DecryptException
Location: /var/www/html/vendor/laravel/framework/src/Illuminate/Encryption/Encrypter.php:244

A correctly stored value is base64 JSON (eyJpdiI6…, {iv,value,mac,tag}), 200–300 chars for short secrets. The APP_KEY never leaves the container, so don't try to encrypt from the outside — create the rows through Eloquent instead:

# write PHP locally -> scp to host -> pct push -> docker cp -> tinker
pct push 102 /tmp/envs.php /tmp/envs.php
pct exec 102 -- docker cp /tmp/envs.php coolify:/tmp/envs.php
pct exec 102 -- docker exec coolify php artisan tinker --execute='include "/tmp/envs.php";'

$fila = new \App\Models\EnvironmentVariable();
$fila->key = 'SESSION_SECRET';
$fila->value = '…';                                   // se cifra al guardar
$fila->resourceable_type = \App\Models\Application::class;
$fila->resourceable_id = 53;
$fila->save();                                         // uuid se autogenera

Three gotchas found doing this:

• DELETE then CREATE — never UPDATE a plaintext row. Saving over an existing bad row throws the same DecryptException first, because a model hook reads value before your assignment is written. Wipe the rows with SQL, then create them via Eloquent.
• The column is is_buildtime, not is_build_time (and is_runtime/is_buildtime are set by the model — don't touch them).
• Two rows per key is correct. Coolify mirrors every variable into a preview copy (is_preview = true), so ids come in pairs. The pre-existing apps show the same shape; it is not a duplicate-insert bug.

8.8 Persistent volumes by this path

local_persistent_volumes uses resource_type / resource_id — not the resourceable_* names that environment_variables uses. Plain SQL is fine here (no encrypted columns):

INSERT INTO local_persistent_volumes (name, mount_path, resource_type, resource_id, uuid, created_at, updated_at)
VALUES ('<app_uuid>-galeria-datos', '/app/data', 'App\Models\Application', <app_id>, '<uuid24>', NOW(), NOW());

Insert the volume before the first deploy. New-CoolifyAppViaDB.ps1 triggers a deploy as its last step, so for an app that needs env vars or a volume, do the whole set of rows in one data-modifying-CTE statement and trigger GET /deploy?uuid= yourself — otherwise the first build is guaranteed to fail and you burn ~15 min of Next.js build time.

8.9 Script bugs fixed 2026-07-31 (were silently breaking repo creation)

• " 20[04-9] " accepted 200 and 204–209 but rejected 201/202/203 — a character-class typo for [0-9]. Present in both gitea_skill/scripts/Invoke-GiteaApi.ps1 and deploy_skill/scripts/Invoke-GitHubApi.ps1. Every repo/hook/release creation returns 201, so it threw after successfully creating the resource — leaving a created repo and an aborted pipeline. Both fixed to " 20[0-9] ".

• Invoke-GitHubApi.ps1 BOM bug (documented in §7.3) is now actually fixed — the POST body is written with [IO.File]::WriteAllText(..., UTF8Encoding($false)).

• git push to GitHub: Authorization: Bearer <PAT> does NOT work. git-over-https wants Basic. For a headless push that leaves no token on disk:

  $basic = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes("x-access-token:$env:GITHUB_TOKEN"))
  git -C $repo -c "http.extraHeader=Authorization: Basic $basic" -c "credential.helper=" push -u github main
  

  (Sync-GiteaRemote.ps1's token <T> header is right for Gitea; it is not for GitHub.)

• New-GitHubRepo.ps1 forces auto_init = $true, which puts a commit on the remote and makes pushing existing history a non-fast-forward. For a mirror of a repo that already has history, POST /user/repos yourself with auto_init = $false.

9. Apps with a sibling database (verified 2026-07-31, demospa — Next.js 15 + Prisma + MySQL 8)

Deployed serenidad-spa as demospa.urieljareth.org (app id 54) with a Coolify-managed MySQL. New, reusable facts beyond §8:

9.1 POST /databases/mysql WORKS — the database API is not part of the 404 namespace

Unlike /applications/*, the /databases/* namespace responds. POST /databases/mysql with {server_uuid, project_uuid, environment_name, environment_uuid, destination_uuid, name, image, mysql_root_password, mysql_database, mysql_user, mysql_password} returns {uuid, internal_db_url}. The DB's internal hostname IS its uuid — so DATABASE_URL=mysql://user:pass@<db-uuid>:3306/<db>. Both the DB and dockerfile-pack apps land on the external coolify network, so service-name DNS works with no extra wiring.

Two gotchas:

• instant_deploy: true did NOT start it. The resource was created with status=exited:unhealthy and no container. GET /databases/{uuid}/start then answered 400 {"message":"Database is already running."} (the status field lies). What actually started it: GET /deploy?uuid=<db-uuid> — the same trigger used for apps.
• Never call /start and /deploy back to back. Doing so recreated the container mid-initialization and left a partial datadir; MySQL then crash-looped forever on --initialize specified but the data directory has files in it. Recovery = docker compose down in /data/coolify/databases/<uuid>/, docker volume rm mysql-data-<uuid>, then one clean up -d. Same "don't queue deploys" rule as §4.

9.2 MySQL 8 init on this host takes ~7 min, and start_period is hardcoded to 5s

Coolify's generated DB compose sets healthcheck.start_period: 5s, but a first-time MySQL 8 init here takes minutes (InnoDB init alone ~53s). Patch /data/coolify/databases/<uuid>/docker-compose.yml to a longer start_period before the first up -d.

Do not trust mysqladmin ping as a readiness signal. The MySQL entrypoint runs a temporary server with --skip-networking while it creates the database and user, so the socket answers (and Coolify reports healthy) while TCP still refuses connections. The log even prints ready for connections ... port: 0 for that temp server. Gate on TCP: mysqladmin -h127.0.0.1 -uroot -p"$MYSQL_ROOT_PASSWORD" ping, and confirm a ready for connections ... port: 3306 line.

9.3 Prisma + MySQL 8: pin mysql_native_password

Add to the DB service's compose command: --default-authentication-plugin=mysql_native_password --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci. 8.0.46 only warns that the flag is deprecated. Set it before the first init so the created user gets native auth (afterwards the user already exists and the env vars are ignored — you'd need ALTER USER).

9.4 The rolling-update healthcheck window is ~2 minutes — do slow work in the background

Coolify polls the container's Dockerfile HEALTHCHECK ~6 times at 30s and aborts with "New container is not healthy, rolling back" if it hasn't passed. An entrypoint that runs prisma db push + seed before starting the server will lose this race on a cold DB — the build succeeds and the deploy still fails.

Working shape: start the server in the foreground and run DB preparation in a background subshell, so the health endpoint answers in seconds while data lands moments later.

preparar_base_de_datos() { ...db push retry loop...; ...seed...; }
preparar_base_de_datos &
exec "$@"          # el proceso en segundo plano sobrevive al exec

Keep the Dockerfile healthcheck tight (--start-period=10s --interval=10s) so it turns healthy inside Coolify's window, and give the health endpoint no DB dependency.

9.5 inject_build_args_to_dockerfile bakes every env var into the image

New-CoolifyAppViaDB.ps1's settings row sets this true, so Coolify rewrites the Dockerfile with an ARG/ENV per variable — the build log fills with SecretsUsedInArgOrEnv: ... (ARG "AUTH_SECRET") and the secrets end up in image layers, violating §2.5 of AGENTS-coolify-apps.md. Set it to false unless the app genuinely needs build-time vars (a Next.js app only does if it reads NEXT_PUBLIC_*, which are inlined at build time — grep the source before deciding).

9.6 Pushing to the GitHub mirror auto-triggers a deploy

The settings row also sets is_auto_deploy_enabled = true, and Coolify's GitHub App gets push webhooks for the whole account — so git push github main queues a deploy on its own. Expect an extra in_progress row in application_deployment_queues; clear it (§8.4) before triggering your own, or just let the automatic one run.

9.7 Next.js: pages that query the DB break docker build

Any App Router page that hits the database without export const dynamic = "force-dynamic" is prerendered during next build, where no DB exists. Pages reading cookies() are already dynamic; public landing/catalog pages usually are not. Validate locally with a deliberately unreachable DATABASE_URL — the build must still finish (Prisma logs errors but they are non-fatal once every DB page is ƒ).

sharp needs no special handling: npm ci on node:22-alpine resolves @img/sharp-linuxmusl-x64 as long as the lockfile was generated with all platform variants (it is, by default), so remote-image optimization works.

---

10. Correcciones verificadas 2026-08-27 (despliegue de escudoverde-site)

10.1 /deploy ahora exige POST, no GET

El §8.4 dice GET /deploy?uuid=&force=true. Hoy responde 405 Method Not Allowed. Con -Method POST funciona y devuelve el deployment_uuid normalmente:

Invoke-RestMethod "$env:COOLIFY_API_URL/deploy?uuid=<app_uuid>&force=true" -Headers $h -Method POST

GET /deployments/{uuid} sigue funcionando igual para el seguimiento.

10.2 El GITHUB_TOKEN de .env.local.ps1 está caducado

New-GitHubRepo.ps1 falla con 401 Bad credentials. El token que sí sirve es el que guarda el Administrador de credenciales de Windows para github.com (cuenta urieljarethbusiness-cpu, scopes gist, repo, workflow). Se recupera sin exponerlo:

TOK=$(printf "protocol=https\nhost=github.com\n\n" | git credential fill | sed -n 's/^password=//p')

Conviene rotar el de .env.local.ps1 o hacer que los scripts caigan a git credential fill.

10.3 Imágenes nginx sin root: el orden dentro del RUN importa

El builder de Coolify corre el docker build sin DAC override para root. Consecuencia concreta con la receta habitual de nginx no-root:

• Si nginx -t va después del chown de /tmp al usuario nginx, el build falla con open() "/tmp/nginx.pid" failed (13: Permission denied) — aunque en local funcione.
• nginx -t crea el fichero pid. Si se queda en la imagen, pertenece a root y el contenedor no arranca al hacer USER nginx.

Orden que funciona en ambos lados:

RUN set -eux; \
    mkdir -p /tmp/nginx/client_body /tmp/nginx/proxy; \
    nginx -t -c /etc/nginx/nginx.conf; \
    rm -f /tmp/nginx/nginx.pid; \
    chown -R nginx:nginx /tmp/nginx /usr/share/nginx/html /var/cache/nginx; \
    chmod 0777 /tmp/nginx
USER nginx

10.4 immutable + nombre de fichero fijo = despliegue invisible

Con Cache-Control: public, max-age=31536000, immutable sobre /assets/, Cloudflare sirvió el CSS anterior durante todo el despliegue siguiente (cf-cache-status: HIT, Age: 1008). El contenedor tenía la versión nueva; el usuario veía la vieja.

Regla para cualquier app estática en esta instancia: o los assets llevan hash en el nombre, o el HTML los referencia con ?v=<hash del contenido>. Sin eso, immutable es una trampa: el despliegue "funciona" y no cambia nada visible.

---

11. Re-verificación completa 2026-08-29 (v4.3.14): el 404 de /applications era Cloudflare

Auditoría integral con SSH + APIs validadas. La instancia corre ahora 4.3.14 (GET /version), actualizada desde 4.3.10. Hallazgos, en orden de importancia:

11.1 La API está COMPLETA — el 404 de /applications/* lo impone Cloudflare, no Coolify

Mismo token de siempre, misma ruta, dos caminos:

Llamada	Resultado
https://coolify.urieljareth.org/api/v1/applications (público, vía Cloudflare)	404
http://192.168.0.117:8000/api/v1/applications (origen, LXC 102)	200 (JSON completo)
/github-apps	404 vía CF · 200 vía origen
/dockerfiles, /sources, /notifications, /private-keys	404 en ambos → no existen como rutas; los nombres correctos están en el spec (ver 11.2)
/version, /resources, /servers, /projects, /teams, /services, /databases, /deployments, /security/keys	200 por ambas vías

Todo el diagnóstico de §1 ("API parcial", v4.1.2) era este mismo bloqueo de Cloudflare, no un recorte de la API. Las secciones §3 (Playwright) y §8 (DB INSERT) siguen siendo fallbacks válidos, pero la vía API debería funcionar llamando al origen ($env:COOLIFY_API_URL_ORIGIN). No se ha ejercitado end-to-end un POST /applications/* contra el origen todavía — verificarlo en el próximo deploy antes de jubilar el flujo UI. El fix de raíz es corregir la regla del edge en el dashboard de Cloudflare.

11.2 La superficie real (del openapi.yaml del propio contenedor)

Dentro del contenedor: /var/www/html/openapi.yaml. Rutas declaradas en 4.3.14 (extraídas el 2026-08-29):

• /applications + /applications/{public, private-github-app, private-deploy-key, dockerfile, dockerimage}
• /databases + por motor: {postgresql, clickhouse, dragonfly, redis, keydb, mariadb, mysql, mongodb}
• /deployments, /deploy, /destinations, /servers (+ /import, /digitalocean, /hetzner, /vultr)
• /github-apps, /gitlab-apps, /security/keys, /s3-storages, /tags
• /notifications/{email, discord, slack, telegram, pushover, webhook}
• /cloud-init-scripts, /cloud-tokens
• /projects, /projects/{uuid}/environments, /resources, /services
• /team, /teams, /team/envs, /team/members
• /version, /health, /enable, /disable, /mcp/enable, /mcp/disable

GET /docs (Scalar UI) redirige a /login — requiere sesión web; para el contrato exacto, leer el YAML del contenedor: pct exec 102 -- docker exec coolify cat /var/www/html/openapi.yaml.

11.3 POST-only confirmado como comportamiento de Coolify (no de Cloudflare)

Contra el origen, GET /deploy?uuid=fake → 405 {"message":"This endpoint has changed to a POST request."}. Es un cambio real de Coolify v4.2 (changelog: los endpoints de estado pasaron a POST-only), igual que §10.1. Otros cambios relevantes de 4.2→4.3: endpoints de logs para db/servicio/contenedor, settings de application incluidas en las respuestas, soporte MCP (read-only) y build pack Railpack (beta, 4.3.1). Breaking: se eliminó el endpoint deprecado de aplicación Docker Compose — usar POST /services. Fuentes: coolify.io/changelog y github.com/coollabsio/coolify/releases.

11.4 Estado de credenciales verificado el 2026-08-29

Credencial	Estado
SSH [email protected] y [email protected] con keys/proxmox_ed25519 (≡ ~/.ssh/coolify_key, sin passphrase)	✅
API Proxmox: token root@pam!openclaw (único en token.cfg)	✅ 200 /version + /cluster/resources — ya en .env.local.ps1
API Coolify: COOLIFY_TOKEN (team root)	✅ 200 — v4.3.14
Gitea: token de urieljareth	✅ 200 /api/v1/user
GitHub: PAT ghp_NFy4… del .env	❌ caducado (401) — reemplazado en .env.local.ps1 por el token vivo del Credential Manager de Windows (gho_…, cuenta urieljarethbusiness-cpu, scopes gist, repo, workflow)
Cloudflare API	❌ sin token — gestionar túnel desde el dashboard
UI Coolify: [email protected] / password habitual	⚠️ probable (heredado de la instancia vieja 192.168.1.175, ver C:\Users\Uriel Jareth\coolify-agent\coolify-agent-skill.md), sin verificar

Inventario completo de llaves SSH de la máquina (incluidas las corruptas de SiteGround y la de la instancia antigua): ACCESS.md en la raíz del repo.