Operations: probes, correlation and doctor

Three things an operator needs and an app never declares: whether the process is alive, whether it can do work, and which line in the log belongs to the complaint someone just filed.

Liveness is not readiness

Route	Question	Touches the database	Answer
GET /healthz, GET /api/health	is this process still serving?	no	always 200 {"ok":true,"status":"alive"}
GET /api/ready, GET /readyz	can it do useful work now?	yes	200 {"ok":true,"status":"ready"} or 503 {"ok":false,"status":"unready","reason":"database"}

They are separate because the answers lead to opposite actions. A failed liveness probe means restart this process; a failed readiness probe means stop sending it traffic and leave it alone. Restarting every application server because Postgres went down does not bring Postgres back.

Both are reachable without credentials, and an invalid Authorization: token … is ignored on them — on any other route it answers 401 before the handler runs, which would otherwise let a monitoring agent's expired key report a healthy process as dead.

A trailing slash is accepted on all four (/readyz/ works). That is not tidiness: an unmatched path falls through to the desk, which answers 200 with index.html, so a probe configured with a stray slash would otherwise pass forever — including while the database is down.

The readiness body is a boolean on purpose. No latency, no version, no queue depth, no database error text. Latency is a feedback channel for anyone measuring their own effect on the site; queue depth leaks business volume and says when the site is under load; the version says which advisories apply; a Postgres error string carries host, port, user and database name. An orchestrator reads the status code and nothing else, so the disclosure would buy nobody anything.

Readiness is memoised for one second. A probe answered every few seconds by every replica would otherwise be a database round trip anyone can trigger at will. The cost is that a database that has just died is reported ready for up to a second.

Probing from a container

A site's image starts FROM ghcr.io/jrvidotti/ddcore:<X.Y>, the official image of the minor series it follows (ddcore deploy docker writes that Dockerfile). Leave the HEALTHCHECK out of it and declare the probe in the orchestrator instead: the entrypoint is the binary itself, and the same image runs ddcore migrate and ddcore doctor as one-shot containers that a baked healthcheck would mark unhealthy.

# docker compose
healthcheck:
  test: ["CMD", "wget", "-qO-", "http://localhost:8090/readyz"]
  interval: 10s
  timeout: 3s
  retries: 3

# kubernetes
livenessProbe:  { httpGet: { path: /healthz, port: 8090 }, periodSeconds: 10 }
readinessProbe: { httpGet: { path: /readyz,  port: 8090 }, periodSeconds: 5 }

The health report

GET /api/health/report is the same picture with the numbers in it, and it requires the System Manager role. It answers { data: … } like any other endpoint:

{ "status": "warn",
  "version": "0.4.1",
  "database": { "ok": true, "latencyMs": 2.1, "conns": 3, "idle": 2, "maxConns": 10 },
  "queue": { "queued": 12, "runnable": 11, "running": 2, "stalled": 0,
             "failedInWindow": 1, "doneInWindow": 340, "cancelledInWindow": 0,
             "oldestQueuedSeconds": 252, "longestRunningSeconds": 3, "windowMinutes": 15 },
  "errors": { "inWindow": 3, "windowMinutes": 15,
              "latest": [{ "id": "…", "method": "job:demo.tasks.sweep", "requestId": "job:412" }] },
  "scheduler": { "enabled": true, "entries": 4 },
  "warnings": ["queue backlog: 214 runnable jobs (limit 100)"] }

status is ok, warn or down. Only down — the database not answering — changes an HTTP status code anywhere; thresholds produce warn and never a 503, because a readiness probe that failed on a deep queue would have the orchestrator restart the very workers draining it.

Two numbers in queue are easy to confuse. queued is every job waiting; runnable is the subset whose runAfter has arrived. Only runnable means behind — a job enqueued to run tomorrow is not a backlog, and it does not age the queue either. stalled is running jobs whose lease expired: the worker holding them is gone, and it is the same set a live worker would put back.

cancelledInWindow is counted apart from failedInWindow on purpose: a job somebody stopped is not a fault, and adding the two together would send an operator hunting a bug that was in fact a decision.

Request correlation

Every request is given an identifier, and the same string appears in four places:

• the X-Request-Id response header;
• the requestId field of any error body;
• the access log line for that request;
• the request_id column of any Error Log row it produced.

An inbound X-Request-Id is honoured when it is 8–64 characters of A–Z a–z 0–9 - _ . — which admits UUID, ULID and a W3C traceparent. Anything else is replaced with a generated id rather than escaped: the value reaches a log line and a text column, so a newline in it would forge a log entry and a megabyte of it would flood the column.

Server-side app code reads it from the session:

export const sync = whitelisted(() => {
  ddcore.log.info(`syncing for ${ddcore.session.requestId}`);
});

ddcore.session.requestId is empty in a job, a migration or a test — there is no request behind that work, and the emptiness is information. A failed job run still gets a handle of its own (job:<id>) on the Error Log row it writes.

App code can write a row of its own and go on: ddcore.errorLog.record(e, { context }) files it on a transaction of its own, so a job that catches a failing step keeps the other steps' work and still leaves the failure where operators look. The row carries the request's id, or job:<id> in a job's body, and ddcore doctor counts it like any other.

In the desk, a DDCoreError carries requestId, and the toast for a 500 shows it in brackets so a user can quote it.

The access log

One line per request, at most:

level=INFO msg=request id=9f1c… method=GET path=/api/resource/Task status=200 ms=14 user=ana@x.com ip=10.0.0.4

Only the path is logged, never the query string: a list filter carries personal data and a recovery link carries a token.

The level is chosen so the log stays readable. A single desk page is around thirty requests through the same chain — the SPA fallback, every asset, every file — so a 200 on a static asset is debug, an /api/… request is info, a 4xx is warn, a 5xx is error, and anything slower than ops.slowRequestMs is warn wherever it came from. DDCORE_DEBUG logs everything.

The server writes the log to stdout, never to stderr: a platform that captures both streams — Railway, Cloud Run, Kubernetes — reads a line on stderr as an error by definition, so an INFO access line would arrive painted red and the level policy above would stop meaning anything. Every other command keeps its log on stderr, because there stdout is the command's own output — an exported NDJSON, a printed key, doctor --json, the MCP JSON-RPC stream.

DDCORE_LOG_FORMAT picks the shape: json for objects, text for lines. With nothing set the process asks the log's destination — a terminal gets text, anything else (a pipe, a file, a container's log stream) gets JSON, since a collector has to parse a text line and grouping by request id, most of what the id is for, stops working. Deployed, that means JSON without configuring anything, and a collector reads the level from the level field of each object instead of guessing it from the stream.

A panic is caught, logged with its stack, written to Error Log, and answered with the ordinary JSON error envelope carrying the id. The panic text stays on the server; the caller gets the id, which is what joins the two.

Thresholds: the ops block

These live in ddcore.json, next to auth, because they are a decision the site makes once and keeps everywhere — staging has to call a backlog a backlog exactly like production, or the rehearsal is not a rehearsal.

{
  "ops": {
    "windowMinutes": 15,
    "queueBacklog": 100,
    "queueAgeSeconds": 300,
    "jobFailures": 5,
    "errorLogEntries": 20,
    "readyTimeoutMs": 2000,
    "slowRequestMs": 2000,
    "jobRetentionDays": 7,
    "jobRetentionFailedDays": 30,
    "webhookRetentionDays": 30,
    "auditRetentionDays": 0,
    "backupMaxAgeHours": 0
  }
}

Every field may be omitted; what is left out keeps the value above. A threshold written as zero or negative is refused at boot rather than quietly replaced — a zero threshold is not "no threshold", it is an alarm that fires on the first job.

The retention windows are the exception, and deliberately so: there zero means keep forever, because "delete nothing" is a real answer for a table of jobs, webhook deliveries or audit events, and the thresholds above have no such reading. There are four such windows — jobRetentionDays, jobRetentionFailedDays, webhookRetentionDays and auditRetentionDays — and only a negative retention window is refused.

backupMaxAgeHours is the other zero-means-off field: when set, the health report and doctor warn once the newest successful ddcore backup is older than that many hours (see backup.md).

ddcore doctor

ddcore doctor [--json] [--strict] [--window N] [--no-update-check]

It reports the version, the site, the database (with the DSN's password redacted), the apps and doctypes — each app with its own version and the ddcore range it declares (see conventions.md) — pending DDL, undeclared structures, pending patches, applied renames, the queue, the Error Log, the scheduler, the workers, mail, outgoing webhooks (on or off, how many are enabled, retrying, and failed in the last day), the public URL, the session policy, the thresholds in force, and the names of the configured secrets — never their values, because this report gets pasted into issues and chat windows. The app settings an app reads with ddcore.env (DDCORE_APP_*) are on an app env line with their values: none of them is a secret.

It also reports where file bytes live (local <dataDir>/files, or the S3 endpoint, bucket, prefix and presigned-link lifetime — never the credentials; see storage.md) and single sign-on: the configured provider ids and whether password sign-in is on, or no providers. Only --json carries each provider's callback URL. Neither check contacts the bucket or the identity provider beyond the discovery probe described in auth.md.

It also reports maintenance mode (and warns while it is on), which core and app versions last migrated the database, and the newest ddcore backup with any failures since (see backup.md).

It also asks GitHub for the newest published release and warns when this binary is behind it:

version:    v0.14.0
            v0.15.0 is available
...
warning:    a newer ddcore release is available: v0.15.0 (this binary is v0.14.0)

It is a warning, so --strict fails on a stale binary. The lookup runs before the engine is loaded, because a binary too old for the apps in front of it is exactly the case where the answer helps and the report has already given up by then. It asks at most once an hour, it never reports what it could not reach — an air-gapped site is a normal site, not a finding — and it says nothing at all when this binary is not a release (a dev build has no version to compare). --no-update-check skips the lookup for one run; DDCORE_UPDATE_CHECK=off skips it for every run, on a machine with no outbound internet access or wherever the request is unwelcome. What actually changed is the changelog, embedded in the binary and served over MCP as ddcore://changelog — the whats_new tool returns the part of it above the running version.

It also reports the vault: whether DDCORE_SECRET_KEY — the environment variable the vault's master encryption key is derived from — is configured, how many secrets ddcore_vault holds, and their key names. As with an integration secret, a value is never printed, but a key name is (see vault.md). Secrets existing without a configured key is a warning, since every one of them fails to decrypt until the key is set again. With the key set, it opens every secret in every tenant and warns about those no key opens and, during a rotation of the key, about those still under DDCORE_SECRET_KEY_PREVIOUS (run ddcore vault rekey, or the MCP tool vault_rekey) — and, once none are, that the previous key can be removed.

It works with the database down. The probe runs before, and independently of, the engine, so an unreachable database produces a report that says so plus every section that needs no database — not a bare error: line. That is the situation the command exists for.

Exit 1 means a critical finding: the database is unreachable, the apps could not be loaded, a migration is refused, or an undeclared structure still holds data. apps: could not be loaded covers every refusal to load, an app whose ddcore range excludes this binary among them; the reason is on the line below it, and the sections that need a loaded engine — the per-app versions and ranges among them — are missing from that report. The rollback guard of backup.md has a critical of its own, this binary is older than the release that migrated the database, since that one is about the binary and not about app code. The likeliest cause in the field is a binary built without -ldflags, which reports the default version 0.1.0 and so refuses every app pinning a real release: check the version line before the apps (see conventions.md). Warnings — pending DDL, a backlog, recent failures — exit 0 unless --strict, because this is mostly run by a person reading it rather than a script failing on it.

--json prints the same report as a JSON object, for a cron that wants to feed it somewhere.

Job administration

The queue is administered from the CLI and from a small HTTP surface. Both read and write ddcore_job, which is a framework table and not a DocType, so it has no list view and no /api/resource route.

A job's user is who queued it. run_as is set only on a job queued with ddcore.enqueue(..., { runAs }) or by a scheduler entry with runAs: the user whose roles and access scopes the job runs under (see scopes). jobs scheduled shows such an entry as method (as user).

ddcore jobs list [--status s] [--queue q] [--method m] [--user u] [--since 1h] [--limit N] [--json]
ddcore jobs show <id> [--json]
ddcore jobs stats [--window N] [--json]
ddcore jobs retry <id>... | --failed [--queue q] [--since 1h] [--limit N] [--force] [--dry-run]
ddcore jobs cancel <id>...
ddcore jobs purge [--done-days N] [--failed-days N] [--dry-run]
ddcore jobs scheduled
ddcore jobs run <fn>
ddcore jobs work [--queue q[,q]] [--workers N]

jobs list shows the queue. What it used to print — the cron declarations — is now jobs scheduled.

Over HTTP, all of it requires the System Manager role: GET /api/jobs (filters as above, limit, start, with_count=1), GET /api/jobs/{id}, GET /api/jobs/stats, POST /api/jobs/{id}/retry, POST /api/jobs/{id}/cancel and POST /api/jobs/purge. The MCP tools list_jobs, get_job, retry_job, cancel_job and purge_jobs are the same operations.

A job's arguments and result are never returned over HTTP or MCP. They are printed by ddcore jobs show and nowhere else. This is not tidiness: the framework's own mail job is enqueued with the rendered message, so a queued password-reset job holds a working recovery link in args.html. The CLI already requires the database; a browser session should not become a second way to read that.

A job carries the request_id of the request that queued it, so the work a request set off can be found from the request and the other way round. Jobs the scheduler queues have none — nothing asked for them.

Worker pools

workers in ddcore.json is how many workers a process runs (ddcore start, ddcore dev, ddcore jobs work). A number — "workers": 2, the default — is that many workers serving every queue, oldest runnable job first. An object is a pool per queue:

"workers": { "default": 2, "bot": 2 }

• a named pool (bot) claims the jobs of its own queue only;
• default, which the object must name, claims the default queue and every queue no other pool is named for — the scheduler's (scheduler) among them;
• a job goes to a queue with ddcore.enqueue(method, args, { queue: "bot" }).

So a queue of slow jobs, or of jobs waiting on ddcore.db.lock, fills its own pool and leaves the others free. A size may be 0 — the queue is named, so the default pool leaves it alone, and another process serves it — but not negative. DDCORE_WORKERS overrides the setting with a number. The VM pool, and so the database connections a process may use, grows with the total.

ddcore jobs work --queue bot starts only the pools it names, at their size in ddcore.json; --workers N sizes each of them instead, and is required for a queue the file names no pool for. That is how a queue gets a process — or a container — of its own:

ddcore jobs work --queue bot --workers 4

Name such a queue in ddcore.json as well ("bot": 0 will do): a queue it does not name is also served by the default pool of every other process. ddcore doctor prints the total and each pool (workers and workerPools in --json).

A job that would wait on a lock can avoid holding a worker at all: ddcore.db.tryLock(key) returns false instead of waiting, and the job queues itself again with runAfterSeconds (see controller-api).

Statuses

queued → running → done, failed or cancelled. A failing job goes back to queued while attempts remain, after a thirty-second pause; enqueue takes maxAttempts to change how many that is, and backoff: "exponential" to make the pause double with each attempt — 30 seconds, 1, 2, 4 minutes and so on, never more than an hour — which is what work talking to somebody else's server wants. A retried job keeps its backoff. Outgoing webhooks use it. The status values are also what --status accepts, so they stay in English on a translated site.

An attempt is not a run. A run interrupted by a worker shutting down or by maintenance gives its attempt back (see "Stopping the process" below), so maxAttempts: 1 stops retries after a failure but does not mean "runs at most once": the job runs again, with the same attempt, and whatever it did outside the database before it was interrupted — an HTTP call, a message — has already happened. The job's starts counts every run a worker began, given-back ones included, and never goes down. A body doing work that must not be repeated reads it from ddcore.job.current() and checks first:

export function settle(args: { payment: string }) {
  const job = ddcore.job.current()!;
  if (job.starts > 1) {
    // an earlier run began: ask the other system whether it already settled
    const done = ddcore.http.get(`https://erp.example.com/payments/${args.payment}`).json();
    if (done.settled) return markSettled(args.payment);
  }
  ddcore.http.post("https://erp.example.com/settle", { payment: args.payment });
  markSettled(args.payment);
}

ddcore.job.current() returns the same JobInfo the callbacks get, in the body and in either callback, and null outside a job — a request, a migration, a function run by ddcore jobs run.

Deduplicating with uniqueKey

enqueue takes uniqueKey to keep identical work from stacking — one job that drains a session's events, say, instead of one job per event:

ddcore.enqueue("myapp.services.bot.process", { session }, { uniqueKey: `myapp:session:${session}` });

While a job with that key is still queued, the call queues nothing and returns that job's id, so it is safe under concurrent requests (a partial unique index enforces it). A job that has been claimed — running or finished — no longer holds the key, so work arriving while it runs queues a new job and is not lost; a job going back to queued for a retry holds it again. The key is global, not per queue or method, so prefix it. Unlike webhooks.emit's key, which throws on a duplicate, a duplicate uniqueKey is silent. The skipped call's args are dropped, so the job must read what it needs when it runs.

Lifecycle callbacks

A job is one transaction the framework owns, so everything it writes becomes visible only when it ends: a Running written at its start commits together with whatever it writes last, and a Failed written before it throws rolls back with everything else. enqueue takes two callbacks for exactly this, each a method path like the job's own, each run in a transaction of its own:

// the controller that queues the work
afterInsert(doc) {
  ddcore.enqueue("myapp.services.importer.run", { name: doc.id }, {
    maxAttempts: 1,
    timeout: 900,
    onStart: "myapp.services.importer.started",
    onFailure: "myapp.services.importer.failed",
  });
},

// services/importer.ts
import type { JobInfo, JobFailure } from "@ddcore/sdk";

export function started(args: { name: string }, job: JobInfo) {
  ddcore.db.setValue("My Import", args.name, { status: "Running" });
}

export function failed(args: { name: string }, job: JobFailure) {
  if (!job.final) return; // a retry is coming
  ddcore.db.setValue("My Import", args.name, { status: "Failed", messages: job.error });
}

• onStart(args, job) runs on every attempt, after a worker claimed the job and before the body, and commits before the body starts. It counts toward the job's timeout. If it throws, the attempt fails like a body that threw: the body does not run, the usual retry rules apply, and onFailure is called.
• onFailure(args, job) runs after the body rolled back, and before the job_done event is published, so a form refreshing on that event reads what it wrote. job.reason is "error", "timeout" or "cancelled"; job.error is the message; job.final is false while another attempt is still coming. It is also called when a job is cancelled before any worker claimed it (job.attempt is 0) and when the worker running it died and its lease expired. It is not called when a worker shutting down interrupts a running job, or the site enters maintenance under one: the attempt is given back, nothing failed (see "Stopping the process" below). It has 30 seconds. If it throws, the error goes to the Error Log (method job:onFailure:<method>) and the job stays failed.
• job is { id, method, queue, attempt, maxAttempts, starts }, plus error, reason and final for onFailure. starts is how many times a worker began the job, given-back runs included (see "Statuses").
• Both run as the job's user with permissions ignored, like the body, and share its Error Log handle, job:<id>. A job queued with runAs runs all three as that user with permissions enforced, and job.runAs names them. A retried job keeps its callbacks.

There is no onSuccess: the body's own writes commit with it. Nothing here hands app code a commit — the callbacks are two more short transactions the framework owns.

Cancelling

Cancelling a queued job is immediate and final. Cancelling a running job records the request and answers cancelling, never cancelled: the worker acts on it within about five seconds, and the job may well commit in the meantime, in which case it is done and the request is simply recorded against it.

When a cancellation does land, the job's transaction rolls back — the VM is interrupted the same way the per-job timeout interrupts it, so nothing it wrote to the database survives. Effects outside the database do not roll back. A message already handed to an SMTP server has been sent, and an HTTP call already made has been made. For the same reason, a retry may repeat them. Nothing here is exactly-once, and a job whose external effects must not be repeated should be written to tolerate being run twice.

The interrupt reaches the job wherever it is waiting on somebody else's server: a cancellation, the job's timeout and a worker shutting down all cut the call in progress — ddcore.http.*, ddcore.files.save({ fromUrl }), ddcore.push.send, an external database query, the framework's own mail and webhook deliveries — and the job stops there. The job's timeout is therefore a bound on its outbound calls too, whatever timeout a single call names. A query on the site's own database is the exception: it runs to its end, so that the rollback can, and a job blocked in one stops when it returns. Requests are unaffected: an outbound call made while serving one is not cut when its client goes away.

A cancelled job is not retried however many attempts remain, and it writes no Error Log row: it is an administrative act, not a fault.

Stopping the process

Stopping a worker is neither a cancellation nor a failure. When ddcore start, ddcore dev or ddcore jobs work is told to stop (SIGTERM, or Ctrl-C), it:

1. stops the scheduler and claims no new job, while the HTTP server finishes the requests in flight;
2. lets the jobs already running finish and commit, for up to shutdownGraceSeconds (ddcore.json, or DDCORE_SHUTDOWN_GRACE_SECONDS; 30 by default);
3. interrupts the ones still running after that — their outbound calls included — and gives them back: each returns to queued with its attempt given back and no onFailure, so a rolling restart does not spend max_attempts on work that has nothing wrong with it;
4. exits once they have, waiting at most about ten seconds more for those writes.

Whatever stops the process must give it longer than the grace — Docker's stop_grace_period and Kubernetes' terminationGracePeriodSeconds — or it is killed before step 3, and a job it was running stays running until its lease expires two minutes later and is then failed like a job whose worker died. A second Ctrl-C ends the process at once, with the same consequence.

Retrying

Retry queues a new job that copies the method, arguments, queue, user and run_as; the failed row is kept as the record, with retried_as pointing at the new one, and the new row carries retry_of back. That keeps job:<id> naming exactly one execution, which is what makes an Error Log row worth looking up. A second retry of the same job is refused unless forced.

Retention

ddcore_job is swept daily by core.services.jobs.sweep, where the scheduler is enabled, and on demand by ddcore jobs purge. Two windows in the ops block:

Key	Default	Meaning
jobRetentionDays	7	how long a done job is kept
jobRetentionFailedDays	30	how long a failed or cancelled job is kept

Outgoing webhook deliveries have a window of their own, webhookRetentionDays (default 30), swept daily by core.services.webhooks.sweep; only Sent and Failed deliveries are removed. See webhooks.

Audit Event has its own window, auditRetentionDays (default 0, keep forever), swept daily by core.services.audit.sweep, where the scheduler is enabled. See audit.

Zero means keep forever, and it is the one place in ops where zero is not "unset" — failures are the evidence of what went wrong and some sites keep them indefinitely, and the same reasoning applies to audit events. A negative value is refused at boot. Only finished jobs are ever deleted; a queued or running job is never touched whatever its age.

Like the auth sweep, this is hygiene and never correctness. Nothing may depend on it having run: if it never ran, the table would only grow.

PDF rendering

The print endpoints' PDF renderer is configured by two environment variables, read on the first PDF request and kept for the life of the process:

• DDCORE_GOTENBERG_URL — the base URL of a Gotenberg service, for example http://gotenberg:3000. It takes precedence over everything else.
• DDCORE_PDF_COMMAND — a command that turns {in} (an HTML file) into {out} (the PDF), for example weasyprint {in} {out}.

With neither set, a local Chrome, Chromium, Brave or Edge is used if one is found; with none, the PDF endpoint answers 503. Changing either variable needs a restart. See print.md.

Maintenance, backup and restore

ddcore maintenance on|off pauses writes, workers and the scheduler on every process without failing readiness. A job already running is not spared: its writes are refused, which costs it an attempt and writes an Error Log row, so drain the queue (ddcore jobs stats) before pausing for a cutover. ddcore backup and ddcore restore produce and consume a checksummed archive of the database, stored files and configuration. ops.backupMaxAgeHours turns on a health warning for a stale backup. See backup.md.

What this does not do

There is no /metrics endpoint and no OpenTelemetry. The authenticated report and doctor --json are both scrapeable today, and a metrics endpoint with no scraper deployed is a product nobody uses.

Nothing records when the scheduler last ran, so scheduler.entries is what this build would install and not proof that a cron is alive.

There is no Desk screen for jobs: administration is the CLI and the API above. Jobs have no priority: within what a pool serves, the oldest runnable job goes first. Give a queue of long jobs a pool of its own (see "Worker pools") so it cannot delay the rest.