Runs & publishing

A run reads the staging set, diffs it against the last applied state and records the changes. Apply builds and publishes a new generation from a run. The Developer API drives the same sync runner and apply runner as every other connector — there is no second engine.

flowchart LR
    S[Staging set] -->|POST /runs| R[Run executes synchronously]
    R -->|GET /runs/:id/diff| D[Inspect changes]
    R -->|POST /runs/:id/apply| P[Enqueue async build + publish job]
    P -->|GET /runs/:id/apply| J[Poll job until succeeded/failed]
    J -->|GET /generation| G[Active generation]

Start a run

Runs execute synchronously — the staging set is already in the database, so discovery is a cheap read (no crawl). The response carries the terminal state.

curl -X POST "$CHATBOT_API/v1/sources/$SID/runs" \
  -H "Authorization: Bearer $CHATBOT_KEY"

{
  "run": {
    "id": "…", "status": "completed",
    "discoveryComplete": true,
    "changeCounts": { "new_page": 2, "removed_page": 1 }
  },
  "applyEligible": true
}

A run is eligible to apply only when it discovered the whole source (status = completed, discoveryComplete = true). The source must be active (not archived) to start a run, otherwise 409 source_inactive.

Inspect the diff (optional)

curl "$CHATBOT_API/v1/sources/$SID/runs/$RID/diff" \
  -H "Authorization: Bearer $CHATBOT_KEY"

Returns per-change details grouped by kind (new_page, unchanged, removed_page, …) plus totals — review before publishing.

Apply (publish)

Apply is asynchronous. Building a generation chunks and embeds the content and can take a while, so the request enqueues a job and returns 202 straight away — it never holds the connection open through the build.

curl -X POST "$CHATBOT_API/v1/sources/$SID/runs/$RID/apply" \
  -H "Authorization: Bearer $CHATBOT_KEY" \
  -H "Idempotency-Key: $(uuidgen)"

{
  "apply": { "id": "…", "runId": "…", "status": "queued", "resultStatus": null, "attempts": 0 },
  "poll": "v1/sources/<sourceId>/runs/<runId>/apply"
}

The POST is idempotent per run: re-POSTing returns the same job, never a second concurrent build. A 404 not_found is returned only when the run itself does not exist.

Poll for the result

GET the same URL until status is succeeded or failed:

curl "$CHATBOT_API/v1/sources/$SID/runs/$RID/apply" \
  -H "Authorization: Bearer $CHATBOT_KEY"

{
  "apply": {
    "status": "succeeded",
    "resultStatus": "published",
    "generationId": "…",
    "counts": { "create": 3, "update": 0, "delete": 0 }
  },
  "run": { "id": "…", "status": "completed" }
}

status	resultStatus	Meaning
queued / running	null	Build is enqueued / in progress — keep polling.
succeeded	published	New generation built and now serving (generationId + counts).
succeeded	already_applied	This run was already applied (idempotent).
succeeded	applied_no_change	Nothing to publish — the result is identical.
failed	not_eligible	Discovery was incomplete — start a fresh run.
failed	conflict / stale_base / stale_plan / lease_lost	Another change moved the base; transient states are retried automatically before this is reported.
failed	build_failed	Build failed; the previously published generation keeps serving unchanged.

Transient outcomes (apply_in_progress, stale_base, lease_lost, stale_plan) are retried automatically with backoff before a job is marked failed. A crashed worker leaves the job recoverable: it is reclaimed and retried once its lease expires.

Worker / dispatch. The build runs out of band. In a single self-hosted deployment the bundled devApplyWorker timer (and an immediate in-process sweep) drains apply jobs; in Azure, set DEV_API_BUILD_DISPATCH=queue to fan jobs out via the dev-apply-build queue. Either way the DB job table is the source of truth, so progress is guaranteed. See deployment-topology.md.

Apply is safe to retry with the same Idempotency-Key.

Check the active generation

curl "$CHATBOT_API/v1/sources/$SID/generation" \
  -H "Authorization: Bearer $CHATBOT_KEY"

{
  "generation": { "id": "…", "state": "published", "publishedAt": "…" },
  "head": { "activeId": "…", "previousId": "…" }
}

Recommended loop

1. Push items (single / batch / snapshot / delete).
2. POST /runs → confirm applyEligible: true.
3. (Optional) GET /runs/:id/diff to review.
4. POST /runs/:id/apply with an Idempotency-Key → 202 with a job.
5. Either poll GET /runs/:id/apply until terminal, or subscribe to webhooks (generation.published, run.failed, apply.failed) instead of polling.