Upgrade and data retention

Use this runbook for an existing deployment. A fresh install has no data to preserve; follow Getting started instead. Do not call an upgrade successful until the post-upgrade checks below have passed.

The contract is deliberately fail-closed: a missing backup, unavailable PostgreSQL, incompatible schema, or failed migration stops the upgrade. It must not continue by deleting a volume or starting a partially migrated control plane.

Before choosing an upgrade path

Record the current release, checkout SHA, Compose file, .env reference, selected Z.AI environment file, and the output of docker compose ps. Keep secret values out of the record. Confirm free space for three PostgreSQL dumps, volume archives, and a temporary restore. Keep the old checkout and image tags until the verification window is over.

An upgrade changes images and tracked configuration only. It does not require docker compose down --volumes. The ordinary docker compose down keeps named volumes; adding --volumes permanently removes the data represented by those volumes and is not an upgrade step.

Backup contract

Create a timestamped, access-controlled directory outside the checkout. The following database list is part of the contract:

Database	Data to preserve
forgejo	repositories, users, organizations, Issues, PRs, comments, Actions metadata, and LFS references
openface_metrics	measured views, downloads, time-series events, likes, and agent identities
openface_maintenance	webhook delivery and maintenance job state

In a v0.4.0 installation that has not completed the v0.5.0 migration, pipeline audit/history and reconciliation state are still stored in the legacy SQLite file /data/agents/pipelines/pipeline-audit.db inside the openface_agent-metrics-data volume. The openface_metrics dump does not contain that legacy state, so the volume archive above is part of the backup contract. After migration, the authoritative state is in the openface_pipeline schema of openface_metrics and is covered by that dump.

For v0.6.0, runner startup also creates the metric_events ledger and its indexes inside openface_metrics, then backfills existing view and like counters with stable idempotency keys. This is an automatic, non-destructive initialization rather than a manual data migration; keep the database dump and restore evidence because the ledger is not automatically pruned.

Before creating any dump or archive, open a maintenance window and quiesce every running Compose service that can write to the archived volumes. Leave only postgres running for pg_dump, and do not start the stopped services again until every dump and archive below has completed. This stops Forgejo, the Spaces runner, maintenance, Actions, and any enabled MCP writers as one consistent operation:

set -euo pipefail
mapfile -t running_services < <(docker compose ps --services --status running)
for service in "${running_services[@]}"; do
  if [[ "$service" != "postgres" ]]; then
    docker compose stop "$service"
  fi
done

The LXC restore helper also requires protected copies of the .env file, the selected Z.AI environment file, and gateway-certs.zip. Copy those files into the same access-controlled backup before dumping databases; do not put their contents in the manifest:

Set OPENFACE_BACKUP_DIR once before running the backup snippets below. Each snippet enables fail-fast mode and reuses that exact directory, so copying a snippet into a separate shell without the directory fails immediately.

set -euo pipefail
export OPENFACE_BACKUP_DIR="${OPENFACE_BACKUP_DIR:-/secure/openface-backups/$(date -u +%Y%m%dT%H%M%SZ)}"
backup_dir="$OPENFACE_BACKUP_DIR"
umask 077
mkdir -p "$backup_dir"
test -f .env
zai_config="${ZAI_AGENT_CONFIG:-$(sed -n 's/^ZAI_AGENT_CONFIG=//p' .env | head -n 1)}"
zai_config="${zai_config:-./maintenance-agent/zai.example.env}"
test -f "$zai_config"
test -d gateway/certs
install -m 600 .env "$backup_dir/openface.env"
install -m 600 "$zai_config" "$backup_dir/zai.env"
python3 - "$backup_dir/gateway-certs.zip" <<'PY'
from pathlib import Path
import sys
from zipfile import ZIP_DEFLATED, ZipFile

root = Path("gateway/certs")
with ZipFile(sys.argv[1], "w", compression=ZIP_DEFLATED) as archive:
    for path in root.rglob("*"):
        if path.is_file():
            archive.write(path, path.relative_to(root))
PY

Dump each database before changing the checkout:

set -euo pipefail
backup_dir="${OPENFACE_BACKUP_DIR:?Set OPENFACE_BACKUP_DIR before running the backup snippets}"
umask 077
mkdir -p "$backup_dir"
docker compose exec -T postgres sh -c 'pg_dump -U "${POSTGRES_USER:-openface}" -Fc forgejo' > "$backup_dir/forgejo.dump"
docker compose exec -T postgres sh -c 'pg_dump -U "${POSTGRES_USER:-openface}" -Fc openface_metrics' > "$backup_dir/openface_metrics.dump"
docker compose exec -T postgres sh -c 'pg_dump -U "${POSTGRES_USER:-openface}" -Fc openface_maintenance' > "$backup_dir/openface_maintenance.dump"

Archive the named volumes used by the deployment. Resolve the actual names with docker volume ls and do not substitute a broad host directory. The following preflight and archive block uses the same MCP detection for the named volume, bind archive, and credential files. It validates the complete MCP set before creating any archive, so a partial MCP backup fails closed:

set -euo pipefail
backup_dir="${OPENFACE_BACKUP_DIR:?Set OPENFACE_BACKUP_DIR before running the backup snippets}"
mcp_enabled="${OPENFACE_MCP_ENABLED:-0}"
compose_profiles="${COMPOSE_PROFILES:-$(sed -n 's/^COMPOSE_PROFILES=//p' .env | head -n 1)}"
case ",${compose_profiles}," in
  *,mcp,*) mcp_enabled=1 ;;
esac
if [[ "$mcp_enabled" != "1" ]] && docker volume inspect openface_mcp-state >/dev/null 2>&1; then
  mcp_enabled=1
fi
if [[ "$mcp_enabled" == "1" ]]; then
  mcp_state_dir="${OPENFACE_MCP_STATE_DIR:-$(sed -n 's/^OPENFACE_MCP_STATE_DIR=//p' .env | head -n 1)}"
  mcp_state_dir="${mcp_state_dir:-./secrets/openface-mcp}"
  mcp_forgejo_token_file="${OPENFACE_MCP_FORGEJO_USER_TOKEN_FILE:-$(sed -n 's/^OPENFACE_MCP_FORGEJO_USER_TOKEN_FILE=//p' .env | head -n 1)}"
  mcp_forgejo_token_file="${mcp_forgejo_token_file:-./secrets/openface-mcp-forgejo-user-token}"
  mcp_admin_token_file="${OPENFACE_MCP_ADMIN_INTERNAL_TOKEN_FILE:-$(sed -n 's/^OPENFACE_MCP_ADMIN_INTERNAL_TOKEN_FILE=//p' .env | head -n 1)}"
  mcp_admin_token_file="${mcp_admin_token_file:-./secrets/openface-mcp-admin-internal-token}"
  if ! docker volume inspect openface_mcp-state >/dev/null 2>&1; then
    echo "MCP is enabled but openface_mcp-state is missing." >&2
    exit 1
  fi
  if [[ ! -d "$mcp_state_dir" || ! -f "$mcp_forgejo_token_file" || ! -f "$mcp_admin_token_file" ]]; then
    echo "MCP is enabled but its bind state or credential source file is missing." >&2
    exit 1
  fi
fi
volumes=(
  openface_forgejo-data \
  openface_agent-metrics-data \
  openface_maintenance-agent-data \
  openface_shared-token \
  openface_forgejo-runner-data
)
if [[ "$mcp_enabled" == "1" ]]; then
  volumes+=(openface_mcp-state)
fi
for volume in "${volumes[@]}"; do
  if ! docker volume inspect "$volume" >/dev/null 2>&1; then
    echo "Required Docker volume is missing: $volume" >&2
    exit 1
  fi
  docker run --rm -v "${volume}:/source:ro" -v "$backup_dir:/backup" alpine \
    tar czf "/backup/${volume}.tgz" -C /source .
done

# The MCP profile's bind state and protected credential sources are not in the
# named-volume loop, so archive them in the same maintenance window.
if [[ "$mcp_enabled" == "1" ]]; then
  tar czf "$backup_dir/openface-mcp-state-dir.tgz" -C "$mcp_state_dir" .
  install -m 600 "$mcp_forgejo_token_file" "$backup_dir/mcp-forgejo-user-token"
  install -m 600 "$mcp_admin_token_file" "$backup_dir/mcp-admin-internal-token"
fi

Keep the .tgz, openface.env, zai.env, gateway-certs.zip, and protected MCP source filenames shown above. With OPENFACE_MCP_ENABLED=1, scripts/restore_lxc_deployment.sh requires and restores the MCP named volume, bind archive, Forgejo service-account PAT, and admin bridge credential, and persists COMPOSE_PROFILES=mcp in the restored .env. Pass replacement target paths through the corresponding OPENFACE_MCP_*_FILE variables when the restored host uses different paths. Do not put registry tokens, HMAC keys, or other secret contents in the manifest.

Write a manifest containing filenames, byte sizes, SHA-256 digests, the current release/SHA, and the health result. Never include .env contents, token contents, passwords, URLs containing credentials, or secret values in that manifest; keep the protected source files in the access-controlled backup. Copy the backup to a second protected location and perform a restore rehearsal before relying on it.

Upgrade sequence

1. From the old checkout, verify docker compose ps, the gateway health page, Forgejo login, a representative repository, an Issue/PR, a Space, metrics, maintenance status, and pipeline history. Save counts and representative IDs in the private backup manifest.

2. Fetch the target tag or reviewed release commit in a separate checkout. Run docker compose config --quiet and inspect environment, image, database, named-volume, and seed changes before stopping anything.

3. Keep the backup and old checkout available. Run docker compose up -d --build postgres, then wait for all three PostgreSQL databases to be healthy. Do not run the seed until the database backup and health checks are complete.

4. Keep spaces-runner stopped while choosing the migration path. A fresh install or a target with no legacy SQLite source can start the runner and verify /healthz after PostgreSQL is healthy. A v0.4.0 source must remain stopped until the explicit migration in the next step succeeds.

5. When the reviewed target contains the PostgreSQL pipeline schema and pipeline_migration.py, migrate an existing pipeline-audit.db with the explicit commands below. Do not run them for a fresh install or when no legacy source exists. Normal v0.5.0 startup never searches for or imports a legacy SQLite file automatically; do not accept new pipeline writes until the migration and comparison succeed:

   docker compose run --rm --no-deps --build spaces-runner \
     python pipeline_migration.py \
     --source /data/agents/pipelines/pipeline-audit.db --verify-only
   docker compose run --rm --no-deps --build spaces-runner \
     python pipeline_migration.py \
     --source /data/agents/pipelines/pipeline-audit.db

   The command validates SQLite integrity and columns, records a source digest in openface_pipeline.sqlite_migrations, is idempotent, and exits non-zero on a conflicting row or incomplete source. Keep the source file until the post-upgrade comparison is complete.

6. Start only the long-running services; keep the one-shot seed service out of the default restart. The --no-deps flag prevents Compose from starting seed through maintenance-agent or the Actions runner:

   docker compose up -d --build --no-deps \
     postgres gateway frontend forgejo spaces-runner maintenance-agent \
     forgejo-actions-dind forgejo-actions-runner

   If MCP was enabled before the maintenance window, or the target release explicitly requires it, set OPENFACE_MCP_ENABLED=1 for this block or keep COMPOSE_PROFILES=mcp in .env, then start its two profile services:

   set -euo pipefail
   mcp_enabled="${OPENFACE_MCP_ENABLED:-0}"
   compose_profiles="${COMPOSE_PROFILES:-$(sed -n 's/^COMPOSE_PROFILES=//p' .env | head -n 1)}"
   case ",${compose_profiles}," in
     *,mcp,*) mcp_enabled=1 ;;
   esac
   if [[ "$mcp_enabled" == "1" ]]; then
     docker compose --profile mcp up -d --build --no-deps mcp-admin openface-mcp
   fi

   Rerun seed only when the release explicitly requires it; the seed must be idempotent and must not delete existing repositories, Issues, likes, or audit history.

Post-upgrade verification

Run the checks in the same order for every release and attach a redacted record to the private deployment change:

Area	Required check
Compose	docker compose config --quiet; all expected services healthy
PostgreSQL	all three databases reachable; verify only the schema versions and migration markers required by the target release (the pipeline marker applies only when #163 is included)
Forgejo	login, repository clone/Files, LFS object, Issue, PR, comment, and history match the pre-upgrade IDs/counts
Space	public/private access boundary, start/status, representative artifact, and environment controls
Metrics	representative view/download/time-series counts, active likes, and agent identity remain present
Maintenance	webhook/job history remains queryable and does not replay unexpectedly
Pipeline	audit rows, retry/cancel/rollback/reconcile state, cursor, and representative run numbers match
Runtime	gateway, frontend, spaces-runner/healthz, maintenance health, and logs show no migration or secret errors

Compare row counts, stable IDs, creation timestamps, Issue/PR numbers, repository history, LFS objects, and pipeline run numbers before declaring success. A missing comparison is a failed verification, not an unknown pass.

Failure and rollback

If a healthcheck, migration, compatibility check, or comparison fails:

• stop the affected services and preserve logs without secret-bearing environment output;
• do not delete named volumes and do not retry a destructive migration blindly;
• if only images/configuration changed and the schema remains compatible, restore the old checkout/image tags and start the stack again;
• if a schema/data migration ran, do not downgrade the schema in place. Restore the PostgreSQL dumps and named volumes into an isolated project or the approved maintenance window, verify the old release there, then switch back only after the restore comparison passes;
• record the failed target version, migration version, backup digest, and recovery result without recording credentials.

The old checkout is not a database backup. A rollback without a verified dump and volume archive is blocked.

Release-note checklist

Every release note and its Japanese page must fill these fields:

• target upgrade path and supported previous versions;
• breaking changes and compatibility checks;
• schema/data migration and whether it is automatic or explicit;
• backup requirement and databases/volumes covered;
• environment-variable, image, Compose, and volume changes;
• rollback condition and restore procedure;
• post-upgrade health/data comparison;
• known issues and the link to this runbook.

See the v0.6.0 release page for the current checklist and the v0.5.0 release page for the preceding pipeline migration.