Skip to content

Artifact Keeper notes

Everything learned about Artifact Keeper v1.10.2 while building this proof of concept, consolidated from registry/README.md, registry/VERIFICATION.md and the three findings documents. "AK" below is Artifact Keeper.

Running it rootless

  • The stock docker-compose.yml and support files from tag v1.10.2 are kept byte-identical; every local change is in registry/compose/compose.override.yml.
  • podman compose (delegating to docker-compose v5.5.1 over the rootless podman socket) handled the compose file unchanged. The docs suggest podman-compose; it was not needed. Nothing needed sudo.
  • .env lives in registry/, the compose files in registry/compose/, so the scripts pass --env-file and -p artifact-keeper explicitly to keep container and volume names stable.
  • Only Caddy is published: 30080 (HTTP: web UI, /api/v1, /rpm/<key>, OCI /v2) and 30443 (HTTPS, Caddy internal CA, unused here).

Docs vs reality

Topic Docs / defaults What v1.10.2 actually does
Secrets quickstart shows only JWT_SECRET in .env Defaults are fatal. JWT_SECRET=change-me-in-production-please is on a placeholder denylist (rejected regardless of ENVIRONMENT); AK_WEBHOOK_SECRET_KEY=REPLACE_ME_... is "set but invalid" and the backend exits: it must base64-decode to exactly 32 bytes. up.sh generates both (openssl rand -base64 48 / -base64 32)
First-boot lock Without ADMIN_PASSWORD, the API is locked until the random password in /data/storage/admin.password is changed. up.sh sets a random alphanumeric ADMIN_PASSWORD, which skips the lock
Image tags compose uses :latest backend honours ARTIFACT_KEEPER_VERSION; web and openscap hardcode :latest. There is no 1.10.2 tag of artifact-keeper-web on ghcr.io, so web runs 1.10.1
Exposed ports Stock compose publishes Postgres on 0.0.0.0:30432 (registry/registry), OpenSearch on 9200, Trivy 8090, OpenSCAP 8091. The override uses ports: !reset []
Short image names postgres:18-alpine, alpine:3.24, caddy:2-alpine Under rootless podman's short-name-mode = "enforcing", postgres:18-alpine silently resolved to an unrelated cached ghcr.io/artifact-keeper/ci-mirror/postgres:18-alpine. The override qualifies them as docker.io/library/...
Scanners Trivy and OpenSCAP always on Moved to a scanners profile. /readyz does not check them; the backend logs Service 'trivy' is unavailable once a minute
Repo visibility The create API takes is_public (alias allow_anonymous_access); there is no visibility field. Repos default to private
RPM upload curl -F file=@pkg.rpm http://localhost:8080/api/artifacts/rpm/<repo> (guides/system-packages.mdx) Returns HTTP 404 Repository not found. Working routes: PUT /rpm/<key>/packages/<file>.rpm and POST /rpm/<key>/upload. The format guides also use port 8080; the compose stack serves on 30080
RPM remote upstream Must be a concrete baseurl (no mirrorlist/metalink). Private-IP upstreams would need AK_SSRF_ALLOW_PRIVATE_CIDRS
Generic repo route backend mounts GET /general/<repo>/<path> The stock Caddyfile has no /general/* route, so the request falls through to the web UI's HTML 404. Use /api/v1/repositories/<key>/download/<path>

API calls that worked

Authentication: the admin user (password in registry/.env) for setup, and a CI token (read:artifacts, write:artifacts, 30 days) minted by bootstrap.sh and used as the password in HTTP basic auth. GET /api/v1/auth/me tells whether a token is still valid.

# repositories
POST /api/v1/repositories                    {"key": ..., "format": "rpm|docker|generic", "repo_type": "remote|local", "is_public": true, ...}
GET  /api/v1/repositories/<key>              (repo id, needed by the signing API)
GET  /api/v1/repositories/<key>/artifacts    (listing; OCI manifests, .sig tags and proxy cache entries show up here)

# RPM
PUT  /rpm/<key>/packages/<file>.rpm          -> 201; same file name again -> 409, stored file kept
GET  /rpm/<key>/repodata/repomd.xml          (generated by the server, no createrepo_c step)

# generic
PUT  /api/v1/repositories/<key>/artifacts/<file>     (token; set Content-Type explicitly) -> 201
GET  /api/v1/repositories/<key>/download/<file>      (anonymous on a public repo) -> 200

# OCI
GET  /v2/token?service=artifact-keeper&scope=repository:<repo>/<image>:pull   (anonymous bearer token)
GET  /v2/<repo>/<image>/manifests/<tag>
GET  /v2/<repo>/<image>/referrers/<digest>

curl -u "admin:$(cat registry/.ak-token)" -T foo.rpm http://localhost:30080/rpm/rpm-edge-site/packages/foo.rpm and podman login -u admin --password-stdin localhost:30080 < registry/.ak-token both work.

RPM repositories

  • Proxies for Rocky, EPEL, RKE2 and k3s work with plain dnf baseurls of the form http://<host>:30080/rpm/<key>. A key with a dot (rpm-rke2-1.36) is accepted.
  • Hosted uploads are write-once per file name: re-uploading returns HTTP 409 and keeps the stored file, even if the local bytes differ (rpmbuild output is not reproducible). Treat NEVRAs as immutable and bump the release.
  • Repodata is regenerated immediately after an upload.
  • Upstream signatures pass through. handlers/rpm.rs proxies repodata/repomd.xml.asc for remote repos; for Rocky (x3) and RKE2 (x2) both repomd.xml and .asc were byte-identical to upstream and verified with the vendor keys. EPEL 10 and Rancher's k3s el9 tree publish no .asc (404 upstream and through AK).

Signing API (repodata)

AK can sign a hosted RPM repo's metadata with a server-side key. The calls, as sent:

GET  /api/v1/signing/keys                                  -> {"keys":[],"total":0}
GET  /api/v1/signing/repositories/<repo id>/config         -> {"sign_metadata":false, ...}

POST /api/v1/signing/keys
  {"name":"rpm-edge-site repodata","key_type":"gpg","algorithm":"rsa4096",
   "repository_id":"<repo id>","uid_name":"Artifact Keeper rpm-edge-site",
   "uid_email":"rpm-edge-site@example.invalid"}
  -> 200 {"id":"<key id>","fingerprint":"56f1a82f...","public_key_pem":"-----BEGIN PGP PUBLIC KEY BLOCK-----...", ...}

POST /api/v1/signing/repositories/<repo id>/config
  {"signing_key_id":"<key id>","sign_metadata":true}
  -> 200 {"sign_metadata":true,"sign_packages":false,"require_signatures":false, ...}

Afterwards AK serves repodata/repomd.xml.asc (application/pgp-signature) and repodata/repomd.xml.key (application/pgp-keys) anonymously, and both gpg and dnf (repo_gpgcheck=1) verify them.

  • key_type must be gpg for rpm (and debian) repos; rsa/ed25519 are rejected.
  • The private key stays inside AK, encrypted with a key derived from JWT_SECRET.
  • Signatures carry a 7-day expiry (SIGNATURE_EXPIRY_SECONDS, issue #1327) and AK re-signs on demand, so a downstream mirror that copies repomd.xml.asc once will go stale.
  • Not tried: sign_packages (AK signing RPMs on upload) and require_signatures; RPMs are signed in our build instead. The web UI's Signing page was not checked.

OCI registry

  • Path-based: images live at host:30080/<repo>/<image>, e.g. localhost:30080/oci-bootc/rocky-edge:10.
  • Anonymous pulls from public repos work with real OCI clients (skopeo --no-creds), but a raw curl of /v2/... gets 401 with Www-Authenticate: Bearer realm="http://localhost:30080/v2/token": clients must do the token dance. The realm follows the request's Host header (http://10.0.2.2:30080/v2/token when asked as 10.0.2.2), so in-VM clients get a reachable token endpoint.
  • Plain HTTP needs --tls-verify=false or a registries.conf insecure = true entry.
  • cosign signatures. AK stores legacy sha256-<digest>.sig tags like any other manifest (485 bytes, one application/vnd.dev.cosign.simplesigning.v1+json layer), and it also implements the referrers API, where cosign 3's default bundle format lands (GET /v2/<repo>/<name>/referrers/<digest> returns a proper OCI index with artifactType). AK neither signs nor verifies images itself, and nothing in its UI or API treats .sig tags as signatures. podman, skopeo, bootc and Anaconda only read the legacy .sig format.
  • Tag copies keep the digest, so promotion with skopeo copy needs no re-signing.
  • OCI DELETE (skopeo delete, as admin) removed scratch tags from the registry, but the manifests stayed in the artifacts API listing until deleted through the REST API too.
  • storage_used_bytes for oci-bootc reported 4.1 GB after several pushes of images that share almost all blobs (about 0.9 GB unique). Either it counts per manifest or blobs are not deduplicated; not investigated.
  • Not tried: promotion rules with require_signature (handlers/promotion_rules.rs).

Pull-through proxies

  • quay.io (oci-quay-proxy) served the Rocky builder image localhost:30080/oci-quay-proxy/rockylinux/rockylinux:10.
  • Docker Hub (oci-dockerhub-proxy) worked first time: podman pull --tls-verify=false localhost:30080/oci-dockerhub-proxy/library/nginx:alpine pulled in 2.2 s cold with the same digest as docker.io/library/nginx:alpine. The short form oci-dockerhub-proxy/nginx:alpine also works (the proxy adds library/).

  • containerd needs a rewrite. containerd mirrors are host-level, while AK serves the proxy under a path prefix. RKE2's registries.yaml:

    mirrors:
  docker.io:
    endpoint: ["http://10.0.2.2:30080"]
    rewrite:
      "^(.*)$": "oci-dockerhub-proxy/$1"

    containerd normalises nginx:alpine to library/nginx, so the request becomes /v2/oci-dockerhub-proxy/library/nginx/manifests/alpine. This mirrors all of docker.io, so every RKE2 system image (rke2-runtime, hardened-kubernetes, hardened-etcd, calico, flannel, coredns, klipper-helm, ...) came through AK as well: 98 cached objects after one node came up. RKE2's generated hosts.toml keeps registry-1.docker.io as the fallback, so an air-gapped site should also block egress. - containerd does not log which mirror served a pull. The proof is that the pod's imageID digest exists in oci-dockerhub-proxy, created in the same second as containerd's PullImage line. - Some by-digest manifests are listed with size 0 in the artifacts API (for example the per-platform manifest containerd resolved from the alpine index). Pulls work; only the listing is off.

Generic repository (raw-edge-keys)

  • Format generic exists in backend/src/models/repository.rs; bootstrap.sh creates raw-edge-keys|generic|local with is_public: true.
  • Upload with PUT /api/v1/repositories/<key>/artifacts/<file>, download anonymously with GET /api/v1/repositories/<key>/download/<file>. Verified from the host, from a podman container and from the installer's %pre.
  • Set Content-Type on upload. Without it, curl --data-binary uploads were stored and later served as application/x-www-form-urlencoded: AK echoes whatever the uploader sent.
  • Write-once per path (409 on a second PUT) unless the repo has versioning_enabled. Replacing a file means DELETE then PUT, and DELETE needs the delete:artifacts scope; the CI token gets 403, so publish-keys.sh logs in as admin only for the delete.
  • The native /general/<key>/<path> route is unreachable behind the stock Caddyfile (see above).

Gotchas, short list

  1. Generate JWT_SECRET and a valid 32-byte AK_WEBHOOK_SECRET_KEY before the first start, or the backend will not start.
  2. Repos are private unless created with is_public: true.
  3. The documented RPM upload path 404s; use PUT /rpm/<key>/packages/<file>.
  4. Uploads (RPM and generic) are write-once per name; deletes need delete:artifacts.
  5. The OCI registry is path-based, so containerd mirrors need a rewrite.
  6. Generic downloads go through /api/v1/repositories/<key>/download/ behind the stock Caddy.
  7. Repodata signatures expire after 7 days and are re-signed on demand.
  8. cosign 3's bundle format is stored fine but is invisible to podman/bootc; sign in the legacy .sig format.
  9. Pin images in the override; the web image has no 1.10.2 tag.

Filed upstream

Issues opened on 2026-10-06 from the findings above:

  • artifact-keeper#4478: Stock compose and .env.example ship JWT_SECRET values the backend rejects at startup
  • artifact-keeper#4479: Compose uses short image names (postgres, alpine, caddy) that podman resolves per host
  • artifact-keeper#4480: Compose publishes Postgres (registry/registry), OpenSearch, Trivy and OpenSCAP on 0.0.0.0
  • artifact-keeper#4481: Compose ignores ARTIFACT_KEEPER_VERSION for openscap and has no way to pin the web image
  • artifact-keeper#4482: Stock compose does not route /pacman or /bazel, and /general only via unreleased web code
  • artifact-keeper#4483: OCI storage_used_bytes and quota ledger count layers twice (image size rows + oci_blobs)
  • artifact-keeper#4484: OCI API: identify cosign signatures and referrers as attachments of the subject image
  • artifact-keeper#4485: Server-side cosign signing for OCI repositories (legacy .sig format) on push or promotion
  • artifact-keeper-web#956: OCI tag list: show cosign signatures on the signed image instead of as separate tags
  • artifact-keeper-web#957: Repository views: truncated type filter and paths, wrapping sizes and long tag names
  • artifact-keeper-site#114: Quickstart does not start on v1.10.2: no secrets step, create-repo example missing name
  • artifact-keeper-site#115: Docker guide and signing page push to localhost:8080/myapp, missing the repository key
  • artifact-keeper-site#116: System packages guide: /api/artifacts/... uploads 404 and ak publish does not exist
  • artifact-keeper-site#117: Generic format guide documents a /generic/{repo} route that does not exist
  • artifact-keeper-site#118: Signing docs: cosign format podman/bootc can verify, signedIdentity, key API, expiry

Already tracked upstream, not refiled: the OCI by-digest delete leftovers (#4450, fixed by #4464; follow-up #4465). Size 0 for image-index rows is intended (#3601).