Results¶
This page collects what the finished pipeline produces: how long each stage takes, and the verification output from a node built by it. The full breakdown is on the Timings.
Timings¶
All numbers are wall clock on one host (24 cores, 125 GB RAM), with the edge node as a QEMU VM (6 vCPU, 8 GiB, 40 GB qcow2, OVMF, virtio, slirp networking) and Artifact Keeper v1.10.2 on the same host. The node numbers are with KVM, signed images, and the installer stage2 served locally.
Build side¶
| Step | Time |
|---|---|
make registry-up, cold start to /readyz |
about 60 s |
base image (RESF recipe, minimal) |
211 s (upstream standard recipe unmodified: 4 min 53 s) |
rpms/build.sh: build and sign two RPMs |
7 s |
| edge image, OS layer rebuilt | 27 s |
| edge image, further release (OS layer cached) | 8 s |
unsigned-test image |
1 s |
| push per image (only new layers) | 1-2 s |
| cosign sign per image | 1 s |
Docker Hub proxy, cold podman pull nginx:alpine |
2.2 s |
Node side, with KVM¶
| Stage | Time |
|---|---|
vm-install total |
65 s |
| power-on to ssh | 25 s |
| ssh to RKE2 node Ready | 61 s (86 s from power-on) |
| nginx-demo Running | already Running when checked (< 21 s after Ready) |
vm-upgrade-unsigned: bootc upgrade refused |
< 1 s |
vm-upgrade: promote (skopeo copy) |
1 s |
vm-upgrade: bootc upgrade pull and stage (3 layers, 7.8 MB) |
6 s |
vm-upgrade: reboot to ssh |
20 s |
vm-rollback: reboot to ssh |
101 s (both times) |
negative install (unsigned-test) until vm-install aborts |
40 s (46 s wall) |
| power-on to working cluster | about 3 min (65 + 25 + 61 + ~20 s) |
Power-on to a working, signed, verified Kubernetes node is about three minutes. Under QEMU's
software emulation (no /dev/kvm) the same path takes about 17 minutes. The slowest part of
an install turned out to be downloading the installer's 750 MB stage2 from the mirror, so the
harness caches it with deploy/fetch-media.sh and serves it next to the kickstart. The
timings page has the KVM against software-emulation comparison stage by stage.
Verification output¶
make vm-verify on a freshly installed node:
=== bootc status ===
spec: image: 10.0.2.2:30080/oci-bootc/rocky-edge:10 transport: registry
booted: version: 10.2-3 imageDigest: sha256:d4f3ec69bbd3...
=== getenforce ===
Enforcing
=== kubectl get nodes -o wide ===
NAME STATUS ROLES VERSION OS-IMAGE KERNEL-VERSION
edge-node-01 Ready control-plane,etcd v1.36.5+rke2r1 Rocky Linux 10.2 (Red Quartz) 6.12.0-211.61.1.el10_2.x86_64
=== workload image / imageID ===
default/nginx-demo-... docker.io/library/nginx:alpine docker.io/library/nginx@sha256:df221db8...
=== Artifact Keeper: oci-dockerhub-proxy ===
98 cached objects; images: library/nginx, rancher/hardened-calico, rancher/hardened-coredns,
rancher/hardened-etcd, rancher/hardened-kubernetes, rancher/klipper-helm, rancher/rke2-runtime, ...
OK: nginx-demo runs nginx:alpine @ sha256:df221db8..., and that manifest is cached in oci-dockerhub-proxy
After the day-2 upgrade in Step 5, bootc status shows
10.2-4 booted with 10.2-3 as the rollback, and the MOTD reads
edge-site-config 1.0-4.el10: day-2 update via bootc upgrade (release 4). After
bootc rollback it is back on 10.2-3, manifests included.
What it adds up to¶
- One source of truth. Artifact Keeper holds the base operating system packages, the
Kubernetes packages, the site package, the base image, the edge image releases, their
signatures, the public keys, and every container image the cluster pulled. Every pull we
could trace was served by Artifact Keeper. "What is running on node 37?" is
bootc statuson the node and a digest lookup in one registry. - Cheap day 2. Promotion is a tag copy, rollback is built in, and a site configuration change costs each node 7.8 MB.
- Verified at the point of use. Every RPM and image, and all repodata whose publisher signs
it, is verified by the consumer that uses it: dnf, podman, the installer and
bootc. - Image mode fails differently than package mode. Neither of the two real bugs (the
read-only
/optand the missingbubblewrap) showed up in a build or a lint. Keep a CI stage that boots the image and upgrades it.
The Findings overview has the full lab record behind these results.
Next: Next steps.