Status: CUTOVER DONE (2026-10-17): company-dns.mediumroast.io is served by V4, image tag 10172026-0eaeb0b, digest sha256:50bfef01cf781543478b6f3e26231ae937b2f9387676b00550d8b51f717d28cb, the same image staging ran (checked pod by pod). V3 stands by with its four pods, taking no traffic, as the rollback (the Ingress backend is the one switch; see the runbook, part 5). The 48-hour soak against production passed (2026-10-20: 10.4 million requests, no errors, every canary correct; the cluster-side watcher did not run, see the performance report, section 3b). Next: the freeze on 21 October, the event on 22 to 25 October, then WP8 (retire V3) and WP9/WP10 after about 26 October. Section 8 has the gates.
Owner: michael.hay@mediumroast.io
Scope: everything between “V4 is verified on staging” and “company-dns.mediumroast.io is served by V4, V3 is gone from the cluster, and V3 is only source code in the
repository”. It follows v4-staging-rollout.md (staging, done) and v4-staging-scripts.md (the script conventions this plan extends), uses the
sizing results in v4-sizing-test.md, and the cluster facts in onprem-k8s-migration.md. Out of scope: the dev tier, new V4 features, the
quarterly data build itself (only the gate that it exists before the V3 build is switched off, WP9), moving the repository’s layout.
No addresses or internal host names appear here (the repository is open source): <build-node>, <traefik-lan-ip> and the like are placeholders, as elsewhere.
| Fact | Source |
|---|---|
V3 production: namespace company-dns, Deployment company-dns, 4 replicas (a deliberate ceiling), requests 100m / 256Mi, limits 500m / 1Gi, port 8000, Ingress for company-dns.mediumroast.io (HTTPS, plus an HTTP redirect Middleware), certificate company-dns-io-tls. |
k8s/prod/ (to be confirmed against the live Deployment: kubectl -n company-dns get deploy company-dns -o yaml; an earlier V3 manifest drifted from production) |
V4 staging: namespace company-dns-staging, 20 replicas at 1000m / 1Gi, the per-IP quota 6000 a minute with a burst of 600, SQL endpoint on. Verified: HTTPS valid, 401 without and 200 with a token, the API suite 57 OK / 3 skipped on tag 10112026-f255f85. |
k8s/staging/, v4-bringup.md, v4-sizing-test.md |
V4 answers V3’s URLs. Every route is also served at its /V3.0/ and /V2.0/ path in V3’s envelope; the one deliberate difference is the additive top-level total. / redirects to /ui/; /health, /docs and the UI exist. |
v4/README.md, the differences ledger |
| Capacity at 20 pods: about 1900 lookups a second (2300 repeated), p95 about 400 ms; embeddings follow the pod count (about 19 a second per pod). | v4-sizing-test.md |
Deploy tooling: scripts/build-and-deploy-v4.sh (build, push, apply with the tag substituted, smoke test, certificate, HTTPS check, suite, rollback, build log) and scripts/seal-secrets-v4.sh, both staging only today. V3 has scripts/build-and-deploy.sh and svc_ctl.sh. |
scripts/ |
A monthly GitHub workflow rebuilds and pushes V3’s latest image, which is how V3’s data (EDGAR especially) stays current. V4 has no equivalent: its EDGAR catalog is made during an image build, and where the SIC files are delivered from is still open (item H of v4-release-to-staging.md). |
.github/workflows/main.yml |
| Hackathon, 22 to 25 October: up to 300 teams, mostly the US SIC lookups with client-side caching, some other SIC data, little or no embedding search. The platform partner’s integration test (V3 swapped for V4) comes first. | the owner, 2026-10-11 |
Certificates on this cluster only renew because of the CoreDNS hosts block; the earliest renewal is 2026-10-19. |
onprem-k8s-migration.md |
https://company-dns.mediumroast.io is served by V4 from a pinned, registry-pulled image that is the one that passed staging (same tag, same digest), at the size chosen in D2,
with production credentials and rules that staging and V3 never saw, and the SQL endpoint off (unless D4 says otherwise).| # | Decision | Recommendation | Why |
|---|---|---|---|
| D1 | How traffic moves. In-place replacement of V3’s Deployment, a second namespace with a new host, or blue/green inside company-dns. |
Blue/green in company-dns: V4 runs as its own Deployment and Service (company-dns-v4) beside V3 with no Ingress traffic until the cutover; the cutover is one change of the Ingress backend from V3’s Service to V4’s; rollback is the same change back. The host, certificate and redirect do not change, so nothing depends on DNS or a new certificate. |
Instant, tested rollback; no DNS or certificate step on the day; V3 keeps its pods through the soak. A weighted split (a Traefik IngressRoute with TraefikService) was considered and rejected for now: more moving parts than a flip, and the staging run is the rehearsal. |
| D2 | Production shape. | The staging shape, 20 x 1000m / 1Gi, COMPANY_DNS_REPLICAS=20, for the hackathon; revisit after 26 October (the sizing numbers say 12 pods carry about 1260 lookups a second if the load turns out lower). V3’s four pods stay during the soak, so 20 + 4 pods share the nodes. |
Measured, not guessed. A replicas and COMPANY_DNS_REPLICAS change is one line each. |
| D3 | What is deployed. | Promote, do not rebuild: the exact tag (and digest) that passed staging. The script refuses a tag the build log does not show as having passed the staging suite. | Build once; the thing tested is the thing run. |
| D4 | Profiles, SQL and quotas in production. | Production’s own Secret and rules (sealed with a production secrets directory, never reusing staging tokens). SQL stays off in production. Profiles: mediumroast.io (bypass), and one for the platform partner and the hackathon if they need a quota of their own. Per-IP quota for the hackathon 6000 a minute, burst 600; the anonymous tier stays at its default (5 a minute) unless the owner decides otherwise. |
Least privilege; the SQL endpoint is experimental. Hackathon teams are anonymous or identified-by-User-Agent callers per IP, so the per-IP numbers are what matter. |
| D5 | When to cut over, relative to the hackathon. | Before it, with at least five days of soak (cutover by about 15 October), keeping V3 as the standing rollback through the event. The alternative, running the hackathon on the staging host and cutting over afterwards, leaves production untouched during the event but makes the event depend on an environment with the SQL endpoint on and shared test credentials. | The event is the real load test; better on the endpoint the partner will actually use. |
| D6 | When V3 leaves the cluster. | After the hackathon and a short quiet period (about 26 October to 2 November), not before: V3 is the cheapest rollback during the riskiest week. Until then it can run at two replicas to free cores. | Headroom matters less than a safe fallback for that week. |
| D7 | What “V3 as source code” means. | Tag, banner, retire; no restructure now. Tag v3-final; a banner in the README and in docs/ saying V3 is no longer deployed and V4 is the service; stop the monthly workflow (or repoint it at V4, WP9); keep company_dns.py, lib/, html/, makedb.py, the root Dockerfile, requirements.txt and svc_ctl.sh where they are; keep GHCR’s V3 package (do not delete tags; mark the last production tag in the changelog). Moving V3 into a v3/ directory is possible later and is a separate, mechanical change. |
Moving files breaks links, the parity tests and history for no operational gain. |
| D8 | What the parity tests become. api_tests L3 compares V4 to a live V3. |
Before the cutover: run L3 once against live V3 (still open from the staging checks) and keep the report. After V3 is gone: L3 runs against a locally started V3 container from the v3-final tag, or is retired and its report kept as the record; the owner picks. |
Keeps the evidence that V4 matches V3 without keeping V3 deployed. |
Each has a deliverable and a way to see it is done. Merges are the owner’s.
main to staging with scripts/build-and-deploy-v4.sh (20 pods, the rate-limit settings in the image).v4-staging-rollout.md WP7 at the new size: the log-level edit without a restart; the rate-limit check across 20 pods (v4-bringup.md step 6.3, parallel requests, some 429s); CORS; the SQL capacity numbers recorded; a rollback drill (roll to the previous tag and back); a soak under a steady synthetic load (the repeated mix at a modest rate) for as long as the calendar allows (at least 48 hours; 72 preferred), watching restarts, memory and latency.total layer; the Cargo crate versions at 4.0.0, a changelog entry for V4 and for the total addition on V3’s paths (WP8 of the staging plan).company-dns.mediumroast.io; dex-staging-tls renewed (due 2026-10-19); the apex name’s second public address checked before its November renewal.k8s/prod-v4/: Deployment company-dns-v4 (20 replicas, port 4000, probes on /health (a patient liveness probe, timeoutSeconds: 5 and failureThreshold: 6, as in the staging Deployment: at extreme overload a busy pod missed a 3 s check and was restarted, see docs/performance-report.md section 5d), locked down as staging, maxSurge 4, the same anti-affinity), Service company-dns-v4, the rules ConfigMap, the log-level ConfigMap, the SealedSecret. No Ingress, Middleware or Certificate in this directory: those stay in k8s/prod/ and keep serving V3 until the flip. No scripts in the directory.COMPANY_DNS_SQL_ENABLED absent (off); COMPANY_DNS_CORS_ORIGINS explicit; the rate-limit settings per D4; COMPANY_DNS_REPLICAS equal to replicas (a test pins it, as for staging).k8s/test_staging_manifests.py: placeholder tag UNSET, replicas agree, no SQL, no secrets in the tree, the Service selects the Deployment, no Ingress in the directory.kubectl apply --dry-run=server accepts the directory.scripts/seal-secrets-v4.sh --env prod: tokens and credentials into a separate directory outside the repository (mode 600), sealed to k8s/prod-v4/sealedsecret-credentials.yaml; refuses to run if the production directory would be the staging one; --rotate, --reseal as for staging.Extend scripts/build-and-deploy-v4.sh (not a new script) with a production mode, keeping its conventions (preflight listing every problem, --check, --status, --dry-run, annotations on the Deployment, the build log):
--env prod selects k8s/prod-v4/ and the company-dns namespace; no build in this mode (--tag names a tag already pushed); refuses a tag that the build log does not show passing the staging suite, or one not built from a clean tree.--cutover points the Ingress backend at company-dns-v4 (a patch, recorded with the tag and time in an annotation and the log) after the pre-checks of WP5; --rollback-cutover points it back at V3’s Service; both print the exact state before and after.--resolve, the suite, then a check from outside the LAN left to the owner (a phone on mobile data), which the script asks for and waits on.scripts/test_v4_scripts.py for the new mode, including: refusing an unproven tag, refusing the staging secrets directory, the flip and the flip back, nothing touched in --dry-run.--dry-run --env prod against the live cluster prints a plan and changes nothing.company-dns; wait for 20 ready pods spread over both nodes.perf_tests/load.py at a modest level to see the new pods under load beside V3’s pods; V3’s request latency before and after (to see the co-tenancy cost).Preconditions (all must hold): WP1 gate report accepted; WP5 done; the rollback command tested on the day with a harmless flip and flip back; a quiet period chosen; someone available to watch.
--cutover. Within a minute: /health, a lookup from each of the three families, an authenticated request, / redirect, the UI loads.429 counts in the pod logs, p95 latency from a probe, pod restarts, memory, node load. Rollback triggers: any sustained 5xx rate above 1%, p95 above two seconds on lookups for five minutes, a crash loop, a client-breaking difference found. Rollback is --rollback-cutover, then the cause is written up.
replicas and COMPANY_DNS_REPLICAS together).v4-sizing-test.md and decide D2 again.After D6’s quiet period, in this order, each step reversible until the last:
k8s/prod/: git mv V3’s out, V4’s in), pointing at the V4 Service. Rename V4’s Deployment and Service to company-dns only if that is wanted; renaming is a second cutover, so default is keep company-dns-v4.scripts/build-and-deploy.sh and V3’s use of svc_ctl.sh retire (the long-term plan in v4-staging-scripts.md §6 folds V4 into svc_ctl.sh; this plan does not require it).kubectl -n company-dns get deploy shows only V4, and the public name still passes the suite.V3’s monthly workflow is what keeps V3’s data fresh. Do not stop it until V4 has its own path: the EDGAR catalog rebuilt on a schedule (ingest-edgar already exists, and the image build makes the catalog), and a decided location for the SIC files (v4/scripts/fetch_data.py, v4/data-manifest.json). Then replace main.yml with a V4 build (the quarterly workflow of the earlier plans, item D) that builds and pushes a candidate tag that must still pass staging and the promotion rule of WP4 before production. Until then the V3 workflow keeps running harmlessly.
v3-final on the last commit that built V3’s production image; record that tag and image in the changelog.docs/: a banner that V3 is no longer deployed and V4 (v4/) is the service; the README’s run instructions put V4 first and describe svc_ctl.sh as V3’s local controller; the changelog gets a V4 entry and a migration note (what changed for clients: the additive total, the redirect at /, anything the ledger lists).api_tests L3 per D8; perf_tests baselines stay as the V3 record.| When | What |
|---|---|
| 11 to 13 October | Merge the staging PRs; deploy to staging at 20 pods; WP1 verification gate and the soak begin; WP2 to WP4 written and tested without touching production. |
| 14 October | Platform partner’s integration test on staging; WP1 report accepted; WP3 production secrets; WP5 deploy V4 beside V3 and pre-checks. |
| 15 to 16 October | WP6 cutover (D5), watch window; start of the soak with V3 as the standing rollback. 19 October: dex-staging-tls renewal checked. |
| 17 to 21 October | Soak; daily notes; fix anything found; freeze on 21 October (no deploys during the event unless a rollback trigger fires). |
| 22 to 25 October | The hackathon. |
| 26 October to 2 November | Review the real traffic; re-decide the size (D2); WP8 begins after the quiet period; WP9 and WP10 in parallel. |
If the gates slip, the fallback is D5’s alternative: cut over after the event and let the hackathon use whichever endpoint is proven, decided no later than 17 October.
| Risk | Mitigation |
|---|---|
| A client depends on a V3 behaviour V4 does not reproduce. | The differences ledger and L3 report go to the known clients before the cutover; the partner’s integration test; rollback is one command. |
| V4’s 20 pods and V3’s 4 crowd the nodes (other workloads share them). | Requests are half the limits, the nodes’ usage is recorded in WP5, V3 can drop to two replicas, the sizing numbers allow 12 pods. |
| The per-pod share of the rate limit is too small for a bursty team (burst 600 over 20 pods is 30 each). | Settings are env vars; the step-6.3 check; raise the burst, or scale the quota, by a ConfigMap or env change and a restart. |
| Certificates fail to renew (CoreDNS block missing or reset). | WP1 check; the table in the cluster plan; re-apply the block if DNS addons are toggled. |
| A single bad pod or a bad tag reaches all traffic at once on the flip. | Pre-checks in WP5; the watch window; the rollback triggers; no deploy during the event without a trigger. |
| V3’s data goes stale if its workflow is stopped too early, or V4’s data is stale from day one. | WP9 gate; record the data dates in the build log and the changelog. |
| The sealed production secret or tokens leak, or staging tokens work in production. | Separate directories and sealing, the script refuses to mix them, a check that a staging token gets a 401 in production. |
| A history question: commit messages mention node names. | Not changed here; the owner decides separately whether to rewrite history. |
user-agent "YOUR_APP_NAME YOUR_CONTACT_EMAIL", which has no e-mail address or URL and so is classed as an unidentified caller; measured on staging, a client sending it got about a dozen answers and then 429s. A team that follows Zetaris’s 15-second pacing can still load its three data sets at 5 a minute, but if the teams’ Zetaris instances fetch from a shared address (a managed cluster, a venue), all 300 teams share one anonymous allowance and most will be refused. Update, 2026-10-20: Zetaris changed its scripts (host moved to production, an identifying User-Agent required, and run_sql.py refuses the placeholder); only a team pasting the statements into the console by hand can still send it. Option (a) is done; (b), the raised anonymous tier, stays in place for the event. Options: (a) the owner has Zetaris replace the placeholder with a contact-bearing default and tell teams to set their own (no change here); (b) raise the anonymous tier for the event window with COMPANY_DNS_RATE_LIMIT_ANONYMOUS_PER_MINUTE and _BURST (a ConfigMap-free env change and a restart; keep it well below the standard tier); (c) both. Recommendation: (c), decided before the freeze on 21 October. See docs/integrations/zetaris.md.mediumroast.io profile bypass.
The other bypass: a trusted Origin or Referer (v4/crates/server/src/rate_limit.rs, trusted_origin.rs). A caller that sends Origin: https://mediumroast.io (or a subdomain, or localhost) is not rate limited at all, and the header is set by the caller (docs/origin-bypass-check.md). It exists because browsers cannot hold credentials or set a User-Agent with a contact in it: V4’s own web page at /ui/ and the html/ app call the API from the visitor’s browser with neither, so without the bypass a visitor is an anonymous caller (5 a minute by default). The first-party applications that can use a Basic-auth profile do not need the bypass; the browsers do. There is no setting for it today (the field is always on). Options: (A) remove it: the UI then lives on the anonymous tier, which breaks it after the event when the tier goes back to 5 a minute; (B) recommended: keep the trust but give a trusted origin the standard tier, 6,000 a minute per address, instead of no limit: the UI works, and spoofing the header gains no more than putting an @ in the User-Agent already does; (C) a setting COMPANY_DNS_TRUST_ORIGIN (default on) so production can switch it off. Decided 2026-10-16: B, written in pull request 164 (rate_limit.rs, with tests; verified on a local build: origin_check.py reports CLOSED, and a browser User-Agent with the UI’s Referer gets the normal burst). It needs a new image: a staging deploy and perf_tests/origin_check.py --expect closed against it before that tag is promoted (the soak is not repeated on staging: it runs on production, see the decisions of section 8). The cutover promotes the new tag.Cutover, 2026-10-17 (done): the runbook’s parts 1 to 4 were followed on the build node: the image 10172026-0eaeb0b was built and put on staging (container smoke test, the API suite and origin_check.py --expect closed passed), then deployed to production beside V3 (20 pods, ten on each node, V3 untouched; the production token accepted, a staging token refused, the suite 44 tests OK), then both Ingress backends were pointed at V4 and the public name, the V3-shaped paths, the redirect and the UI answered. Three things went wrong on the way, all fixed in the repository: the container smoke test was throttled once the trusted-Origin bypass closed (#166), origin_check.py seemed to hang because the build node does not resolve the names to Traefik’s address (#167, and a runbook step to make it so), and kubectl port-forward looked like a hang (the runbook now says it stays running).
Decided by the owner on 2026-10-16: (1) the anonymous tier is raised for the event (600 a minute, burst 60, in k8s/prod-v4/deployment.yaml, to be put back after about 26 October) and the owner has Zetaris replace the placeholder User-Agent (open question 3 is closed by doing both); (2) the cutover is on 17 October; (3) the soak is extended rather than accepted as it is, and it runs on production after the cutover, not on staging (part 6 of the runbook: a 48-hour soak against production with V3 standing by); (4) the trusted-Origin bypass becomes a standard-tier grant, option B of open question 4 (pull request 164; the cutover promotes the image that has it).
Evidence is what is in the repository; “owner to confirm” means nothing in it settles the item.
| Gate (WP1) | State |
|---|---|
| Staging at 20 pods with the rate-limit settings | Done (staging runs 20 x 1000m, 6000 a minute, burst 600). The patient liveness probe (pull request 160) is merged but not yet applied to staging. |
| Rate-limit check across 20 pods | Done: each address gets its own allowance since the ingress fix (performance report 5c, 15 October); the limit is shared fairly across pods. |
| CORS | Done (14 and 15 October, docs/soak-scenarios.md scenario 5). |
| Rollback and change drills under load | Done: rollback, roll forward, lost pods, restart of all pods, scaling (performance report sections 4 and 4b, 13 and 14 October). |
| Soak | Partly: 24 hours at 60 requests/s from outside the network, no errors (section 3). The gate asks for 48 hours at least; the shorter run, the ceiling and rollout evidence are what exists. Owner to accept or extend. |
| Capacity | Done: lookups ceiling about 2,100 a second (re-measured 16 October through the single ingress pod); semantic searches per route (section 5d). |
L3 parity against live V3; the UI pass; the cost of the total layer; changelog |
Owner to confirm (the changelog has the V4 entry). |
| Platform partner’s integration test | In progress: the Zetaris scripts work against staging; the test from outside the owner’s network is pending. |
| Cluster hygiene | The CoreDNS block and the ingress source address are done. dex-staging-tls is due to renew on 19 October. |
| Work package | State |
|---|---|
| WP2 production manifests | Written: k8s/prod-v4/ (Deployment, Service, rules and log-level ConfigMaps; no Ingress, Middleware or Certificate, no SQL) and k8s/test_prod_v4_manifests.py. Not applied. kubectl apply --dry-run=server has to be run from the build node (there is no cluster context here). |
| WP3 production secrets | Script ready (scripts/seal-secrets-v4.sh --env prod --apply, tested with stubs: its own directory, never staging’s; the Secret company-dns-v4-credentials; one profile, mediumroast.io, because the server refuses to start if the rules name a profile the Secret lacks). Running it needs the build node. |
| WP4 promotion and cutover script | Not written; the runbook replaces it for the first cutover: the same steps by hand, one command each, with the rollback (Ingress backend flipped back). The script follows, from what the runbook teaches. |
| WP5 to WP7 | Runbook ../v4-production-runbook.md: part 1 (build and put on staging), 2 (prove it on staging, write down the tag), 3 (production beside V3, no public effect), 4 (the cutover, 17 October), 5 (rollback), 6 (after, with the soak on production). Needs the owner at the build node. |
The calendar. D5 asked for a cutover by about 15 October with five days of soak before the freeze on 21 October, and for the fall-back decision no later than 17 October. Today is the 16th: a cutover on the 17th to 19th still leaves two to four days of soak with V3 as the standing rollback; later than that, the fallback (the event on the staging host, cut over after it) is the safer path. The decision is the owner’s.