company_dns

Migrate company_dns from Azure Container Apps to on-prem MicroK8s

Partially superseded (2026-09-30) by v4-deployment.md: this doc’s reusable, still-true facts — the live cluster survey (nodes, namespaces, ingress/Traefik resolution, TLS, storage classes, SealedSecrets availability) and the proven k8s/prod/ manifest pattern — are now carried forward and kept current in that doc’s §1.3/§3.8, since V4 needs them too (specifically to add dev/staging environments this doc explicitly deferred, its own line below: “No introduction of dev/staging tiers (can be a later follow-up if needed)” — that follow-up is v4-deployment.md §3.4). Everything below this notice stays exactly as it is, unedited: it’s the historical record of V3’s actual Azure→K8s migration (already complete), not a living document — don’t add new V4 decisions here, they belong in v4-deployment.md.

Status: Migration complete. company_dns is live on the cluster at https://company-dns.mediumroast.io (verified: TLS cert issued and valid, /health and representative endpoints returning correct responses), the Azure Container App has been deleted, and the Azure deploy steps have been removed from CI. Remaining work is entirely optional/deferred: DNS Phase 2 (dedicated Dynamic DNS host, see below) and, separately, whether to pursue the caching/async-I/O tier of the EDGAR/Wikipedia performance work now that the two concrete bug fixes are live in production. Owner: michael.hay@mediumroast.io Target cluster: on-prem MicroK8s HA cluster, worker nodes host-1 and host-2 Reference implementation: mediumroast.io repo, V6_K8s branch, k8s/prod/ + scripts/build-and-deploy.sh

Goal

Stop running company_dns as an Azure Container App and run it instead as a Deployment on the existing on-prem MicroK8s cluster, alongside other mediumroast.io services, with TLS termination handled the same way (cert-manager + Traefik + Let’s Encrypt) and no change to the public hostname (company-dns.mediumroast.io).

Why this shape

Non-goals

Current state (as of this plan)

Azure Container App (to be decommissioned)

Defined in container-app-config.yaml (removed from the repository when the Azure deployment was decommissioned):

Setting Value
Image ghcr.io/miha42-github/company_dns/company_dns:latest
Ingress external, target port 8000, CORS *
Resources 0.5 CPU / 1Gi memory
Scale min 0, max 10 replicas, 300s cooldown
Env COMPANY_DNS_HOST=company-dns.mediumroast.io

Deployed by .github/workflows/main.yml on a monthly cron (0 12 1 * *) and on manual dispatch, via azure/container-apps-deploy-action@v2, using secrets: AZURE_CREDENTIALS, AZURE_RESOURCE_GROUP, AZURE_CONTAINER_APP_NAME, AZURE_CONTAINER_APP_ENV.

Target cluster (reference: mediumroast.io k8s/prod/)

Setting Value
Ingress controller Traefik (ingressClassName: traefik)
TLS cert-manager, ClusterIssuer/letsencrypt-prod (cluster-wide, already exists — HTTP-01 via Traefik)
LoadBalancer IPs MetalLB pool <metallb-pool>
Scheduling 2 effective worker nodes: host-1, host-2 (pod anti-affinity spreads replicas across them)
Secrets Bitnami SealedSecrets controller (used for private creds; not needed here — image is public)
Existing namespace pattern one namespace per app+tier, e.g. mediumroast-web

These rows are taken from the reference repo’s docs, not a live query of the cluster. Section “Live cluster survey” below has open items to confirm before writing manifests for real.

Live cluster survey — run before finalizing manifests

Run these and report back (paste output into this doc’s “Survey results” subsection, or hand output back in chat):

microk8s kubectl get nodes -o wide
microk8s kubectl get namespaces
microk8s kubectl get pods -A -o wide
microk8s kubectl -n ingress get svc,pods
microk8s kubectl get clusterissuer
microk8s kubectl get storageclass
microk8s kubectl -n mediumroast-web get certificate
microk8s kubectl get crd | grep -i sealedsecret
microk8s kubectl get ingressclass

What we’re confirming:

  1. host-1 / host-2 are still the effective worker nodes (i.e. no topology change since the mediumroast.io doc was written).
  2. letsencrypt-prod ClusterIssuer still exists and is Ready — if so, we reuse it as-is (no new certmanager.yaml needed, just a Certificate resource for company-dns.mediumroast.io).
  3. Traefik ingress class name and entrypoints (web / websecure) match what’s assumed below.
  4. No naming collision with company-dns as a namespace name.
  5. MetalLB / external IP path is unchanged (only matters if DNS needs to point at a new IP — see Cutover section).

Survey results

Run 2026-09-25/26 against the live cluster. All core assumptions confirmed; one new item to decide (ingress class choice).

No changes to the plan’s manifest shape are needed from this survey, other than resolving the ingress-class question above before writing k8s/prod/ingress.yaml.

k8s manifests (authored — see k8s/)

Directory in this repo, mirroring mediumroast.io/k8s/:

k8s/
  prod/
    namespace.yaml
    deployment.yaml
    service.yaml
    ingress.yaml
    middleware-redirect.yaml
    certificate.yaml
  README.md          # short "how to deploy" note, links to scripts/build-and-deploy.sh
scripts/
  build-and-deploy.sh

k8s/prod/namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: company-dns
  labels:
    app.kubernetes.io/name: company-dns

k8s/prod/deployment.yaml

Key points, adapted from mediumroast-website’s deployment and from container-app-config.yaml’s existing resource sizing:

k8s/prod/service.yaml

ClusterIP, port 80 → targetPort: 8000, matching the service pattern used for mediumroast-website.

k8s/prod/ingress.yaml

Two Ingress resources, ingressClassName: traefik on both (confirmed as the correct class — see Survey results above), copied field-for-field from the live mediumroast-website Ingress config (not just the repo file, which matches):

CORS: confirmed — company_dns.py already sets FastAPI’s CORSMiddleware with allow_origins=["*"] at the app layer (company_dns.py:94-101); the old Azure corsPolicy block in container-app-config.yaml was redundant with it. No ingress-level CORS annotations needed.

k8s/prod/middleware-redirect.yaml

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: redirect-https
  namespace: company-dns
spec:
  redirectScheme:
    scheme: https
    permanent: true

Identical to mediumroast.io/k8s/prod/middleware-redirect.yaml, just in the company-dns namespace, metadata.name: redirect-https kept the same as the reference. Gotcha confirmed from the live data above: Traefik’s annotation-based Middleware reference on a plain Kubernetes Ingress (as opposed to a native IngressRoute) is namespace-prefixed — mediumroast-web’s middleware, named redirect-https, is referenced as mediumroast-web-redirect-https@kubernetescrd, i.e. <namespace>-<name>@kubernetescrd, not just <name>@kubernetescrd. So company-dns’s ingress.yaml must reference it as company-dns-redirect-https@kubernetescrd.

k8s/prod/certificate.yaml

Certificate resource, secretName: company-dns-io-tls, dnsNames: [company-dns.mediumroast.io], issuerRef: letsencrypt-prod (ClusterIssuer), renewBefore: 720h — mirrors mediumroast.io/k8s/prod/certificate.yaml.

CI/CD changes

Per decision: GitHub Actions keeps building/pushing the image only; the cluster deploy step is manual/local, matching how mediumroast.io runs scripts/build-and-deploy.sh from a workstation with cluster access.

Resolved (image tagging): pinned dated tags, not :latest + imagePullPolicy: Always — same rollback-safety reasoning mediumroast.io used. Implemented in scripts/build-and-deploy.sh: it does its own docker build/docker push to ghcr.io/miha42-github/company_dns/company_dns:<TAG> (default tag MMDDYYYY) rather than reusing whatever CI most recently pushed to :latest — this keeps “what tag is deployed” fully explicit and independent of CI’s monthly/dispatch cadence, and sidesteps needing to docker pull+retag an existing :latest image (which doesn’t actually save a build). The script then sed-updates the image: field in k8s/prod/deployment.yaml, applies everything in k8s/prod/, and waits for rollout — see k8s/README.md for the full flow (first-cutover ordering vs. routine redeploys).

Remaining CI change (not yet done — see Migration step 10): in .github/workflows/main.yml, remove the Azure Login and Deploy to Azure Container App steps once Azure is decommissioned. The Build and push step to ghcr.io:latest can stay as-is even though the k8s deploy script no longer consumes it directly — other things may still reasonably want a floating :latest (e.g. someone doing docker pull to try the service locally), and removing it isn’t necessary for this migration.

Migration steps (execution order)

  1. Survey the live cluster (see above) and update this doc’s assumptions. Done.
  2. Author manifests (k8s/prod/*.yaml) and scripts/build-and-deploy.sh; resolve the Dockerfile’s non-root user question. Done — see k8s/prod/, scripts/build-and-deploy.sh, k8s/README.md, and the Dockerfile’s new USER company_dns step.
  3. Deploy in parallel with Azure still live… Done, with two real bugs hit and fixed along the way (not caught by review — only surfaced by the actual cluster rejecting the pods):
    • CreateContainerConfigError: cannot verify user is non-root — the Dockerfile’s USER company_dns set a named user, but Kubernetes’ runAsNonRoot admission check can only verify a numeric UID without executing anything in the image. Fixed in PR #95: USER 10001 (numeric) instead.
    • deployment "company-dns" exceeded its progress deadline on the redeploy — kubectl apply reported the Deployment unchanged even though the image content had changed, because scripts/build-and-deploy.sh’s default tag (date +%m%d%Y, day granularity) collided with the same day’s earlier failed attempt. Same tag string → no diff → no new rollout, and imagePullPolicy: IfNotPresent would have reused the stale locally-cached image even if it had rolled out. Unblocked in the moment with an explicit unique tag (./scripts/build-and-deploy.sh 09262026-2); the default tag format is still day-granularity as of this writing — follow-up: worth tightening to include time or a git SHA so same-day redeploys during active debugging don’t collide again.
  4. Smoke test… Done — /health, merged EDGAR+Wikipedia firmographics for IBM, and the SPA root all verified via direct port-forward before DNS was touched.
  5. Apply Ingress + Middleware + Certificate… Done.
  6. Cut over DNS — Phase 1 (CNAME to www.mediumroast.io)… Done. Propagated within minutes; cert-manager’s solver picked it up and the certificate went Ready shortly after.
  7. Verify TLS + traffic against the new hostname… Done — TLS handshake verified (Let's Encrypt, correct CN/SAN), /health and the merged firmographics endpoint both returned correct 200 responses over https://company-dns.mediumroast.io.
  8. Decommission Azure Container App immediately… Done — the user confirmed via Azure admin access that the Container App (and some other stale ones) are deleted. GitHub Actions’ Azure steps and container-app-config.yaml removed in the cleanup PR alongside this status update.
  9. Cut over DNS — Phase 2 (dedicated Dynamic DNS host), manual, deferred: once Phase 1 (step 6) has been running and verified for a while, manually register company-dns as its own Namecheap Dynamic DNS-tracked host (same mechanism www.mediumroast.io uses), decoupling company-dns.mediumroast.io from www’s DNS lifecycle. Not a blocker for anything else in this plan — Azure teardown (step 8) and repo cleanup (step 10) don’t depend on it. Done manually by the user; not something this plan schedules a date for.
  10. Remove Azure-specific files from this repo… Done: container-app-config.yaml removed and the Azure Login/Deploy to Azure Container App steps stripped from .github/workflows/main.yml (the Build and push step to ghcr.io:latest stays — still useful independent of Azure). Still outstanding, user-side: removing the now-unused AZURE_CREDENTIALS, AZURE_RESOURCE_GROUP, AZURE_CONTAINER_APP_NAME, AZURE_CONTAINER_APP_ENV, AZURE_SUBSCRIPTION secrets from the repo’s GitHub Actions settings — cosmetic at this point (nothing references them anymore), not urgent.

Step-by-step execution guide

This is the operational checklist — concrete commands, in order, with who runs each one and where. The numbered rationale above (“Migration steps”) explains why; this section is what to actually type.

Where things stand right now (as of this write-up): all of the code for this migration — the EDGAR/Wikipedia performance fixes, the Dockerfile non-root user change, and everything under k8s/ and scripts/ — exists only as uncommitted/local-to-this-worktree changes on the claude/repo-overview-ed2879 branch. Nothing is merged to main yet, and .github/workflows/main.yml has deliberately not been touched — removing its Azure deploy steps is migration step 10, which only makes sense after Azure is actually torn down (step 8/9 below). Until then, the existing monthly/dispatch workflow continues deploying to Azure exactly as before; this migration doesn’t interfere with it.

None of the remaining steps below (image build, kubectl apply, DNS edits, az commands) can be run from this Claude session — no docker, kubectl, or az CLI access here, by design (see “Live cluster survey” above). Steps are marked [you] where you run them yourself, on a machine with the relevant access (your workstation, or directly on host-1).

0. Land the code on main

  1. [Claude] Commit and push the worktree’s changes, open a PR.
  2. [you] Review the PR (bug fixes to lib/edgar.py/lib/wikipedia.py in one commit, k8s/Dockerfile/docs in another) and merge it to main.
  3. [you] On whatever machine you’ll run build-and-deploy.sh from (needs docker, kubectl pointed at the cluster, and git): 
    cd ~/dev/company_dns   # or wherever your clone lives
git checkout main
git pull
  4. [you] Create .env in the repo root (gitignored) with a GitHub PAT that has write:packages scope, if you haven’t already: 
    echo "GITHUB_PAT=ghp_xxxxxxxxxxxxxxxxxxxx" > .env

1. Bring the workload up (no traffic yet)

[you], from the main checkout with kubectl pointed at the cluster:

kubectl apply -f k8s/prod/namespace.yaml -f k8s/prod/deployment.yaml -f k8s/prod/service.yaml

This still references the placeholder image: ...:bootstrap tag in deployment.yaml — the pods will ImagePullBackOff until the first real build. Build and push one now:

./scripts/build-and-deploy.sh

This builds from your current main checkout (so it includes the EDGAR/ Wikipedia fixes and the non-root Dockerfile change), pushes ghcr.io/miha42-github/company_dns/company_dns:<today's-date>, updates deployment.yaml’s image: field to match, re-applies everything in k8s/prod/ (harmless at this point — Ingress/Certificate don’t exist yet so nothing new happens there), and waits for the rollout.

kubectl -n company-dns get pods -o wide
# Expect 2/2 pods Running, spread across host-1 and host-2

2. Smoke test directly against the Service (Azure still live, no DNS touched)

[you]:

kubectl -n company-dns port-forward svc/company-dns 8080:80

In another terminal:

curl -s http://localhost:8080/health | jq .
curl -s http://localhost:8080/V3.0/na/sic/description/oil | jq '.data | keys[:3]'
curl -s http://localhost:8080/V3.0/global/company/merged/firmographics/IBM | jq '.code, .message'
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8080/

Compare a couple of these against the live Azure instance (https://company-dns.mediumroast.io/...) to confirm parity. This is also the first real chance to observe whether the EDGAR/Wikipedia fixes actually moved the needle on latency for merged/firmographics — worth timing both. Stop the port-forward (Ctrl-C) once satisfied.

3. Apply Ingress + Middleware + Certificate (still before touching DNS)

[you]:

kubectl apply -f k8s/prod/ingress.yaml -f k8s/prod/middleware-redirect.yaml -f k8s/prod/certificate.yaml
kubectl -n company-dns get certificate
# STATE will show it's not Ready yet - expected, DNS doesn't point here yet

4. DNS cutover — Phase 1

[you], manually on Namecheap — see “DNS cutover — Namecheap instructions” → “Phase 1” below for the exact record edit. Then:

dig @<public-resolver> +short company-dns.mediumroast.io

kubectl -n company-dns get certificate -w
# wait for READY=True, then Ctrl-C

5. Verify over the real hostname

[you]:

curl -sv https://company-dns.mediumroast.io/health
curl -s https://company-dns.mediumroast.io/V3.0/global/company/merged/firmographics/IBM | jq '.code'
curl -s -o /dev/null -w '%{http_code}\n' https://company-dns.mediumroast.io/

If any of this fails: revert the Namecheap CNAME back to company-dns-dev.bluetree-243cfc38.westus.azurecontainerapps.io. (see DNS cutover → Phase 1 → Rollback) and stop here — diagnose before proceeding. Azure is completely untouched at this point, so this rollback is instant.

If it all passes: per the aggressive-migration decision earlier in this doc, proceed straight to Azure teardown — no extended burn-in.

6. Tear down Azure

[you] — now unblocked, you have Azure admin access. Find the actual resource names first if you don’t have them memorized (the GitHub Actions secrets AZURE_RESOURCE_GROUP / AZURE_CONTAINER_APP_NAME / AZURE_CONTAINER_APP_ENV hold these, but secret values aren’t readable from GitHub once set — easier to just ask Azure):

az account show   # confirm you're on the right subscription
az containerapp list -o table

Then follow “Teardown plan — Azure Container App” below step by step (backup export → delete Container App → delete Container App Environment if unused → remove GitHub Actions secrets → remove the asuid.company-dns TXT record → remove container-app-config.yaml from the repo).

7. Update the GitHub Actions workflow

[you or Claude], once Azure is torn down: remove the Azure Login and Deploy to Azure Container App steps from .github/workflows/main.yml, in the same PR that removes container-app-config.yaml. This is intentionally the last code change, not something to do preemptively — until Azure is actually gone, that workflow is the thing keeping Azure in sync if anyone pushes to main or manually dispatches it in the meantime.

8. (Later, unscheduled) DNS Phase 2

[you], whenever — see “DNS cutover — Namecheap instructions” → “Phase 2” below. Not blocking, not time-boxed.

DNS cutover — Namecheap instructions

All DNS steps in this section are performed manually by the user, not by Claude — noted here for completeness/reference only. This is a deliberate two-phase approach: Phase 1 is the lower-risk option to prove the cluster deployment actually works before investing in a dedicated DNS host for it; Phase 2 decouples company-dns from www’s DNS lifecycle once that’s proven, and is not time-boxed or required for anything else in this plan.

Current live records for company-dns.mediumroast.io (looked up 2026-09-26, dig):

Type Host Value
CNAME company-dns company-dns-dev.bluetree-243cfc38.westus.azurecontainerapps.io.
TXT asuid.company-dns A2A0F9669E8D47DF65FD3500D85253C2C75BE9DA140D7801B25E4A70F30BEAA4

The TXT record is Azure’s custom-domain ownership verification record (asuid.<subdomain> is the format Azure Container Apps requires to bind a custom hostname) — it has no effect once the CNAME no longer points at Azure, but do not delete it until Azure teardown (Teardown step 6): if you need a fast rollback to Azure before teardown, re-pointing the CNAME back works instantly only if the TXT verification record is still there — Azure re-checks it.

Reference pattern already working on this cluster: www.mediumroast.io is a plain Namecheap Dynamic DNS A record, currently <public-ip> (the home router’s public IP, auto-updated by a DDNS client whenever the IP changes — see ~/dev/mediumroast.io/CLAUDE.md). No separate router port-forward is needed for company-dns — port 80/443 → Traefik (<traefik-lan-ip>) is already forwarded for www, and Traefik does host-header-based routing to pick the right Ingress (company-dns.mediumroast.io → the company-dns namespace’s Ingress). The only thing that needs to change is where the DNS name resolves to.

Phase 1 — CNAME to www.mediumroast.io (migration step 6)

Rather than registering company-dns as its own DDNS-tracked A record up front, point it at the name that’s already DDNS-tracked, to isolate “does the cluster deployment work” from “does a new DDNS host work”:

Type:  CNAME Record
Host:  company-dns
Value: www.mediumroast.io.
TTL:   Automatic (or match whatever TTL www.mediumroast.io uses)

This way company-dns.mediumroast.io resolves to whatever IP www.mediumroast.io currently has, with zero extra DDNS configuration.

Steps in the Namecheap dashboard:

  1. Log in to namecheap.com → Domain List → mediumroast.io → Manage → Advanced DNS tab.
  2. Find the existing record: Type CNAME Record, Host company-dns, Value company-dns-dev.bluetree-243cfc38.westus.azurecontainerapps.io.
  3. Edit that record’s Value field only — change it to www.mediumroast.io. (keep the trailing dot, matching Namecheap’s own convention for CNAME targets). Leave Host (company-dns) and Type (CNAME Record) unchanged.
  4. Leave the asuid.company-dns TXT record alone for now (removed later, during Azure teardown).
  5. Save.

Verify propagation:

# From an external resolver (bypasses any local DNS cache)
dig @<public-resolver> +short company-dns.mediumroast.io
# Should eventually return the same A record www.mediumroast.io resolves to

dig @<public-resolver> +short www.mediumroast.io

Namecheap’s DNS typically propagates within minutes, but can take up to the old record’s TTL to fully clear cached resolvers worldwide. Once it resolves to the cluster’s IP, confirm cert-manager picked it up:

microk8s kubectl -n company-dns get certificate -w
# WAIT for READY=True, then Ctrl-C

curl -v https://company-dns.mediumroast.io/

Rollback (Phase 1): edit the same CNAME record’s Value back to company-dns-dev.bluetree-243cfc38.westus.azurecontainerapps.io. — Azure Container App is untouched until the Teardown section runs, so this is a fast, low-risk revert as long as teardown hasn’t happened yet (and the asuid.company-dns TXT record — Azure’s ownership check — hasn’t been deleted).

Phase 2 — dedicated Dynamic DNS host (migration step 9, deferred)

Once Phase 1 has been live and verified for a while, replace the CNAME with company-dns’s own DDNS-tracked A record, using whatever Dynamic DNS client mechanism already updates www.mediumroast.io (see ~/dev/mediumroast.io/CLAUDE.md for how that client is configured — this plan doesn’t duplicate that setup detail since it’s specific to the DDNS client already running on the user’s network, not to company_dns). This removes the indirection through www’s CNAME so company-dns.mediumroast.io has no dependency on www.mediumroast.io’s own DNS record continuing to exist or resolve correctly.

Not a blocker for Azure teardown or repo cleanup — those only need Phase 1 working. This phase has no target date in this plan; it’s a follow-up cleanup step, done manually, at the user’s discretion.

The ingress keeps the callers’ addresses (added 2026-10-15)

A manual change to the cluster, made by the cluster administrator and recorded here. Without it, the per-address rate limit of company_dns cannot tell callers apart.

The problem. The traefik Service in the ingress namespace had externalTrafficPolicy: Cluster, so Kubernetes replaced every outside caller’s address with a node’s address before Traefik saw it. The application, which limits by the last X-Forwarded-For entry, saw one of two node addresses: all callers shared about two allowances. Measured on 2026-10-14: two machines on two addresses got 200 requests a second together (exactly two allowances, burst about 1,225), split 139 and 60, with a correlation of -0.96 between them (docs/performance-report.md, section 5b). Hand-over document: ingress-source-address-fix.md.

The change (2026-10-15).

Verified (2026-10-15). 273 consecutive requests through the address during the change all answered 200; no Traefik pod restarted; the certificate self-check works from a pod on the node without Traefik. Then, with the probe perf_tests/limit_probe.py from three machines on three addresses at the same time: each got 100 requests a second (the configured rate) with the full burst of about 600, 300 a second together, and no trade-off between them (docs/performance-report.md, section 5c).

To undo: kubectl -n ingress patch svc traefik -p '{"spec":{"externalTrafficPolicy":"Cluster"}}' (and the Helm value).

Still to check: with Local in layer 2 mode one Traefik pod carries all the external traffic at a time (the node that announces the address). It idled at about 12 to 13 millicores after the change; whether one pod holds the event load (about 2,100 requests a second measured through both pods) is tested by repeating the ceiling run. Confirmed 2026-10-15 (the owner): the MetalLB address range is not managed by the router’s DHCP service, so no device can be given the ingress’s address.

In-cluster DNS for the public names: the CoreDNS hosts block (added 2026-10-11)

Updated 2026-10-15: the block now answers with Traefik’s ClusterIP (kubectl -n ingress get svc traefik -o jsonpath='{.spec.clusterIP}'), not the LAN address shown below, because of the ingress change above. Where the text below says <traefik-lan-ip> for what CoreDNS answers, read <traefik-clusterip>; the router’s port forward still points at the MetalLB address.

A manual change to the cluster, not a file in k8s/, so it is recorded here. Without it, cert-manager cannot issue or renew any certificate on this cluster.

Why it exists. cert-manager checks an HTTP-01 challenge itself, from inside the cluster, by fetching http://<host>/.well-known/acme-challenge/<token> before it asks Let’s Encrypt to validate. The public names resolve to the home router’s public address (<public-ip>). On 2026-10-11 the router’s port mapping was changed from a node (earlier the build node, then the second worker node) to Traefik’s MetalLB address <traefik-lan-ip>. From then on the LAN, including every pod, could not reach the public address (the Firewalla does not loop LAN traffic back; its staff say loopback is automatic and not configurable, so we could not turn it on). cert-manager’s self-check timed out for every name (failed to perform self check GET request ... context deadline exceeded), which blocked the new company-dns-staging certificate and would have blocked every renewal (the earliest was dex-staging-tls, due 2026-10-19). Reaching Traefik from outside still works (the public name, ports 80 and 443 forwarded to <traefik-lan-ip>): verified from a phone on 5G.

The change. In kube-system’s coredns ConfigMap, a hosts block above the forward line answers the cluster’s public names with Traefik’s LAN address, for pods only:

    hosts {
      <traefik-lan-ip> company-dns-staging.mediumroast.io company-dns.mediumroast.io dev.mediumroast.io staging.mediumroast.io www.mediumroast.io dex.mediumroast.io mediumroast.io
      fallthrough
    }

applied with kubectl -n kube-system edit configmap coredns and kubectl -n kube-system rollout restart deploy/coredns. Checked from a pod: the names resolve to <traefik-lan-ip> and Traefik answers (a 301 to HTTPS for a path that is not a challenge token); the staging certificate then issued on cert-manager’s next retry.

Rules for keeping it right

Host Namespace Certificate (renewal, 2026-10-11)
company-dns.mediumroast.io company-dns company-dns-io-tls (2026-11-25)
company-dns-staging.mediumroast.io company-dns-staging company-dns-staging-io-tls (issued 2026-10-11)
dev.mediumroast.io mediumroast-dev dev-mediumroast-io-tls (2026-11-26)
staging.mediumroast.io mediumroast-staging, vault-staging staging-mediumroast-io-tls (2026-11-19)
www.mediumroast.io, mediumroast.io mediumroast-web, vault-prod mediumroast-io-tls (2026-11-18)
dex.mediumroast.io vault-staging dex-staging-tls (2026-10-19)

(The bare mediumroast.io also has a second public address, <a-second-public-ip>, which Let’s Encrypt may be sent to at renewal. A separate thing to check before 2026-11-18.)

The V4 procedure that uses this is in docs/v4-bringup.md (“The CoreDNS block”).

Teardown plan — Azure Container App

Requires Azure CLI + credentials not available in this session — to be run by the user, or handed to Claude with az access and explicit go-ahead.

Decision (aggressive migration posture): no soft “scale to zero and wait” window and no multi-day observation period. Once migration step 7 (smoke test against the new https://company-dns.mediumroast.io) passes, proceed straight to deletion in the same session. Fallback to Azure is reserved for an actual detected failure during cutover/smoke-test — not scheduled as a matter of course after a successful migration.

  1. Confirm migration step 7 (smoke test against the new hostname) passed.
  2. Export/record the current Container App config for rollback reference — cheap insurance, do this regardless of the faster timeline: az containerapp show -n <name> -g <resource-group> -o yaml > azure-company-dns-backup.yaml (keep outside the repo, or in docs/plans/ as a dated snapshot — it may contain resource IDs but no secrets since none are configured in container-app-config.yaml).
  3. Delete the Container App directly: az containerapp delete -n <name> -g <resource-group>.
  4. Delete the Container App Environment if nothing else uses it: az containerapp env delete -n <env-name> -g <resource-group> (confirm nothing else in the resource group depends on this environment first — this check still applies even on the faster timeline, since it’s irreversible and affects things beyond company_dns).
  5. Remove the now-unused AZURE_* GitHub Actions secrets from the repo settings.
  6. On Namecheap, delete the asuid.company-dns TXT record (Azure’s ownership-verification record — no longer meaningful once the Container App itself is deleted, and leaving it around is just clutter).
  7. Remove container-app-config.yaml (the file no longer exists) from the repo in the same PR that strips the Azure steps from the workflow.

Rollback plan

Open questions for follow-up (not blocking the plan doc, but before execution)

  1. Does company_dns.py already set FastAPI CORSMiddleware… Resolved: yes, company_dns.py:94-101. No action needed.
  2. Confirm the Dockerfile can run as non-root without breaking makedb.py’s file writes… Resolved: makedb.py still runs as root during the build (before the USER company_dns switch), and the Dockerfile chown -Rs /app after, so the non-root runtime user owns everything it needs to read.
  3. Confirm whether / is a safe probe target… Resolved: use the existing /health endpoint instead — it’s lighter than / and clearly intended for exactly this.
  4. Decide on :latest + Always vs. pinned/dated tags… Resolved: pinned dated tags, implemented in scripts/build-and-deploy.sh.
  5. Confirm who has Azure admin access… Resolved: the user has Azure admin access and will run the Teardown az commands themselves (this session still has neither az CLI nor Azure credentials, so Teardown remains a user-executed step either way).
This site is open source. Improve this page.