The atlas

Knowing a tool means knowing when it is the wrong one. This is a working position on each domain, written so an experienced engineer can disagree with it.

Reference

Engineering judgement, not a list of past engagements. For what I have delivered, see the record.

Cloud architecture

Pick failure domains first. Services are the easy part of the diagram.

Try the Blast radius lab →

AWS: Accounts are the real isolation boundary. Separate production by account rather than by naming convention.
Microsoft Azure: Subscriptions and management groups play the same role. Availability zones are not automatic per service, so check each one.
Google Cloud: Projects and global VPCs change the network design: one VPC can span regions, which simplifies routing and widens mistakes.
Multi-cloud: Usually a cost, not a strategy. It is justified by regulation, acquisition or one specific capability, and rarely by avoiding lock-in.
Networking: Plan CIDR ranges early, because re-addressing later forces replacement. Private endpoints beat traffic through NAT on both security and cost.
High availability: Two zones only help if one can carry the full load. Capacity is part of the failover design.
Cost: Attribute before optimizing. Data transfer and idle non-production capacity are common and quiet sources of spend.

DevOps and automation

A pipeline is a series of arguments that a change is safe. Each stage should catch something the previous one could not.

Try the Release path lab →

GitHub Actions: Use OIDC federation so workflows hold no long-lived cloud keys. Pin third-party actions to a commit.
GitLab CI/CD: Protected environments and merge request pipelines give a strong default path with little configuration.
Jenkins: Keep the logic in versioned pipeline files and run ephemeral agents. A hand-configured controller is a liability.
Release management: Build once and promote the same digest. Rebuilding per environment means you never tested what you shipped.
GitOps: The cluster pulls desired state, so CI needs no cluster credentials and rollback is a revert.
Progressive delivery: A canary is only as good as its analysis. Without automated comparison it is just a slower rollout.

Containers and orchestration

Most Kubernetes incidents are about requests, limits and probes. Understand those before adding more platform.

Try the Scheduler lab →

Docker and Compose: Multi-stage builds, minimal base images and a non-root user. Compose remains the right tool for local development.
Amazon EKS: The control plane is managed, but upgrades are still your work. Check add-on and API compatibility first.
Google GKE: Autopilot trades control for less node management. It is a good default unless you need node-level access.
Deployments and StatefulSets: StatefulSets give identity and stable storage, but zonal volumes tie pods to a zone. Prefer a managed database when you can.
Services and Ingress: Readiness probes decide who receives traffic. A liveness probe that checks dependencies can turn a small failure into a restart storm.
Helm and Kustomize: Helm for packaging and distributing. Kustomize for environment overlays on manifests you own.
Autoscaling: HPA reacts to pods, and node autoscaling reacts to Pending pods. Both depend on accurate requests.

Infrastructure as code

The plan is the code review. If nobody reads the plan, the code is only documentation.

Try the Plan reader lab →

Terraform: Declarative and graph-driven. Its value is the reviewable diff between desired and actual state.
State management: Use remote state with locking. Split state by blast radius, so a network change plan cannot touch the database.
Reusable modules: Create one when a pattern has repeated and stabilized. Modules that wrap one resource add indirection and nothing else.
Environments: Same modules, different inputs. Differences between environments should be visible in a short variables file.
Ansible: For configuring hosts that must exist. Baking images is usually better for anything in an autoscaling group.
Provisioning safety: Use prevent_destroy on stateful resources, and detect drift on a schedule so surprises appear before the next change.

Observability and reliability

Alert on what users experience. Causes belong on dashboards, and symptoms belong on pagers.

Try the Error budget lab →

Prometheus: Cardinality is the main operational risk. Never put user IDs or request IDs in labels.
Grafana: One overview per service, built around rate, errors and duration, beats many dashboards nobody opens.
CloudWatch: The natural home for managed AWS service signals. Watch custom metric and log ingestion costs.
Azure Monitor: Log Analytics and KQL give strong ad-hoc analysis. Set retention and table plans deliberately.
Logs and traces: Structured logs carrying a trace ID connect the signals. Sample traces, and keep the errors.
Incident management: Clear roles, a timeline and blameless reviews that produce changes rather than reminders to be careful.
SRE practice: An error budget turns "is it reliable enough?" into a number that product and engineering can both act on.

Security and DevSecOps

Prefer controls that make the unsafe path impossible over checks that report it afterwards.

Try the Access trace lab →

IAM: Short-lived roles rather than long-lived keys. An explicit deny beats every allow, and SCPs set the ceiling.
Microsoft Entra ID: Central identity with Conditional Access. Lifecycle matters as much as sign-in: leavers must lose access everywhere.
SSO: SAML or OIDC federation with group-based role mapping, so access follows the org chart and not individual tickets.
SSL/TLS: Automate issuance and renewal with ACM or cert-manager. Expired certificates are an outage you can entirely prevent.
Network security: Private by default, with explicit paths in. Network policy limits lateral movement inside the cluster.
Scanning: Block on fixable criticals and track the rest. A gate that fails on everything soon gets disabled.
Hardening: Non-root containers, read-only filesystems, no public buckets by policy, and secrets that never enter an image.

AI infrastructure and MLOps

A model is a build artifact with a data dependency. Most delivery discipline carries over, and drift is what is new.

Try the Release path lab →

Model deployment: Version the model, code and data reference together, and promote the same artifact through environments.
Model serving: Batching and cold start dominate GPU serving. Measure latency at realistic concurrency.
Scalable workloads: Scale on queue depth or in-flight requests, because GPU utilization alone says little about user wait.
CI/CD for ML: Add data validation and evaluation against the current model. Shadow traffic is a cheap and safe final check.
Managed AI services: Bedrock, Azure AI and Vertex AI remove serving work. Compare cost per request at your volume before self-hosting.
AI in engineering workflows: Useful for drafting and triage. Keep humans and existing gates on anything that changes production.

The atlas

Knowing a tool means knowing when it is the wrong one. This is a working position on each domain, written so an experienced engineer can disagree with it.

Reference

Engineering judgement, not a list of past engagements. For what I have delivered, see the record.

Cloud architecture

Pick failure domains first. Services are the easy part of the diagram.

Try the Blast radius lab →

AWS: Accounts are the real isolation boundary. Separate production by account rather than by naming convention.
Microsoft Azure: Subscriptions and management groups play the same role. Availability zones are not automatic per service, so check each one.
Google Cloud: Projects and global VPCs change the network design: one VPC can span regions, which simplifies routing and widens mistakes.
Multi-cloud: Usually a cost, not a strategy. It is justified by regulation, acquisition or one specific capability, and rarely by avoiding lock-in.
Networking: Plan CIDR ranges early, because re-addressing later forces replacement. Private endpoints beat traffic through NAT on both security and cost.
High availability: Two zones only help if one can carry the full load. Capacity is part of the failover design.
Cost: Attribute before optimizing. Data transfer and idle non-production capacity are common and quiet sources of spend.

DevOps and automation

A pipeline is a series of arguments that a change is safe. Each stage should catch something the previous one could not.

Try the Release path lab →

GitHub Actions: Use OIDC federation so workflows hold no long-lived cloud keys. Pin third-party actions to a commit.
GitLab CI/CD: Protected environments and merge request pipelines give a strong default path with little configuration.
Jenkins: Keep the logic in versioned pipeline files and run ephemeral agents. A hand-configured controller is a liability.
Release management: Build once and promote the same digest. Rebuilding per environment means you never tested what you shipped.
GitOps: The cluster pulls desired state, so CI needs no cluster credentials and rollback is a revert.
Progressive delivery: A canary is only as good as its analysis. Without automated comparison it is just a slower rollout.

Containers and orchestration

Most Kubernetes incidents are about requests, limits and probes. Understand those before adding more platform.

Try the Scheduler lab →

Docker and Compose: Multi-stage builds, minimal base images and a non-root user. Compose remains the right tool for local development.
Amazon EKS: The control plane is managed, but upgrades are still your work. Check add-on and API compatibility first.
Google GKE: Autopilot trades control for less node management. It is a good default unless you need node-level access.
Deployments and StatefulSets: StatefulSets give identity and stable storage, but zonal volumes tie pods to a zone. Prefer a managed database when you can.
Services and Ingress: Readiness probes decide who receives traffic. A liveness probe that checks dependencies can turn a small failure into a restart storm.
Helm and Kustomize: Helm for packaging and distributing. Kustomize for environment overlays on manifests you own.
Autoscaling: HPA reacts to pods, and node autoscaling reacts to Pending pods. Both depend on accurate requests.

Infrastructure as code

The plan is the code review. If nobody reads the plan, the code is only documentation.

Try the Plan reader lab →

Terraform: Declarative and graph-driven. Its value is the reviewable diff between desired and actual state.
State management: Use remote state with locking. Split state by blast radius, so a network change plan cannot touch the database.
Reusable modules: Create one when a pattern has repeated and stabilized. Modules that wrap one resource add indirection and nothing else.
Environments: Same modules, different inputs. Differences between environments should be visible in a short variables file.
Ansible: For configuring hosts that must exist. Baking images is usually better for anything in an autoscaling group.
Provisioning safety: Use prevent_destroy on stateful resources, and detect drift on a schedule so surprises appear before the next change.

Observability and reliability

Alert on what users experience. Causes belong on dashboards, and symptoms belong on pagers.

Try the Error budget lab →

Prometheus: Cardinality is the main operational risk. Never put user IDs or request IDs in labels.
Grafana: One overview per service, built around rate, errors and duration, beats many dashboards nobody opens.
CloudWatch: The natural home for managed AWS service signals. Watch custom metric and log ingestion costs.
Azure Monitor: Log Analytics and KQL give strong ad-hoc analysis. Set retention and table plans deliberately.
Logs and traces: Structured logs carrying a trace ID connect the signals. Sample traces, and keep the errors.
Incident management: Clear roles, a timeline and blameless reviews that produce changes rather than reminders to be careful.
SRE practice: An error budget turns "is it reliable enough?" into a number that product and engineering can both act on.

Security and DevSecOps

Prefer controls that make the unsafe path impossible over checks that report it afterwards.

Try the Access trace lab →

IAM: Short-lived roles rather than long-lived keys. An explicit deny beats every allow, and SCPs set the ceiling.
Microsoft Entra ID: Central identity with Conditional Access. Lifecycle matters as much as sign-in: leavers must lose access everywhere.
SSO: SAML or OIDC federation with group-based role mapping, so access follows the org chart and not individual tickets.
SSL/TLS: Automate issuance and renewal with ACM or cert-manager. Expired certificates are an outage you can entirely prevent.
Network security: Private by default, with explicit paths in. Network policy limits lateral movement inside the cluster.
Scanning: Block on fixable criticals and track the rest. A gate that fails on everything soon gets disabled.
Hardening: Non-root containers, read-only filesystems, no public buckets by policy, and secrets that never enter an image.

AI infrastructure and MLOps

A model is a build artifact with a data dependency. Most delivery discipline carries over, and drift is what is new.

Try the Release path lab →

Model deployment: Version the model, code and data reference together, and promote the same artifact through environments.
Model serving: Batching and cold start dominate GPU serving. Measure latency at realistic concurrency.
Scalable workloads: Scale on queue depth or in-flight requests, because GPU utilization alone says little about user wait.
CI/CD for ML: Add data validation and evaluation against the current model. Shadow traffic is a cheap and safe final check.
Managed AI services: Bedrock, Azure AI and Vertex AI remove serving work. Compare cost per request at your volume before self-hosting.
AI in engineering workflows: Useful for drafting and triage. Keep humans and existing gates on anything that changes production.