Engineering operations
How we run the system that runs the product: how we plan and stay AI-forward, automation routines, alerting, incidents, testing, security, and the SLA we hold ourselves to.
You built it, you run it. How we run engineering covers service ownership and the on-call model; this page is the operating layer underneath it: how we plan and stay AI-forward, the automation that watches the system, how we alert and respond, how we test and secure what ships, and the SLA we commit to. We grade all of it every week with the engineering health score.
The rule that ties it together: a human sets the standard, agents run the checks. Nothing here replaces an engineer's judgment. It replaces the manual sweep that an engineer used to do by hand and forget to do under load.
Planning and staying AI-forward
Running the system is half the job. The other half is building the next thing well and keeping the way we build ahead of the field. That is an operating discipline, not a side project, and the engineering health score grades it every week alongside uptime and security. A stable platform that has stopped building is not a safe engineering org; in the AI era it is a stalling one.
Deep cross-repo planning. Intempt is dozens of repos, not a monorepo, so any real feature crosses services. Before a major project starts, the eng lead breaks it down across the platform repos, maps the dependency order between them, and matches each slice to the engineer whose depth fits it. The plan names the repos, the order, and the owner per slice, so engineers run in parallel instead of blocking on each other. Who is strong where decides the assignment; the dependency structure between the repos decides the sequence.
AI-forward engineering. We already build with agents every day, as how we run engineering sets out. Keeping that edge is the lead's standing research: reading how Anthropic and other AI-first teams actually work (self-verifying loops that write code from a design file and run their own tests, specialized sub-agents working in parallel, test-driven development guided by an agent, automated pull-request review, incident triage fed straight from stack traces) and trialing the methods that move our output. A practice that was ahead of the field last quarter is the floor this one. Output per engineer rises because the method improves, not because the week gets longer.
High-reliability practice. Three and four nines are not reached by being careful; they are reached by an operating model. The lead studies how the teams that hold that bar run and adopts what fits: an error budget that ties the pace of shipping to the reliability we have left, a change freeze when the budget is spent, progressive delivery that limits the blast radius of a change, and blameless postmortems that feed prevention rather than blame. Most outages come from changes, so the model governs change instead of relying on heroics.
The automation layer
Nine automations watch engineering health. Eight are scheduled cloud routines that run in the claude.ai cloud and post to Slack as the team. The ninth is a GitHub Actions machine, separate because it needs each repo's build toolchain, which the cloud sandbox does not have.
| Routine | Runs | Posts to | What it watches |
|---|---|---|---|
| Security monitor | Weekdays 09:00 | #t-security | The security backlog: open findings, triage, what is aging |
| Cluster monitor | Weekdays 09:00 | #t-ops | Cluster and infrastructure status |
| PR monitor | Weekdays 09:00 | #t-release | The open-PR board across the org, sorted by what each PR is waiting on |
| Bug routine | Weekdays 09:00 | #t-eng, #t-qa | The bug board: open count, aging, mean time to resolve, work in progress |
| Changelog monitor | Daily 09:00 | #t-product | What merged to production, written as a customer-facing digest |
| Changelog rollup | Fridays | #t-product | The week's shipped work, opened as a docs changelog PR |
| Maintenance monitor | Weekdays, when changed | #t-eng | New findings from the code-maintenance machine |
| Weekly exec summary | Fridays | Email draft, #t-eng | The week across every routine, drafted as an exec summary |
The ninth, the code-maintenance machine, runs in GitHub Actions: a daily read-only scan writes findings to a branch and to the tracker, a capped dispatch turns a bounded number of findings into fix branches a day, and each fix is built and tested before it opens a pull request. It is capped on purpose, because the review queue, not the scan, is the constraint.
Together they mean nobody has to remember to check the bug board, the PR queue, the security backlog, or the cluster on a Monday morning. The check already ran and the result is in the channel.
Alerting and on-call
Alerts come from Grafana Alerting, evaluating rules over Prometheus, Loki, and CloudWatch metrics. A firing alert runs an escalation through Grafana OnCall and lands in Slack, where the team can acknowledge and resolve it. A dedicated external pager for the highest-severity alerts is the direction we are building toward, so a critical page reaches a phone even when nobody is in Slack.
On-call is a small, named rotation, not the whole team:
- Primary holds the pager 24/7 and owns restoration: rollback, restart, scale, cordon, silence. The primary stabilizes first and diagnoses second.
- Backup takes the code fix when restoration needs a real change, so the primary stays free to command the incident.
- Last resort is the founder, paged only when the ladder above has not acknowledged.
- A product owner runs customer communication.
Escalation timers run on acknowledgment, not resolution: if the page is not acked, it climbs the ladder. The pay model for on-call is time off, not a cash premium, and it is set out in on-call compensation, which is the single source of truth for those numbers.
Incident response
An incident is not only an outage. A customer war room, a regression, a security event, or a bad launch all count, and anyone pulled in is eligible under the on-call policy. The lifecycle:
- Detect. An alert fires, or a human reports it.
- Triage. The responder acknowledges first, which stops the escalation ladder, then classifies severity. For a critical incident, a short holding statement goes out before deeper diagnosis.
- Declare. The responder declares; no approval needed. A dedicated incident channel is opened and the timeline is kept live in it.
- Mitigate. The responder owns restoration and stays incident commander. A code fix pages the owning engineer, who fixes while the commander keeps running the incident.
- Communicate. Early updates are holding statements with no speculative root cause or ETA. Substantive customer updates come from the product owner. Status is posted at status.intempt.com.
- Close. A critical incident gets a blameless postmortem: what happened, the timeline, the root cause, and the fix. Every incident is reviewed once more in a weekly pass with a named owner and a next date.
Severity here is our internal operating scale and is separate from the customer-facing severity in the SLA below. A postmortem blames the system, never the person.
How we test
Testing is layered, and the gate differs by layer. The short version: unit and fast checks gate the merge; the heavy integration and browser suites run on a schedule against a live environment.
| Layer | What runs | When it gates |
|---|---|---|
| Java services | Unit tests, Testcontainers integration tests, Checkstyle, and a mutation gate on the billing service | On push to the build pipeline |
| Central integration | The Citrus end-to-end suite in the integration-tests-platform repo, against a live environment | Daily, plus a weekly QA dispatch |
| Console | Type-check on every build; the component and Playwright suites run on a weekly QA pass | Build gates the merge; browser suite runs weekly |
| Desktop and mobile | The fullest CI estate we run: unit, component, accessibility, Rust, lint, and dependency-audit jobs | On every pull request |
| CLI | Unit and command tests | On pull request and on merge |
| Website and docs | Content gates (banned copy, navigation, lead-form tracking, recipe catalog) plus a page-render health crawl | On pull request; the crawl also gates the deploy |
The QA skill pipeline sits on top of this for feature work: scenarios are written from the spec, mechanized into a scenario document, then run as automated tests, live browser checks against staging, or live API checks. A pull request review walks back to the feature's requirements first and stops if there are none, rather than reviewing a diff with nothing behind it.
Coverage is enforced by mutation testing where it matters most (billing), not by a blanket percentage everywhere. A test that has never failed has never been tested, so a new guard is proven red before it is trusted green.
How we secure what ships
Security runs as a shared pipeline across the repos, plus a monthly audit and a human-paced fix lane.
| Control | What it does | Blocks a merge? |
|---|---|---|
| Secret scan | Catches a new secret in the diff | Yes |
| Dependency scan | Flags vulnerable dependencies | Yes, at critical |
| Image scan | Scans the container image | Yes, at critical |
| Infrastructure scan | Checks Dockerfiles and manifests for misconfiguration | Reports, ratcheting toward blocking |
| Code analysis | Static analysis on the higher-risk web and service code | Reports |
A monthly audit scans every repo, scores findings by severity and exploitability, and files them to the tracker. Fixing is deliberately separate from scanning and runs one finding to one pull request, because scanning is cheap and wide while fixing is expensive and serial, and the review queue is what limits it.
Merges are protected the same way everywhere: production branches move only by fast-forward from staging, every change needs a code-owner approval, and the pipeline files themselves are owned so the gate cannot be edited without the owner.
The SLA we commit to
Our public commitment is on the SLA page. Two tiers:
- Standard, on every paid plan: 99.9% uptime on data collection and the console, with latency targets on experiences, segmentation, pipelines, and analytics.
- Premium, for Enterprise on an Order Form: 99.99% on data collection, 99.95% on the console, tighter latency targets, and 24/7 response.
Support response is by severity, fastest for a Sev 1 (a critical production outage or data or security incident). When we miss a commitment, the remedy is a service credit on the next invoice, capped at 20% of the monthly fee, claimed within 30 days. The SLA page is authoritative for every number and definition; this page is how engineering operations holds us to it.
The engineering health score is how we check, every week, that the operating layer on this page is actually working, from security to QA to reliability.
Want to work at Intempt? Check out our open positions.