Cutting deploy cost by a third without adding a single dashboard

2026-02-18T00:00:00+00:00

A platform org of about ninety engineers came to us with a familiar complaint: releases used to take a day, now they take most of a week, and nobody could point to exactly why. Headcount had roughly doubled in eighteen months. Deploy frequency hadn't moved.

We ran our usual two-week diagnostic: trace an actual change from ticket to production, write down every wait, every handoff, every manual step. Not the pipeline in the wiki — the one engineers actually route through when the wiki's version is slower than asking someone directly in Slack.

What we expected to find

Going in, the obvious suspect was the CI pipeline itself. Slow test suites, flaky infrastructure, that kind of thing. It's usually where teams look first, because it's the part they can see.

What we actually found

The pipeline was fine. Fifteen minutes, mostly test execution, nothing embarrassing. The other four days and change were almost entirely queueing: waiting for one of two people who understood a particular legacy service well enough to approve changes to it, waiting for a weekly change-advisory meeting that had quietly become mandatory for anything touching billing, waiting for a staging environment that was shared across six teams and perpetually in a broken state left by whoever used it last.

None of this showed up in any dashboard, because none of it was a system anyone was measuring. It was organizational scar tissue — reasonable decisions made individually, two years' worth of them, that added up to a four-day queue nobody had designed on purpose.

What we changed

Split the legacy-service approval bottleneck into a documented ownership boundary with a second trained approver, cutting median wait from a day and a half to under two hours.
Replaced the blanket change-advisory meeting with an actual risk classification, so only the changes that needed a human review before shipping got one.
Gave each team its own disposable staging environment instead of a shared one, which also — as a side effect nobody asked for — cut a class of "worked on staging, broke in prod" incidents to roughly zero.

Where it landed

Median lead time from merge to production dropped from just under five days to a little over three. Cost per deploy — engineer-hours burned on process rather than the change itself — came down by a bit more than a third. We left the pipeline-tracing setup running so the platform team can watch the number themselves instead of taking our word for it next quarter.

The fix, in the end, was mostly deleting things: a meeting, a shared environment, a single point of failure in an approval chain. That's usually where it is.

What we actually check before an eval harness ships

2025-11-04T00:00:00+00:00

Most eval harnesses we inherit on day one of an engagement are built to answer "does this response sound plausible" rather than "did the model do the thing correctly." Those are different questions, and only one of them tells you whether it's safe to change the prompt, swap the model, or ship the next version.

Before we trust an eval harness enough to gate a deploy on it, we run it through the same four checks every time.

1. Does it fail on cases you know are wrong?

The single most common problem: an eval set with no negative examples. If every case in the suite is a real, correctly-handled request, a harness that always returns a fixed canned answer can still score close to 100%. We seed at least a third of the set with inputs we know should fail — ambiguous requests, adversarial phrasing, cases the current model already gets wrong — specifically to confirm the scoring function can tell the difference.

def score(response: str, expected: Expectation) -> float:
    if expected.must_contain and expected.must_contain not in response:
        return 0.0
    if expected.must_refuse and not looks_like_refusal(response):
        return 0.0
    return semantic_similarity(response, expected.reference)

A surprising amount of "eval harness" code we see is just the last line of that function — pure semantic similarity against a reference answer, with no hard failure conditions above it. That catches drift in tone. It does not catch a model confidently returning the wrong account balance.

2. Is the eval set actually yours?

Generic benchmarks tell you how a model performs on the internet's idea of your problem. They don't tell you how it performs on your actual traffic, which has its own weird distribution of edge cases: the customer who pastes in a support ticket instead of a question, the request in a language your prompt wasn't tuned for, the input that's technically valid JSON but structurally nonsense. We build the eval set from real logged requests — sanitized, but real — every time.

3. Does a regression in the eval mean a regression for users?

We run this one backwards: take a change everyone agrees made things worse — a prompt edit that was rolled back, a model downgrade that got reverted — and confirm the eval score actually drops when we replay it. If the harness doesn't notice a change that the team already knows was bad, it isn't measuring the thing that matters, no matter how good the pass rate looks.

4. Who looks at the failures?

An eval harness that reports a number to a dashboard nobody opens isn't a safety mechanism, it's a decoration. Before we sign off on one, there's a named person whose job includes reading the failing cases after every run, not just the aggregate score. Usually that turns up a category of failure the eval set doesn't cover yet, which goes back into check 2.

None of this is exotic. It's the same instinct as a good test suite: assert on outcomes, not vibes, and make sure something fails when it's supposed to. The harness that survives all four checks is the one we're willing to gate a production deploy on. Most of what we're handed on day one survives one or two.

0xE8 - Blog