Steve Recks

Category: Reliability and Quality

  • Stay Off the Pitch

    We’re becoming coaches, not footballers. That’s the idea I’ve been kicking around with colleagues lately. And no, not literally kicking, because most of what I know about football (soccer, for American readers) comes from Ted Lasso.

    In football, the coach stays off the pitch during the match. That’s not just etiquette, it’s the rules. The coach shapes the team all week in training: the patterns, the positioning, the principles, the set pieces. Then on match day, they stand in the technical area and let the players play. They don’t run out to adjust a player’s foot angle before a shot or dictate exactly how to sell a fake. Those details absolutely matter, but they’re not the coach’s to micromanage. A coach who keeps stepping onto the pitch has a worse team, not a better one.

    The shift I’ve been noticing in our work with AI agents feels a lot like that. For most of our careers, we’ve been the footballers. The craft was in the execution: the specific lines of code, the exact shape of a function, the judgment calls at the keystroke level. That craft still matters. But increasingly, the models and the harnesses around them are capable of handling that layer of the work, and the place our judgment creates the most value has moved up a level. It’s in defining what the system should do, what the hard constraints are, what the edges of “good” look like, what failure modes we won’t tolerate. It’s in the training week, not the match.

    The uncomfortable part, at least for me, is that staying off the pitch is hard. Our whole careers have trained us to dive in when something looks off. Seeing an imperfect line of code and not touching it requires real restraint. And here’s what makes the restraint worth building: it isn’t just that stepping onto the pitch wastes your time. Over-instructing the model actively makes it worse. There’s a growing body of research on what some are calling “over-prompting”: the finding that past a certain point, piling on more rules, examples, and step-by-step directions degrades output quality rather than improving it. It turns out the coach who keeps shouting foot-angle corrections from the sideline doesn’t just waste their own breath. They distract the player from playing.

    The part of the metaphor I don’t want to lose: the coach still owns the outcome of the game. Staying off the pitch isn’t the same as checking out. A coach who gives players latitude on execution is still accountable for whether the team wins, whether the strategy was right, whether the players were prepared. That maps almost perfectly onto our work. We’re not handing off responsibility when we let the system do what it’s equipped to do. We’re reserving our attention for the parts where only we can make the call, and we still own the whole outcome of what ships.

  • Where Harness Engineering Earns Its Keep

    Every time an AI agent fails, harness engineering asks a blunt question: what could we encode so this class of mistake can’t happen again?

    That habit is the most distinctive thing I see in this new discipline, and it’s worth unpacking what it actually looks like in practice.

    For anyone who hasn’t run into the term yet, a harness is everything around the model: prompts, constraints, documentation, tools, feedback loops, and architectural rules that shape how an agent behaves. Harness engineering is the discipline of designing all of that deliberately. Strip away the new label and a lot of the mechanics are familiar. Orchestration loops are workflow engines. Sandboxed execution is containers plus capability security. Linter-enforced constraints are evolutionary architecture. Observability for agent runs is SRE practice wearing a new hat. None of it would surprise a senior engineer.

    So where does the rename actually earn its keep? A few places stand out.

    Legibility as a design goal. You’re structuring code, docs, and processes so a probabilistic reasoner can navigate them in one pass. The operational rule is blunt: if the agent can’t see it in context, it effectively doesn’t exist, regardless of whether it lives in Slack, a Google Doc, or someone’s head.

    Context as a first-class resource. What the agent can see, in what order, with what priority directly determines what it will do. Hand it a 1,000-page instruction manual and you crowd out the task itself.

    The compounding loop. When a human teammate keeps making the same kind of mistake, the mature engineering move is to add a lint rule or a CI check, not to send a reminder every Monday. Harness engineering takes that instinct and makes it central. Every time the agent fails, you ask what rule, doc, tool, or test would make that class of failure mechanically hard to repeat, and then you encode it. Each fix compounds across every future agent run. Treating each failure as a gap in the environment rather than a gap in the model is, I think, the working habit most worth adopting from this discipline.

    What stands out to me is the inversion of the job: you’re writing less of the code and more of the constraints, feedback loops, and legibility that govern the thing writing the code. That’s the piece the rename actually captures.

    Curious whether others are noticing this shift in their own work. When your agent-assisted workflow gets better, is it mostly from better prompts, or mostly from better everything-around-the-prompts?

  • Checklists: How You Keep Standards Intact Under Speed

    In complex systems, the enemy is not ignorance.

    It’s cognitive overload.

    That’s why checklists work. Not because engineers can’t think, but because thinking under pressure is expensive. Your working memory shrinks. You miss the obvious. You forget the one step that matters.

    AI makes this better and worse at the same time.

    Better, because it can surface options and remind you of steps you forgot.

    Worse, because it can produce confident output that feels complete, even when it skipped the one check that would have caught the error.

    So the move is not “trust the model more” or “ban the model.”

    The move is: make correctness repeatable.

    A lightweight checklist does that, especially for the moments that matter:

    • Before you merge
    • Before you ship
    • Before you declare something “done”

    Three questions to keep in mind

    1. What would prove this wrong?
    2. What did we assume without verifying?
    3. What’s the blast radius if we’re wrong?

    Checklists aren’t bureaucracy.

    They’re how you keep your standards intact when speed is tempting.

  • Use AI to Widen Thinking, Use Evidence to Narrow Decisions

    AI is great at widening your thinking.

    It is not great at narrowing your decisions.

    That sounds subtle, but it explains a lot of the “why did we ship that?” moments.

    An LLM can brainstorm alternatives, surface edge cases, draft an implementation, even argue both sides. It’s a force multiplier for exploration.

    But commitment requires something different: evidence.

    • What requirement does this satisfy?
    • What constraint does this respect?
    • What test proves it works?
    • What could fail in production?

    If you let the tool make decisions by vibe, you get fast output and slow rework.

    If you use it to widen the solution space, then use checks to narrow the final choice, you get something better: speed you can trust.

    A practical pattern

    1. Explore wide with the model (options, tradeoffs, failure modes).
    2. Decide narrow with artifacts (acceptance criteria, diffs, tests, review).

    Use AI to widen thinking.

    Use evidence to narrow decisions.

  • Why AI Loops Can Compound Errors

    The risky part of AI-assisted engineering is not a single wrong answer.

    It’s a loop.

    Generate a draft. Apply it. Run into an error. Paste the error back. Generate a fix. Apply it. Repeat.

    That loop feels productive because something is always happening.

    But loops compound mistakes. A small misunderstanding in step one can turn into a large, confident mess by step five, especially when the tool is optimizing for momentum instead of truth.

    If you want speed and correctness, you need stop conditions and guardrails.

    Guardrails that keep loops honest

    • Time-box the loop: “Two retries, then reassess the approach.”
    • Force diffs, not dumps: “Propose a minimal patch, not a full rewrite.”
    • Pin the intent: “Restate the goal and constraints before changing code.”
    • Add a verification step: “What tests or checks prove this fix is correct?”
    • Escalate when the error repeats: “If the same symptom appears twice, stop and diagnose.”

    The goal is not to slow the tool down.

    It’s to keep the tool from dragging you forward in the wrong direction.

  • Acceptance Criteria: The Fastest Reliability Upgrade

    A lot of “AI bugs” aren’t bugs.

    They’re missing acceptance criteria.

    If you ask an LLM to “build the feature,” it will build a feature. Usually a plausible one. Sometimes even a good one.

    But plausibility is not correctness.

    The fastest way to turn AI-assisted work from vibe-driven to reviewable is to write acceptance criteria before you generate anything. Not as bureaucracy. As a forcing function.

    A simple acceptance-criteria pattern

    • Inputs: what the system receives
    • Outputs: what it must produce
    • Rules: what must always be true
    • Edge cases: what can’t break
    • Failure behavior: what it should do when it cannot proceed

    Then give the model a job that’s actually tractable:

    “Given these criteria, propose an implementation plan. Highlight any ambiguous or conflicting requirements.”

    What changes immediately

    1. You catch the missing decisions while they’re still cheap.
    2. Review becomes about whether criteria are satisfied, not whether the output “looks right.”

    If you want reliability, don’t start with generation.

    Start with a definition of done.

  • “Hallucinations” Are Often a Context Problem

    Most “hallucinations” aren’t random.

    They’re what happens when you leave a gap and ask a system built to complete patterns to keep going anyway.

    LLMs are not shy about uncertainty. They will produce a clean, fluent answer that sits right on top of missing context. Not because they’re trying to deceive you, but because the engine is optimized to continue the most likely sequence.

    If you want reliability, your job is not “tell it to stop hallucinating.”

    Your job is to make guessing expensive and stopping easy.

    Simple guardrails that help

    • Ask for uncertainty up front: List what you would need to know to be confident.
    • Force a boundary: If information is missing, say 'INSUFFICIENT INFO' and stop.
    • Separate facts from inferences: Two sections: Verified vs Assumptions.
    • Require sources when it matters: Cite where each claim comes from.

    The mindset shift

    When the output is wrong, don’t only blame the model.

    First ask: Did I leave it room to guess?

  • Do LLMs Have “Peak Hours”? And Should We Be Asking for SLAs?

    Have you ever wondered what a service-level agreement for an LLM would even look like?

    The other day, I was using Microsoft Teams and noticed an option to try GPT-5. When I clicked it, a disclaimer popped up saying results might vary depending on when the request is made. I don’t remember the exact wording, but it definitely made me pause—because I’ve heard this same idea come up again and again.

    Plenty of developers swear that model quality dips during “prime time,” like Wednesday afternoons, compared to early Saturday mornings. Anecdotes aren’t data, but the consistency of these stories is hard to ignore.

    Personally, I’d rather wait a little longer and get the same quality every time.

    Why variability might be real

    What’s interesting is how little the major providers say about quality variability. My hunch—and something I’d genuinely love to test—is that API usage (where you pay per call) may give you more consistent results.

    But flat-rate services like Copilot, ChatGPT Plus, or Claude’s paid plans may group you into a shared-capacity pool.

    A plausible mechanism (no inside info required)

    Under the hood, these systems tokenize your prompt, run it through layers of neural computation, and look for the most relevant patterns they can find within a set compute budget.

    “Thinking harder” basically means the model gets to take more inference steps. If usage is high, it may not take those extra steps. It just settles faster, which can feel like a dip in quality.

    A lot of this is based on general principles of how these systems work rather than inside information, but the clues are piling up.

    Where this may be headed

    I suspect we’re heading toward a world where enterprises start asking for real SLAs around model consistency.