Evaluating and Iterating on LLM Outputs

Before changing anything, you need to know why the output is wrong. There are exactly two reasons an LLM gives you a bad answer:

1. Prompt failure — the model had the capability, but you didn’t give it enough signal to use it. This is fixable by rewriting the prompt.
2. Model limitation — the model genuinely cannot do what you’re asking, regardless of how you phrase it. Rewriting the prompt won’t help.

Conflating these two is the source of most “prompt engineering rabbit holes.” Practitioners spend hours rephrasing a request the model is fundamentally incapable of answering. The right move — change the architecture, use retrieval, switch models — never gets tried because the diagnosis was wrong.

The five most common prompt failures, in order of frequency:

1. Missing role or goal — the model doesn’t know who it’s answering for or what a good answer looks like
2. Unspecified output format — the model defaults to prose when you need JSON, or vice versa
3. No examples — a complex output structure isn’t learnable from description alone
4. Ambiguous scope — “summarize” means nothing without a target length and audience
5. No grounding constraints — the model invents plausible-sounding details because nothing told it not to

The most common model limitations:

• Knowledge cutoff — no live information after training
• Arithmetic and reasoning over large numbers — LLMs are poor calculators
• Long-context faithfulness degradation — in very long prompts, the model loses track of facts stated early on ^{[Lost in the Middle: How Language Models Use Long Contexts]}
• Data coverage gaps — rare languages, niche domain knowledge, and fictional constructed languages (like the Klingon example)

Once you’ve diagnosed what is wrong, you apply a structured loop — not trial and error:

The discipline of “change exactly one thing” is crucial. If you rewrite the role, add an example, and change the output format all at once, and the output improves, you have no idea which change did it. Next time you hit a different problem, you have no transferable knowledge. Worse: two of those changes may be helping while the third is quietly introducing a new failure.

The iteration loop in action — watch how three sequential prompt improvements turn a useless first response into a production-ready output:

Notice the progression in the widget: the prompt moved from vague instruction → role + goal → format contract → grounding rules. Each step is a response to a specific diagnosed failure, not a gut-feel rewrite.

Not everything is a prompt failure. Knowing where model limitations live prevents wasted iteration.

Knowledge cutoff. Models have a hard training cutoff. Asking for current stock prices, yesterday’s news, or a just-published paper will produce either a refusal or confident hallucination. The fix is always architectural: retrieval, search tools, or API integration — not a better prompt.

Arithmetic and exact computation. LLMs calculate by pattern-matching, not by executing arithmetic. They are unreliable beyond simple single-digit operations. ^{[Measuring Mathematical Problem Solving With the MATH Dataset]} For any calculation that matters, send the expression to a calculator tool and have the model interpret the result — not compute it.

Long-context faithfulness. Counterintuitively, models often do worse on facts stated early in a very long prompt. The model attends more reliably to recent context. Strategies: put critical instructions at both the top and the bottom, and break long documents into smaller chunks if possible.

Rare languages and domains. A model trained on 99% English text will produce unreliable output in languages with sparse training data. The same applies to highly specialized domains — esoteric legal systems, rare programming languages, niche scientific fields — where the training corpus is thin.

Here is the surprising consequence of systematic iteration: every test case you accumulate — every input/output pair you’ve verified — is an asset. Without a harness, fixing one case silently breaks another. With a harness, you know immediately.

The pattern is simple:

1. Every time you confirm a prompt works correctly on an input, record that input/output pair as a test case.
2. Every time you iterate the prompt, run all existing test cases against the new version.
3. A regression (previously-passing case now failing) is a stopping signal — the change introduced a hidden trade-off.

This is not sophisticated engineering. It’s a text file with input, expected, and actual columns. The discipline is in running it every single iteration.

// test-prompt.ts — a minimal harness, no framework needed
type TestCase = {
  input: string;
  expected: string;  // the string the output should contain
};

const cases: TestCase[] = [
  { input: "I love this product!", expected: "POSITIVE" },
  { input: "Arrived broken.",       expected: "NEGATIVE" },
  { input: "When does it ship?",    expected: "NEUTRAL"  },
];

async function runSuite(prompt: string) {
  let pass = 0;
  for (const tc of cases) {
    const output = await callModel(prompt.replace("{{input}}", tc.input));
    const ok = output.includes(tc.expected);
    console.log(`${ok ? "✓" : "✗"} [${tc.expected}]  got: ${output.trim()}`);
    if (ok) pass++;
  }
  console.log(`\n${pass}/${cases.length} passing`);
}