Denys Miasnychenko

Software Architect / DevOps

Nav

Keeping an LLM honest about what it doesn't know

Scope

If you work with LLMs a lot, you’ve run into this. You discuss a solution with a model and get an answer with specific code or facts, then find out later that the code was deprecated six months ago and the model keeps pushing year-old solutions. Worse, you write code with correct modern syntax and the model “corrects” it to an outdated version. The saddest case is a modern model insisting that something current doesn’t exist. A recent Claude Opus might claim that Anthropic’s newest model is Sonnet 3.5, and a recent Gemini Pro would recommend Gemini 1.5 Pro as the latest model, claiming it has a million-token context.

This isn’t a bug in one particular model. It’s a property of any LLM: the model doesn’t distinguish between what it “knows” from training and what is true now. On top of that, training data is embedded in the model’s weights, so by definition it counts for more than your short prompt.

The problem goes beyond amusing anachronisms. Old APIs and unknown library calls mean, at best, reinventing the wheel: the model writes an interpreted script to replace a built-in function because it doesn’t know the function exists. It can also cost real money. Once a model silently replaced my call to gpt4.1-mini with gpt-4o (claiming “there’s no such thing as mini”), and I went from fractions of a cent per request to several cents.

What doesn’t work

The first reaction is to write something like “always verify versions before answering” in CLAUDE.md (I work with Claude Code/Claude.ai, so all examples here use that stack). This doesn’t work because the instruction is too abstract. The model agrees and keeps answering from training data, especially since not every request looks like it needs that kind of check.

The second attempt is to put specific library and API versions directly in CLAUDE.md. That works better but doesn’t scale. With several projects on different stacks, one complete list of versions in one place is wrong, separate lists per project get confusing, and you still have to maintain them.

The third idea is to tell the model “it’s 2026, your data is outdated.” The model agrees, then writes code for React 17 and includes haiku-3.5.

What I arrived at is a three-layer system, with each layer covering its own part of the problem. No single magic instruction did the job.

Layer 1: automatic temporal context injection (Claude Code)

The first layer is a SessionStart hook, a bash script that runs every time a Claude Code session starts. It gets the current date, computes the gap from the training data cutoff date, and injects the result into the context:

CURRENT_DATE=$(date "+%Y-%m-%d")
TRAINING_CUTOFF="2025-01-01"
# ... calculating the difference ...

cat << EOF
Current date: ${CURRENT_DATETIME}
Your knowledge cutoff: approximately early 2025
Time gap: ~${MONTHS_GAP} months of potential changes

HIGH-RISK LIBRARIES (frequent breaking changes):
Next.js, React, LangChain, OpenAI SDK, Anthropic SDK,
Pydantic, FastAPI, Prisma

CRITICAL EPISTEMIC WARNING:
Your training data is ${MONTHS_GAP}+ months old. During this period:
- Libraries have released new major versions
- APIs have changed, been deprecated, or added new features
- Best practices may have evolved
- New tools and frameworks may have emerged

MANDATORY VERIFICATION PROTOCOL:
1. For ANY question about versions, APIs, current state -> SEARCH FIRST
2. Never claim "this doesn't exist" without verification
3. If code looks unfamiliar -> assume it's valid modern syntax
4. Mark uncertain claims: "Based on training (may be outdated)..."

EOF

The full script I use is here: inject-temporal-context.sh.

Why count months when you could just write “data is outdated”? Because “your data is outdated by 14 months” is a more specific signal, and the model responds to specifics better than to abstract warnings. The list of high-risk libraries also works as an explicit trigger: if you see LangChain in the project code, search the documentation instead of guessing.

Newer Claude versions have fresher knowledge, but not by much: the training cutoff still trails the release by months. Even then, nothing guarantees that knowledge in the domain you need is current right up to that date.

Layer 2: verification rules in CLAUDE.md (Claude Code)

The hook provides context but doesn’t define behavior. An epistemic rule does that. It lives in the project’s CLAUDE.md, gets included via @epistemic.md, or sits as a separate file in rules/.

The core of it is a hierarchy of truth sources, not a list of rules:

The response format matters too. The model has to label the source of each claim:

The labels aren’t decoration. They change how the model behaves: when it has to label the source, it hallucinates less, because labeling forces it to “think” about where the information came from.

The “Anti-Hallucination” block, with its specific prohibitions, matters on its own:

Then there’s explicit permission to say “I don’t know.” It sounds strange, but the model needs that permission. By default LLMs are optimized for helpfulness and would rather give a wrong answer than say “I’m not sure.” An explicit “Admitting uncertainty is BETTER than confident hallucination” shifts that default slightly.

Here is my file, almost in its entirety:

## Epistemic Hierarchy (Priority of Truth Sources)

1. **Project Files & User Context** (HIGHEST): package.json, requirements.txt, go.mod are authoritative. User-provided facts = ground truth. Unknown feature/API = assume NEW, not error.
2. **External Tools & Documentation**: Web search, fetched docs, MCP responses override training data.
3. **Your Training Data** (LOWEST): "Legacy Archive" -- reliable for syntax/logic, unreliable for versions/APIs/events.

## Mandatory Verification

ALWAYS verify before answering about:
- Library/framework versions or release dates
- LLM versions, names and parameters
- API signatures, method parameters, return types
- Deprecated vs current approaches
- "Does X exist?" / "Is Y still supported?"
- Any fact that could have changed since training

## Response Format

When providing technical information:
- `VERIFIED (from [source]): [info]` -- confirmed via search/docs
- `FROM TRAINING (may be outdated): [info]` -- unverified
- `UNCERTAIN: [info] -- recommend verification` -- low confidence

When diagnosing errors or unexpected behavior:
- `VERIFIED: [fact]` -- read from code, logs, config, or stated by user
- `HYPOTHESIS: [assumption]` -- inference that needs confirmation, always mark as such
- Present multiple hypotheses with equal weight when uncertain

## Anti-Hallucination

Do NOT:
- "Correct" user code to older syntax you're familiar with
- Claim "this doesn't exist" without verification
- Silently downgrade modern patterns to legacy equivalents
- State version numbers from memory as facts

Instead:
- Unfamiliar code -> assume valid modern syntax
- Uncertain existence -> "let me check" or ask user
- Suggesting alternatives -> explain WHY, confirm user's version first
- Stating versions -> mark as "from training, verify current"

## Version Handling

1. Check project files (package.json, etc.) FIRST
2. Version specified -> use THAT version's API
3. No version info -> ASK user
4. User states version -> trust it, even if unfamiliar

## Permission to Say "I Don't Know"

You are explicitly encouraged to say:
- "I'm not certain about the current API -- let me check"
- "This might have changed since my training"
- "I don't recognize this, but assuming it's valid modern syntax"

Admitting uncertainty is BETTER than confident hallucination.

Layer 3: user preferences in claude.ai

For the claude.ai web interface, I set the same principles in user preferences. It’s the same Knowledge Freshness Protocol adapted to the preferences format: source priorities, mandatory search triggers, the labeling format and prohibited actions.

There are no hooks or project files here, but there is web search. The rule “ALWAYS use web search before answering about: library versions, API signatures, deprecations” makes the model search instead of guessing.

What this achieves in practice

Since I set this up, “silent” errors, where the model confidently writes code for the wrong version, have dropped sharply. Not to zero: the model still sometimes ignores instructions, especially after compact. But its behavior changed. Instead of “here’s your code,” it more often says “let me check the current API version” and goes to the documentation.

Source labels also turned out to be useful beyond honesty, as a quick way to judge an answer. If you see nothing but FROM TRAINING labels, double-check everything. If you see VERIFIED labels, you can trust it.

Limitations

This is not a silver bullet. LLMs are non-deterministic, so there’s always some chance the model ignores even CLAUDE.md. A few specific cases:

After compact, the context is compressed and the rules lose priority. The SessionStart hook helps (it’s at the beginning of the session), but epistemic rules from CLAUDE.md may weaken in long sessions. I rarely use compact, though, and with a 1M context window the chance of forgetting is lower, but not zero.

The model can “bypass” the rules if a question seems too simple. “What’s the latest React release?”: it will search. “Write a useEffect”: it might not check whether the API has changed.

Web search doesn’t always return relevant results. Sometimes the model searches, finds nothing, and falls back to training data without the FROM TRAINING label. I have a separate story about web search and fetch, but that’s a topic for another time.

Reproducible configuration

The whole setup is three files:

  1. ~/.claude/hooks/inject-temporal-context.sh: a hook for Claude Code that calculates the date and injects context (download)
  2. epistemic.md: verification rules, included in CLAUDE.md via @epistemic.md or placed separately in ~/.claude/rules/ (the full text is in Layer 2 above)
  3. The Knowledge Freshness Protocol text for user preferences in claude.ai

The hook is registered in ~/.claude/settings.json:

{
  "hooks": {
    "SessionStart": [{
      "hooks": [{
        "type": "command",
        "command": "$HOME/.claude/hooks/inject-temporal-context.sh"
      }]
    }]
  }
}

Adapt TRAINING_CUTOFF and FAST_MOVING_LIBS to your stack. The list of high-risk libraries should be your own; there’s no universal one. If you write in Rust and Go, the problem packages are different.