Preventing LLM Hallucinations in Multi-Agent Workflows

The Prompt Engineering Trap

When developers build multi-agent systems, their first instinct is to try to prevent hallucinations using prompt engineering. They write massive system prompts: "You are a helpful assistant. NEVER output negative numbers. ALWAYS output JSON."

This works in demos, but fails catastrophically in production. Large Language Models (LLMs) like GPT-4, Claude, or Llama are inherently probabilistic. They operate by predicting the next token based on a vast, fuzzy latent space. They do not have a deterministic runtime. They will, eventually, ignore your prompt.

If you rely exclusively on prompt engineering to secure your application state, you are building on sand. A single hallucinatory output—such as assigning a negative discount, booking a hotel for zero days, or attempting to transfer funds to an invalid UUID—can crash your orchestrator, or worse, silently corrupt your database.

Types for your Agent's Memory

Instead of hoping the LLM behaves, you must guarantee that the system rejects bad behavior. In traditional software engineering, we achieve this using Types and Invariants. We need the exact same concept for AI memory orchestration.

An Invariant is a strict mathematical or logical rule that must evaluate to True before state can be updated. If the invariant fails, the update must be blocked, and the system must remain exactly as it was before the update was attempted. This is the definition of a transactional boundary.

Common AI Hallucination Vectors

• Type Mismatches: The agent returns a string "150" instead of an integer 150.
• Boundary Violations: The agent calculates a refund amount greater than the original purchase price.
• State Transition Violations: The agent attempts to mark an order as shipped before it is marked as paid.

Enforcing Strict Invariants with StateGuard

StateGuard allows you to define these exact mathematical and logical rules for your Python-based agent workflows. It sits as a firewall between your agent's output and your application's state or database.

1. Basic Type and Boundary Invariants

Here is how you wrap a raw Python dictionary in a GuardedState. StateGuard will monitor every mutation to this dictionary.

from stateguard import GuardedState

# Define your invariants as simple lambda functions or standard defs
def valid_status(state):
    return state["status"] in ["pending", "processing", "completed", "failed"]

# Wrap your raw Python dictionary in a GuardedState
memory = GuardedState(
    initial_state={
        "status": "pending", 
        "retries": 0,
        "refund_amount": 0.0
    },
    invariants=[
        lambda state: state["retries"] <= 3,
        lambda state: state["refund_amount"] >= 0,
        valid_status
    ]
)

If an agent hallucinates and attempts to set the status to completed_but_weird, StateGuard will immediately raise a ValueError and reject the memory mutation. The hallucination is contained, and your application remains stable.

2. Complex State Transition Invariants

Often, a single value is valid, but the transition from the old state to the new state is invalid. StateGuard supports transition invariants that evaluate the delta.

# Ensure an agent cannot skip steps in a checkout process
def valid_checkout_flow(old_state, new_state):
    flow_order = {"cart": 1, "payment": 2, "shipping": 3, "completed": 4}
    old_step = flow_order.get(old_state["step"], 0)
    new_step = flow_order.get(new_state["step"], 0)
    
    # Only allow moving forward exactly one step, or falling back to cart
    return new_step == old_step + 1 or new_step == 1

memory.add_transition_invariant(valid_checkout_flow)

Handling the Rejection

What happens when an agent violates an invariant? StateGuard doesn't just crash. It provides a transactional context manager that allows your orchestrator to gracefully handle the failure and retry.

try:
    with memory.transaction() as state:
        # LLM generated output applied here
        state['refund_amount'] = -50.0  # Agent hallucinates a negative refund
except ValueError as e:
    # StateGuard intercepted the invalid write.
    # The 'memory' object is instantly rolled back to its previous pristine state.
    
    # 1. Log the hallucination
    logger.warning(f"Agent hallucination blocked: {e}")
    
    # 2. Feed the error back to the LLM for correction
    agent.invoke({
        "role": "user", 
        "content": f"Your last action was invalid: {e}. Try again."
    })

Conclusion

Stop trying to prompt your way out of hallucinations. Treat your AI agents as untrusted external APIs. By putting StateGuard between your agent and your database, you guarantee that no matter what text the LLM generates, your application state remains logically sound.