Deploying AI Agents: Local vs Prod - like launching a rocket

I scrapped 5 agents before ironing out all mistakes that gave median Eval score of 0.3

Top mistakes deploying AI agents

Mistake	Est. %	Popularity	Significance
Tuning Agent against an uncalibrated judge	~80%	🥇	🥈
No frozen goldens - "looks better" as metric	~60%	🥈	🥇
Wiring all 6 MCPs before first Agent run	~35%	🥉	🥉

Local agent needs fast iteration - few frozen golden cases to verify against captcha style, fast reruns, print statements are fine, what info is missing, fine to blowup context, judge calibration.

Prod agent is like a rocket, might be easy to deploy fix, but blowout might be huge, so at this stage no agent tuning only safety - provenance, proper tracing, long running limits (timeouts, ram, context summary), tools failsafe, evals

Golden - set of CORRECT questions and answers to check agent output against. Judge Calibration - process of tuning LLM-as-a-Judge on Goldens up to statistical goal

Local dev checklist - applied scientist’s job

1. 1 working mcp connection - add more fast on 6.
2. Agent and Judge output shapes
3. Initial Agent run - only one
4. Few manual frozen goldens
5. Judge calibration
6. Agent’s prompts/mcp/rag calibration

# 1. working mcp connection
gh = await mcp.connect("github")
await gh.call("search_repos", {"q": "user:mikkicon"})

# 2. output shapes - agent / judge
class Answer(BaseModel): text: str; sources: list[str]
class Verdict(BaseModel): ok: bool; why: str

# 3. initial agent run
print(await run_agent("Is this inbox message worthy? \n {message}"))

# 4. few manual frozen goldens
goldens = [{"q": "Apple - AI Engineer", "must_cite": "job.md", "label": True}]

# 5. judge calibration
assert all(judge(g["answer"]).ok == g["label"] for g in goldens)

# 6. prompts / mcp / rag calibration
score = sum(judge(await run_agent(g["q"])).ok for g in goldens) / len(goldens)

When your Agent hits 95% confidence within clopper-pearson (LINK TO NEXT POST ON EVAL) - you can move to Prod stage

Production checklist - engineer’s job

1. *Statistically grounded goldens
2. Proper Tracing & tool guards
3. Context management - memory offload, summarization
4. Infra - provenance, timeouts, RAM, ...
5. Cron eval smoke

# 1. statistically grounded goldens
assert len(goldens) >= 200
assert wilson_lower(score, len(goldens)) > 0.9

# 2. proper tracing & tool guards
with langfuse.trace(user_id=uid):
    out = await guarded(tool, args, retries=2, allow=["read"])

# 3. context mgmt - mem offload, sum
if tokens(msgs) > 120_000:
    msgs = [summarize(msgs[:-10]), *msgs[-10:]]

# 4. infra - timeouts, ram
await asyncio.wait_for(run_agent(q), timeout=60)  # lambda: 1024mb, 5 concurrent

# 5. cron eval smoke
@cron("0 * * * *")
def smoke(): alert_if(eval(goldens[:20]) < baseline - 0.05)

My minimal AI agent stack on mikenova.ai

	I use	Better
LLM	OpenRouter / Local via Ray-Serve + vLLM	Direct + reasoning Anth/OpenAI/Gemini/Local
Backend	NextJS Lambda	Service with WebSockets
RAG	Static Json - 181 chunks × 384 dims = 2MB $0/mo	Pinecone/MongoDB Vector DB
Memory	MongoDB	Cassandra
MCP - Web	Tavily MCP for web search	-
MCP - GitHub	GitHub MCP for my GitHub projects search	-
Tracing + Logs	MongoDB	Langfuse

Start with simple agent - e.g. https://github.com/Mikkicon/agent-secops/blob/main/agent/main.py