How I Test a Grounded Assistant Before It Talks to a Recruiter
Lexical retrieval, refusal rules, and a 270+ question eval set on nicoloperena.com. The public kit you can clone and run.
Most AI demos look smart until you ask a question the documents cannot support. They invent a salary, follow a prompt injection, or quote a stack you never shipped.
I do not ship that. The recruiting assistant on this site answers from retrieved pages, refuses what is not in those pages, and gets scored against a fixed question file. The production set is 270+ questions. A smaller public version lives in grounded-assistant-kit.
This is the loop, not a model brand.
Retrieve first, then talk
The production sales assistant on Nexrena is Gemini-backed chat with lexical retrieval, guardrails, lead scoring, and Postgres session analytics. The ops copilot uses OpenAI tool-calling over live CRM and inbox data, with scoped writes and a confirmation step before mutations.
The hiring-site assistant is the same idea with a tighter job: stay inside the portfolio, resume, and case studies. If retrieval returns nothing useful, the answer is a refusal, not a guess.
Lexical overlap is enough for this corpus. I do not need a vector database to rank a few dozen pages I wrote. Embeddings become worth the ops cost when the corpus is large, messy, or owned by other people. This one is not.
Guardrails are boring on purpose
Two classes of questions should never get a generated answer:
- Injection (“ignore the documents”, “you are now”).
- Off-policy (salary, secrets, anything the pages do not contain).
The public kit encodes that as tests. A case can require a refusal. A case can require a substring from a retrieved chunk. A case can forbid a word that would mean the model left the documents.
If those tests fail, CI fails. That is the whole point of npm run eval.
What 270+ questions are actually for
The large set is not for a leaderboard. It is so a change to retrieval, chunking, or copy cannot silently break the story I tell recruiters. Titles, dates, stack, and outcomes have to stay consistent with the pages on this site.
The kit is the readable version of that contract: a JSON corpus, a JSON question file, and a TypeScript runner. Clone it. Run it. Do not take my word for the loop.
Why this is the rare part
Plenty of engineers can wrap a chat API. Fewer will fail a deploy because a grounded answer drifted. That is the difference I want a hiring manager to see.
Related: Nexrena platform case study · public architecture repo