AI Systems Engineering Problem

Why My “Smart” AI Chatbot Kept Making Things Up (And What Actually Fixed It)

Three weeks after we launched our internal support chatbot, our head of customer success walked into my desk area holding her laptop like it had personally offended her.

“It told a customer we offer refunds after 90 days,” she said. “We don’t. We never have. Where did it get that?”

I didn’t have an answer. Not a good one, anyway. And that’s the moment I really understood what a lot of AI engineering guides don’t tell you: getting a chatbot to sound confident is easy. Getting it to actually be right, consistently, on real company data, is a completely different problem.

If you’ve built anything with retrieval-augmented generation (RAG) — hooking an LLM up to your own documents so it can answer questions about your product, your policies, your codebase, whatever — you’ve probably hit some version of this. Let me walk you through what actually went wrong for us, and what fixed it, because most of the advice I found online was either too theoretical or assumed I had a data science PhD.

The setup (and why it seemed fine at first)

We were using a pretty standard stack: documents chunked and embedded, stored in a vector database (we used Pinecone, though Chroma and Weaviate would’ve worked the same way), and a retrieval step that pulled the “most relevant” chunks before sending them to the model along with the user’s question.

In demos, it looked great. Ask it about our shipping policy, it nails it. Ask about pricing tiers, spot on. We shipped it.

Then real users started asking real questions — messy, oddly phrased, sometimes combining two topics in one sentence — and things started falling apart quietly. Not with error messages. With confident, well-written, wrong answers. That’s the scary part. A broken script throws an error. A broken RAG pipeline just lies to you politely.

Mistake #1: I trusted chunk size defaults

When I first set up our document pipeline, I used the default chunking size from a tutorial — 512 tokens with some overlap — and didn’t think about it again. AI Systems Engineering Problem

Turns out that our refund policy document had the actual refund window mentioned in one paragraph, and the exceptions to that window in the next paragraph. Because of how the chunks split, the model was retrieving the exceptions paragraph without the base policy paragraph. It had half the story and confidently filled in the rest.

What fixed it: I went back and chunked by logical section (using markdown headers as split points) instead of a fixed token count. It’s more manual work upfront, but for policy-heavy or structured documents, it made a night-and-day difference. For narrative or conversational content, token-based chunking is still fine — just don’t assume one chunking strategy fits every document type.

Mistake #2: I never tested retrieval separately from generation

This one embarrassed me a little. For weeks, whenever the bot gave a wrong answer, I assumed the model was hallucinating.

Eventually I logged what chunks were actually being retrieved for each bad answer, and realized something obvious in hindsight: half the time, the retrieval step was pulling irrelevant chunks. The model wasn’t hallucinating. It was doing exactly what you’d expect a smart model to do — trying to construct a plausible answer from garbage context.

Lesson learned: Debug retrieval and generation as two separate problems. Before touching your prompt, check what documents actually got pulled for a failing query. Tools like LangSmith or even just a simple logging script that prints the top-k retrieved chunks will save you hours of prompt-tweaking that was never going to fix the real issue.

Mistake #3: Cosine similarity isn’t the same as “actually relevant”

This one took me the longest to internalize. Vector search finds chunks that are semantically similar to the question. That’s not the same as finding the chunk that answers the question. AI Systems Engineering Problem

Example: someone asked “can I get a refund if the item arrived damaged?” Our vector search kept surfacing a generic “returns and exchanges” FAQ instead of the specific damaged-item policy, because the damaged-item doc used different wording and scored slightly lower on similarity, even though it was the actually correct answer.

What helped:

• Adding a re-ranking step after initial retrieval (Cohere’s rerank API is easy to bolt on and made a noticeable difference for us)
• Writing document titles and summaries that use the same phrasing customers actually use, not internal jargon
• Increasing the number of retrieved chunks from 3 to 6, then letting the model pick, instead of assuming the top 3 were automatically the best 3

Mistake #4: No fallback for “I don’t know”

Early on, the system prompt basically said “answer the user’s question using the context provided.” It never said what to do if the context didn’t actually contain the answer. So the model did what LLMs do when you don’t give them an out — it improvised.

The fix was almost embarrassingly simple: I added an explicit instruction telling the model that if the retrieved context doesn’t clearly answer the question, it should say so and offer to connect the user to a human, instead of guessing. Wrong answers dropped noticeably after this one change. It’s a small prompt addition with a big real-world impact, because it gives the model permission to admit uncertainty instead of always sounding certain.

Step-by-step: how I’d approach this if starting over

1. Chunk by structure, not just token count, especially for policy docs, contracts, or anything where meaning depends on surrounding context.
2. Log every retrieval, not just final answers. Store the query, the retrieved chunks, and the response together so you can actually debug patterns later.
3. Add a re-ranking step. Initial vector search casts a wide net; re-ranking narrows it to what’s actually relevant.
4. Test with real, messy questions, not the clean ones you’d write yourself. Ask your support team or a few real users to try to “break” it before launch.
5. Give the model explicit permission to say “I’m not sure.” This alone prevents a huge chunk of confident-but-wrong answers.
6. Set up an evaluation set. Even 30–50 real question/answer pairs that you check against periodically will catch regressions when you update your docs or swap models.
7. Monitor in production. We eventually added a simple thumbs up/down on each answer, which gave us a steady stream of real failure cases instead of relying on someone walking over to complain. AI Systems Engineering Problem

A mistake I still see people make

I’ve talked to a few other engineers going through similar builds, and the most common trap isn’t technical at all — it’s assuming that because the LLM is powerful, the system around it doesn’t need much engineering discipline. It does. Arguably more, because failures are silent and confident-sounding instead of loud and obvious.

Retrieval quality, chunking strategy, evaluation, and honest fallback behavior are where the real engineering work lives. AI Systems Engineering Problem

Where things stand now

Our chatbot still isn’t perfect — nothing built on top of an LLM is going to be 100% right 100% of the time, and I’ve made peace with that. But wrong answers dropped by more than half after these changes, and more importantly, when it doesn’t know something, it says so instead of making it up.

If you’re building something similar, don’t just tune your prompts when things go wrong. Check what your retrieval step is actually feeding the model first.

Click For More: