The RAG Reality Check

Ever asked your enterprise AI assistant a question only to receive a vague, irrelevant answer? You’re not
alone. While Retrieval-Augmented Generation (RAG) has revolutionised how AI systems access knowledge, the difference between a mediocre implementation and an exceptional one is night and day.

At Katonic AI, we’ve spent years refining our RAG capabilities to deliver enterprise-grade results. Today, I’m sharing six powerful improvements you can make to transform your RAG system from merely functional to genuinely impressive.

Why RAG Optimisation Matters for Your Business

Before diving into the technical improvements, let’s talk about why this matters. Poorly optimised RAG systems:

• Retrieve irrelevant information, wasting users’ time
• Miss critical context, leading to inaccurate responses
• Provide inconsistent experiences across different queries
• Struggle with complex document formats and structures

Each of these issues directly impacts user adoption, trust, and ultimately the ROI of your AI investment. The good news? Most RAG issues can be solved with the right configuration.

1. Configure Your System and Persona Prompts

Think of system prompts as the invisible instruction manual for your AI assistant. When properly configured, they establish:

• Core guidelines for AI behaviour
• Consistent response formats
• Appropriate guardrails for sensitive topics
• Stable baseline performance across conversations

On the Katonic Platform, you can easily configure system prompts by navigating to: ACE →Configuration → Prompt Personalisation → Search Knowledge Prompt → System Prompt

But system prompts are only half the story. Your AI’s personality—its tone, style, and communication approach—is defined by persona prompts. A well-crafted persona:

• Creates a consistent, recognisable voice that builds user trust
• Adapts to your target audience (professional, casual, educational)
• Enhances user engagement through relatable communication
• Can represent domain-specific expertise (legal, medical, technical)

To configure persona prompts in the Katonic Platform: ACE → Persona Management → Create new ‍persona

Once created, users can switch between personas using the dropdown at the top of the chat interface—perfect for different departments or use cases.

2. Optimise Chunk Size and Overlap

Have you ever noticed how some RAG systems nail specific factual questions but struggle with complex
topics? Or conversely, how they sometimes provide general context but miss the precise details you
need? That’s often down to chunk size configuration.

Chunk size refers to how your documents are divided for embedding and retrieval. The impact on accuracy is significant:

• Smaller chunks (100-500 tokens) provide precise retrieval for specific questions but may miss
  broader context
• Larger chunks (1000+ tokens) capture comprehensive context but can retrieve irrelevant
  information
• Optimal sizing matches your typical query complexity—shorter for factual queries, longer for
  complex reasoning
• Once satisfied, deploy the application

Just as important is chunk overlap—how much text is shared between adjacent chunks:

• Prevents context fragmentation by ensuring related information isn’t artificially separated
• Maintains semantic continuity across chunk boundaries
• Creates beneficial redundancy that increases the chance of retrieving relevant information

For most applications, a 10-20% overlap works well, but complex documents with context spanning multiple paragraphs may benefit from 20-50% overlap.

3. Fine-tune Your Number of Retrieved Chunks

This often-overlooked parameter controls how many chunks the system retrieves before generating a response:

• Too few chunks (1-3) might miss critical information
• Too many chunks (15+) introduce noise and irrelevant content
• The sweet spot is typically 8-10 chunks for balanced retrieval

To adjust this on the Katonic Platform: ACE → Configuration → Application Settings → Chat Accuracy‍Settings

One financial services client saw their RAG response accuracy jump from 67% to 89% simply by optimising this parameter based on their specific document types and query patterns.

4. Apply Vision Indexing for Complex Data

Standard text-based chunking works well for straightforward documents, but what about complex
structured files, tables, or diagrams? That’s where vision indexing comes in.

The Katonic Vision Reindex feature helps fetch more accurate details from complex structured files by
using AI vision capabilities to understand document layout and structure.

To apply vision indexing: ACE → Knowledge Management → Select knowledge → Knowledge Objects
tab → Preview button → Reindex Using Vision

We’ve seen this make a dramatic difference for clients with complex financial reports, legal documents,and technical manuals—information that would be lost in standard text chunking is properly preserved and made retrievable.

5. Leverage Metadata Filtering

Not all knowledge is created equal. Sometimes you need information from specific document types or
categories. Metadata filtering constrains retrieval to the most relevant sources.

Users can select document types directly in ACE Chat or chat with a specific document by typing “@” and
selecting the document name.

A telecommunications client used this feature to create separate knowledge bases for consumer
products, enterprise solutions, and internal policies. When answering customer queries, their support
teams could instantly filter to only the relevant document categories, dramatically improving response
accuracy.

6. Choose the Right Embedding Model

The embedding model you select fundamentally impacts how well your system understands and retrieves information:

• Higher-dimensional models often capture semantic relationships more effectively
• Domain-specific embeddings trained on relevant data can dramatically improve performance
• Multilingual models provide better results for international content

To reset your embedding model: ACE → Configuration → Application Settings → Model‍Configuration → Reset Embedding Model

Don’t underestimate the impact of the right embedding model.One healthcare client switched from a general embedding model to a domain-specific one and saw a 43% improvement in retrieval precision for medical terminology.

Bonus Tip: Query Rephrasing

Even with all these optimisations, sometimes users don’t ask questions in the most effective way. Query
rephrasing automatically reformulates questions to better match how information is stored.

Users can leverage this on the Katonic Platform by typing their original question and pressing ALT + L
from the keyboard in ACE chat.

The Business Impact of RAG Optimisation

These improvements aren’t just technical tweaks—they deliver measurable business value:

• Reduced support costs: One client reduced tier 2 support escalations by 37% after RAG optimisation
• Higher user satisfaction: Average satisfaction scores increased from 3.6/5 to 4.7/5 for another
  customer
• Faster time-to-information: Average time to find critical information dropped from 4.2 minutes to
  under 30 seconds
• Increased AI adoption: System usage increased by 215% after RAG improvements made responses
  noticeably more relevant

Getting Started with RAG Improvements

The best part about these RAG improvements? They don’t require data science expertise to implement.
The Katonic AI Platform provides intuitive interfaces to make these adjustments with just a few clicks.

Whether you’re just starting your RAG journey or looking to optimise an existing implementation,
focusing on these six areas will yield significant improvements in accuracy, relevance, and user
satisfaction.

‍

‍