> For the complete documentation index, see [llms.txt](https://docs.karini.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.karini.ai/recipes/knowledgebase-recipe/retrieval-evaluation.md).

# Retrieval Evaluation

## Overview

**KB Retrieval Evaluation** is a quality assessment tool that measures how effectively a knowledge base recipe's vector database retrieves relevant documents for a given set of queries. It isolates the retrieval stage of the RAG (Retrieval-Augmented Generation) pipeline, enabling teams to diagnose and optimize document retrieval independently of LLM answer generation.

The system compares retrieved documents with a ground truth dataset and calculates standard retrieval metrics at a configurable **Top-K** depth, including:

* **Precision**
* **Recall**
* **MRR**
* **NDCG**
* **F1 Score**

#### Prerequisites

To run **Retrieval Evaluation**, the following prerequisites must be in place:

* A **knowledgebase recipe** with a dataset node connected to a vector database node.
* A **fully embedded dataset** with embedding processing completed.
* A **configured embeddings model** linked to the dataset-to-vector database edge.
* A **configured vector database node**, such as OpenSearch.
* An **evaluation dataset in CSV format** containing queries and their expected relevant documents.

#### Evaluation Dataset Format

The evaluation dataset is a CSV file that defines the queries and their expected relevant results.

#### Required Columns

| Column Name                | Required        | Description                                                                        |
| -------------------------- | --------------- | ---------------------------------------------------------------------------------- |
| `input`                    | **Yes**         | The search query to evaluate retrieval against                                     |
| `ground_truth_chunk_ids`   | **Conditional** | Comma-separated list of expected relevant document/chunk IDs from the vector index |
| `ground_truth_chunks_text` | **Conditional** | Text content of expected relevant chunks for text-based matching                   |

{% hint style="info" %}
**Note:** At least one of `ground_truth_chunk_ids` or `ground_truth_chunks_text` must be present in the CSV.
{% endhint %}

#### Example

```
input,ground_truth_chunk_ids"What is the refund policy?","chunk_001,chunk_002""How do I reset my password?","chunk_042"
```

#### Metrics and Top-K Configuration

Retrieval Evaluation supports multiple metrics. Each metric measures a specific part of retrieval quality.

| Metric           | What It Measures                                  | Interpretation                                                                                                        | When It Is Useful                                |
| ---------------- | ------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ |
| **Precision\@K** | How many retrieved chunks are relevant.           | A high score means most retrieved chunks are useful. A low score means too many unrelated chunks are returned.        | Use when irrelevant results should be reduced.   |
| **Recall\@K**    | How many expected relevant chunks were retrieved. | A high score means most expected chunks were found. A low score means important content is missing.                   | Use when completeness is important.              |
| **MRR**          | How early the first relevant result appears.      | A high score means the first useful chunk appears near the top. A low score means relevant chunks are ranked too low. | Use when the top result is most important.       |
| **NDCG\@K**      | Whether relevant chunks are ranked near the top.  | A high score means relevant chunks are ordered well. A low score means the best results are buried lower.             | Use when ranking order affects answer quality.   |
| **F1\@K**        | Balance between Precision and Recall.             | A high score means the system retrieves relevant chunks while also covering expected content.                         | Use when both relevance and completeness matter. |

#### Top-K Configuration

Top-K defines the number of chunks retrieved for each query. The default value is **5**.

| Top-K Range | Retrieval Behavior                                                                          | Recommended Use                                    |
| ----------- | ------------------------------------------------------------------------------------------- | -------------------------------------------------- |
| **1–3**     | Retrieves fewer chunks and usually improves Precision, but may miss some relevant content.  | Use when only the most relevant chunks are needed. |
| **4–7**     | Provides a balanced retrieval set for relevance and coverage.                               | Suitable for most Knowledgebase Q\&A use cases.    |
| **8–20**    | Retrieves more chunks and usually improves Recall, but may include more irrelevant content. | Use when broader coverage is required.             |

| Top-K Impact     | Explanation                                                                                                                                                 |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Lower Top-K**  | Usually improves Precision because fewer, more focused chunks are retrieved. However, Recall may decrease because some relevant chunks may not be included. |
| **Higher Top-K** | Usually improves Recall because more chunks are retrieved. However, Precision may decrease because additional chunks may include irrelevant content.        |

#### Steps to Follow

#### 1. Prepare Dataset

Create a CSV file with representative user queries and expected relevant chunks. Use real user questions wherever possible.

#### 2. Open Recipe Configuration

Open the Knowledgebase recipe and go to the **Recipe Configuration** tab. Locate the **Retrieval Evaluation** section.

<figure><img src="https://415930246-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F7ZrVuiAUMyuYVvrK5KaB%2Fuploads%2Fr8Is9E6MmkqlSqARvHjd%2Fimage.png?alt=media&amp;token=0a0c5013-8757-4e2b-a0ef-21e4599e0840" alt=""><figcaption></figcaption></figure>

#### 3. Upload Dataset

Upload the CSV file. The system validates the file and enables the evaluation options after successful validation.

<figure><img src="https://415930246-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F7ZrVuiAUMyuYVvrK5KaB%2Fuploads%2F911weKXm47QNc7sVgmRE%2Fimage.png?alt=media&amp;token=1135b3c7-259c-467f-bae6-304f1e5e68be" alt=""><figcaption></figcaption></figure>

#### 4. Select Metrics and Top-K

Choose one or more metrics and set the Top-K value.

#### 5. Run Evaluation

Click **Run evaluation**. The system creates an evaluation run and starts processing the queries.

#### 6. Monitor Status

The evaluation status changes as the run progresses.

| Status       | Meaning                                |
| ------------ | -------------------------------------- |
| `pending`    | Evaluation is created but not started. |
| `processing` | Evaluation is running.                 |
| `completed`  | Evaluation completed successfully.     |
| `error`      | Evaluation failed.                     |
| `canceled`   | Evaluation was stopped.                |

#### 7. View Results

Once completed, open the results page to view summary scores and detailed query-level results.

### Results

The results page shows overall performance and detailed evaluation records.

\ <br>