Skip to content

Degrade retrieval to test CI gate - #8

Closed
NaveenBuidl wants to merge 2 commits into
mainfrom
degrade-retrieval
Closed

NaveenBuidl wants to merge 2 commits into
mainfrom
degrade-retrieval

Conversation

@NaveenBuidl

Copy link
Copy Markdown
Owner

Intentional degradation PR to verify the eval CI gate blocks lower retrieval quality.

Changes:

  • retrieval_k: 4 -> 1
  • retrieval_k: 1 -> 0

Expected:

  • Eval Gate should fail if retrieval quality drops or retrieval becomes invalid.

Copilot AI review requested due to automatic review settings April 9, 2026 08:41

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR intentionally degrades retrieval configuration to validate that the evaluation CI gate fails when retrieval quality drops or retrieval becomes invalid.

Changes:

  • Comments out the prior retrieval_k: 4 value.
  • Sets retrieval_k to 0 in config.yaml.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread config.yaml
Comment on lines +4 to +5
# retrieval_k: 4
retrieval_k: 0

Copilot AI Apr 9, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting retrieval_k: 0 is likely to make retrieval invalid: app/rag.py passes this directly as n_results to Chroma’s collection.query, which commonly requires n_results >= 1 and may raise at runtime (failing CI for a crash rather than an eval-quality regression). If the goal is to degrade quality while keeping the pipeline valid, set this to 1 instead; if the goal is to test invalid config handling, consider adding explicit validation (fail fast with a clear error) rather than relying on downstream library behavior.

Copilot uses AI. Check for mistakes.
@NaveenBuidl NaveenBuidl closed this Apr 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants