Degrade retrieval to test CI gate - #8
NaveenBuidl wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR intentionally degrades retrieval configuration to validate that the evaluation CI gate fails when retrieval quality drops or retrieval becomes invalid.
Changes:
- Comments out the prior
retrieval_k: 4value. - Sets
retrieval_kto0inconfig.yaml.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # retrieval_k: 4 | ||
| retrieval_k: 0 |
There was a problem hiding this comment.
Setting retrieval_k: 0 is likely to make retrieval invalid: app/rag.py passes this directly as n_results to Chroma’s collection.query, which commonly requires n_results >= 1 and may raise at runtime (failing CI for a crash rather than an eval-quality regression). If the goal is to degrade quality while keeping the pipeline valid, set this to 1 instead; if the goal is to test invalid config handling, consider adding explicit validation (fail fast with a clear error) rather than relying on downstream library behavior.
Intentional degradation PR to verify the eval CI gate blocks lower retrieval quality.
Changes:
Expected: