perf(quant): bypass pcre.match for exact dynamic config patterns - #2987
Conversation
_resolve_dynamic_override now short-circuits anchored literal patterns with an O(1) exact lookup, avoiding one pcre.match per pattern per module during model loading. True regex patterns still compile and match in original order; mixed exact+regex configs preserve first-match precedence. Add tests covering fully-exact, negative-exact, mixed, prefix, and large-scale exact configs.
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| cache_key = id(dynamic) | ||
| override_cache = _DYNAMIC_OVERRIDE_CACHE.setdefault(cache_key, {}) | ||
| cached = override_cache.get(module_name, _DYNAMIC_NO_MATCH) | ||
| if cached is not _DYNAMIC_NO_MATCH: | ||
| return cached |
There was a problem hiding this comment.
🔴 Per-module quantization overrides added at runtime are silently ignored, giving some layers the wrong precision
Per-module quantization settings are cached the first time a module is looked up (_resolve_dynamic_override at gptqmodel/quantization/config.py:1616-1620) and are never rebuilt when the settings dictionary is modified afterwards, so rules added later return the previously cached answer instead of the new one.
Impact: The lm_head layer (and MoE expert modules excluded mid-run) end up quantized with the base/default settings instead of their intended overrides, silently producing an incorrectly quantized model.
Stale id()-keyed cache vs. post-construction mutation of the dynamic dict
The new caches (_DYNAMIC_PATTERN_CACHE, _DYNAMIC_OVERRIDE_CACHE, _DYNAMIC_EXACT_LOOKUP_CACHE, etc.) are keyed by id(dynamic) and are populated on the first lookup, then never invalidated. The comment at gptqmodel/quantization/config.py:1525-1527 asserts the dynamic dict is "immutable after config construction", but that assumption is violated:
gptqmodel/looper/module_looper.py:1433-1434:dynamic_get(lm_head, default=None)is called first (cachingoverride_cache[id(dynamic)][lm_head] = Noneand the compiled pattern set), and only then isdynamic[lm_head] = lm_head_quant_configinserted. Every subsequentdynamic_get(lm_head, "bits", ...)during quantization hits the stale cachedNone(and the stale pattern set that never compiled the new key), so the{bits:8, group_size:32, ...}override is ignored and lm_head is quantized with base config.gptqmodel/looper/stage_subset.py:949-953: negative exclusions-:<escaped_name>are appended toqcfg.dynamicafter the dict has already been used (and cached) during the quant loop; those exclusions are not reflected in laterdynamic_get(...) is Falsechecks.
Because id() values are also reused after garbage collection and the caches keep no reference to the dynamic dict, two different configs whose dynamic dicts happen to share an id could also cross-contaminate. The previous implementation re-iterated dynamic.items() on every call, so it always reflected the current dict contents.
Prompt for agents
The dynamic override resolution now caches compiled patterns and resolved overrides in module-level dicts keyed by id(dynamic) (config.py: _DYNAMIC_PATTERN_CACHE, _DYNAMIC_OVERRIDE_CACHE, _DYNAMIC_EXACT_LOOKUP_CACHE, _DYNAMIC_ALL_EXACT_CACHE, _DYNAMIC_REGEX_PATTERN_CACHE). The design comment assumes the dynamic dict is immutable after construction, but it is mutated at runtime: gptqmodel/looper/module_looper.py:1434 inserts an lm_head rule after calling dynamic_get(lm_head) first, and gptqmodel/looper/stage_subset.py:951 appends negative exclusions after the dict has already been used. Because the id()-keyed caches are never invalidated, these post-hoc rule additions are silently ignored, so lm_head is quantized with base settings instead of the intended {bits:8,...} override, and pruned MoE modules are not treated as excluded. Additionally, id() can be reused after GC (the caches hold no reference to the dynamic dict), risking cross-config contamination. Fix approaches: (a) invalidate/rebuild the caches whenever the dynamic dict changes (e.g. key by a version counter or by a hash/snapshot of the dict, or attach the cache to the QuantizeConfig instance and clear it on mutation), or (b) make the mutation sites clear the relevant cache entries after mutating, or (c) avoid caching resolved overrides for negative/None results that can later be filled in. Ensure the lm_head insertion path and the MoE exclusion path both see fresh results after mutation.
Was this helpful? React with 👍 or 👎 to provide feedback.
| _DYNAMIC_PATTERN_CACHE: Dict[int, List[Tuple[bool, Any, Dict[str, Any], Optional[str], int]]] = {} | ||
| _DYNAMIC_OVERRIDE_CACHE: Dict[int, Dict[str, Any]] = {} | ||
| # Exact-literal fast-path caches: a per-dynamic dict mapping literal module names | ||
| # to their resolved override, and whether every pattern is an exact literal. | ||
| _DYNAMIC_EXACT_LOOKUP_CACHE: Dict[int, Dict[str, Tuple[int, Union[Dict[str, Any], bool]]]] = {} | ||
| _DYNAMIC_ALL_EXACT_CACHE: Dict[int, bool] = {} | ||
| # Pre-separated regex patterns for mixed dynamic configs. | ||
| _DYNAMIC_REGEX_PATTERN_CACHE: Dict[int, List[Tuple[int, bool, Any, Dict[str, Any]]]] = {} |
There was a problem hiding this comment.
🔍 Global dynamic caches grow unbounded across many configs
The five module-level cache dicts (_DYNAMIC_PATTERN_CACHE etc., gptqmodel/quantization/config.py:1528-1535) are keyed by id(dynamic) and never evicted. In a long-running process that constructs many QuantizeConfig objects (e.g. serving/loading many models, or test suites), these caches accumulate entries indefinitely — a slow memory leak. The tests clear them manually via _clear_dynamic_caches, but production code has no eviction path.
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
Summary
dynamic_getcompiled and matched every PCRE pattern for every quantized module during model loading. When aquantize_config.jsoncontains many anchored literal patterns (e.g.+:^model\.layers\.0\.mlp\.down_proj$), this results in millions of unnecessarypcre.Pattern.matchcalls and dominatedGPTQModel.loadtime.What Changed
gptqmodel/quantization/config.py_extract_literal_regex_pattern()to detect regexes that are actually anchored literal strings._get_dynamic_patterns()to compile patterns once and separate exact literals from true regexes._resolve_dynamic_override()that resolves exact literals in O(1) via a dict lookup, falling back to ordered regex matching only for genuine regex patterns.dynamic_get()to use_resolve_dynamic_override()while preserving existing key/sub-key/synonym handling.tests/qcfg/test_dynamic_config.pypcre.matchcalls.Tests
cd /root/repos/GPTQModel /root/vm314t/bin/python -m pytest tests/qcfg/test_dynamic_config.py -qResult:
6 passed, 16 warnings in 7.57sNotes
The fix preserves first-match precedence for mixed configs and only treats patterns as exact literals when they have both leading
^and trailing$anchors and no unescaped regex metacharacters.