fix(langchain): stop emitting space-separated provider names that price at $0 - #666
Conversation
…ce at $0
Cost is resolved by lowercasing the step's `provider` and matching an
llm-costs slug exactly. Matching normalizes case but NOT separators, so a
space makes the value unmatchable and the row silently prices at zero.
Verified against the live data-stream API with identical model and token
counts:
provider="Together AI" -> cost 0.0
provider="Together_AI" -> cost 0.0065
provider="together_ai" -> cost 0.0065
Two causes, both fixed:
1. The unmapped-`ls_provider` fallback ran `.replace("_", " ").title()`,
converting values LangChain already supplies as valid slugs into
unpriceable display strings ("vercel_ai_gateway" -> "Vercel Ai Gateway").
Dropping the `.replace()` fixes every unmapped multi-token provider;
`.title()` is retained since matching is case-insensitive.
2. Six map values used literal spaces. "Together AI" -> "Together_AI" and
"Fireworks AI" -> "Fireworks_AI" in both LS_PROVIDER_TO_OPENLAYER_MAP and
LITELLM_PREFIX_TO_PROVIDER_MAP.
"Hugging Face" -> "HuggingFace" for consistency only: llm-costs publishes no
huggingface provider slug at all, so that provider remains unpriced either way.
Behavior change: `step.provider` for an unmapped `ls_provider` now keeps
underscores ("Some_New_Provider" rather than "Some New Provider"). Consumers
displaying this field verbatim will see the separator.
Adds TestProviderCostSlugs, which asserts no map value contains a space and
that known prefixes resolve to real slugs, with already-correct providers
(gemini, vertex_ai, anthropic, bedrock, deepseek) kept as regression canaries.
The space check catches the specific bug but not the general class: a value can be space-free and still not price anything. Port the invariant openlayer-ts asserts through PROVIDER_COST_SLUG -- every provider value must lowercase to a slug published by llm-costs, or be an explicitly listed display-only vendor. The slug set is vendored so the suite stays offline. huggingface is the sole unpriced entry: llm-costs publishes no hugging* provider at all. Verified red/green: against pristine main this fails with LS_PROVIDER_TO_OPENLAYER_MAP['fireworks'] = 'Fireworks AI' lowercases to 'fireworks ai', which is neither a published cost slug nor a known-unpriced vendor: every row would cost $0
|
Added a second commit that strengthens the guard, after comparing against
This PR now ports that invariant to Python. The original space check catches the specific bug but not the general class — a value can be space-free and still price nothing. Red/green verified — against pristine One deliberate divergence from TS: TS sidesteps Slug set is vendored rather than fetched so the suite stays offline; refresh with: |
Problem
Cost is resolved by lowercasing a step's
providerand matching an llm-costs slug exactly. Matching normalizes case but not separators, so any provider value containing a space is unmatchable and the row silently prices at $0 — with no error anywhere.Verified against the live data-stream API, identical model (
deepseek-ai/DeepSeek-R1) and token counts (1000/500):providersentTogether AI(what we emit today)Together_AItogether_aiTwo causes
1. The unmapped-
ls_providerfallback manufactured the bug. It ran.replace("_", " ").title(), converting values LangChain already supplies as valid slugs into unpriceable display strings —vercel_ai_gateway→Vercel Ai Gateway. This silently zeroed cost for every provider not explicitly enumerated inLS_PROVIDER_TO_OPENLAYER_MAP(30 of the 139 published slugs contain an underscore; hyphen-only slugs were unaffected because.replace()only touched_). Removing the.replace()fixes them all;.title()is kept because matching is case-insensitive.2. Six map values used literal spaces, in both
LS_PROVIDER_TO_OPENLAYER_MAPandLITELLM_PREFIX_TO_PROVIDER_MAP:Together AI→Together_AIFireworks AI→Fireworks_AIHugging Face→HuggingFaceLANGCHAIN_TO_OPENLAYER_PROVIDER_MAPwas already clean (all single-word) and is unchanged.Verification
Live round-trip driving the patched
_extract_model_infoand pricing whatever provider it emits:together_ai/deepseek-ai/DeepSeek-R1Together_AIfireworks_ai/accounts/fireworks/models/code-llama-13bFireworks_AIgemini/gemini-2.5-flash(canary)Googleanthropic/claude-3-5-sonnet-20241022(canary)Anthropictests/lib/integrations/test_langchain_callback.py: 59 passed (44 before + 15 new). Full suite shows 34 failures both before and after in deterministic serial order — no regressions; those are pre-existing (missing optional deps, plus order-dependent trace-context pollution intests/test_tracing_core.py).New
TestProviderCostSlugsasserts no map value contains a space and that known prefixes resolve to real slugs, keeping already-correct providers as regression canaries.step.providerfor an unmappedls_providernow retains underscores:Some_New_Providerrather thanSome New Provider. This also changes the derived step name ("Some_New_Provider Chat Completion").test_ls_provider_title_cases_unknown_valueswas updated accordingly — it previously asserted the buggy behavior. Anything displayingproviderverbatim will show the separator.Same class of bug as OPEN-11695; worth linking a Linear ticket before merge.