Skip to content

fix(langchain): stop emitting space-separated provider names that price at $0 - #666

Merged
viniciusdsmello merged 2 commits into
mainfrom
fix/provider-cost-slugs
Aug 20, 2026
Merged

fix(langchain): stop emitting space-separated provider names that price at $0#666
viniciusdsmello merged 2 commits into
mainfrom
fix/provider-cost-slugs

Conversation

@viniciusdsmello

@viniciusdsmello viniciusdsmello commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Problem

Cost is resolved by lowercasing a step's provider and matching an llm-costs slug exactly. Matching normalizes case but not separators, so any provider value containing a space is unmatchable and the row silently prices at $0 — with no error anywhere.

Verified against the live data-stream API, identical model (deepseek-ai/DeepSeek-R1) and token counts (1000/500):

provider sent resulting cost
Together AI (what we emit today) 0.0
Together_AI 0.0065
together_ai 0.0065

Two causes

1. The unmapped-ls_provider fallback manufactured the bug. It ran .replace("_", " ").title(), converting values LangChain already supplies as valid slugs into unpriceable display strings — vercel_ai_gatewayVercel Ai Gateway. This silently zeroed cost for every provider not explicitly enumerated in LS_PROVIDER_TO_OPENLAYER_MAP (30 of the 139 published slugs contain an underscore; hyphen-only slugs were unaffected because .replace() only touched _). Removing the .replace() fixes them all; .title() is kept because matching is case-insensitive.

2. Six map values used literal spaces, in both LS_PROVIDER_TO_OPENLAYER_MAP and LITELLM_PREFIX_TO_PROVIDER_MAP:

  • Together AITogether_AI
  • Fireworks AIFireworks_AI
  • Hugging FaceHuggingFace

LANGCHAIN_TO_OPENLAYER_PROVIDER_MAP was already clean (all single-word) and is unchanged.

Hugging Face is not actually fixed. llm-costs publishes no huggingface provider slug at all, so that provider stays unpriced regardless of the value used. Changed for consistency only.

Verification

Live round-trip driving the patched _extract_model_info and pricing whatever provider it emits:

case provider emitted live cost
together_ai/deepseek-ai/DeepSeek-R1 Together_AI 0.0065 (was $0)
fireworks_ai/accounts/fireworks/models/code-llama-13b Fireworks_AI 0.0003 (was $0)
gemini/gemini-2.5-flash (canary) Google 0.000186 (unchanged)
anthropic/claude-3-5-sonnet-20241022 (canary) Anthropic 0.0105 (unchanged)

tests/lib/integrations/test_langchain_callback.py: 59 passed (44 before + 15 new). Full suite shows 34 failures both before and after in deterministic serial order — no regressions; those are pre-existing (missing optional deps, plus order-dependent trace-context pollution in tests/test_tracing_core.py).

New TestProviderCostSlugs asserts no map value contains a space and that known prefixes resolve to real slugs, keeping already-correct providers as regression canaries.

⚠️ Behavior change

step.provider for an unmapped ls_provider now retains underscores: Some_New_Provider rather than Some New Provider. This also changes the derived step name ("Some_New_Provider Chat Completion"). test_ls_provider_title_cases_unknown_values was updated accordingly — it previously asserted the buggy behavior. Anything displaying provider verbatim will show the separator.

Same class of bug as OPEN-11695; worth linking a Linear ticket before merge.

…ce at $0

Cost is resolved by lowercasing the step's `provider` and matching an
llm-costs slug exactly. Matching normalizes case but NOT separators, so a
space makes the value unmatchable and the row silently prices at zero.
Verified against the live data-stream API with identical model and token
counts:

    provider="Together AI"  -> cost 0.0
    provider="Together_AI"  -> cost 0.0065
    provider="together_ai"  -> cost 0.0065

Two causes, both fixed:

1. The unmapped-`ls_provider` fallback ran `.replace("_", " ").title()`,
   converting values LangChain already supplies as valid slugs into
   unpriceable display strings ("vercel_ai_gateway" -> "Vercel Ai Gateway").
   Dropping the `.replace()` fixes every unmapped multi-token provider;
   `.title()` is retained since matching is case-insensitive.

2. Six map values used literal spaces. "Together AI" -> "Together_AI" and
   "Fireworks AI" -> "Fireworks_AI" in both LS_PROVIDER_TO_OPENLAYER_MAP and
   LITELLM_PREFIX_TO_PROVIDER_MAP.

"Hugging Face" -> "HuggingFace" for consistency only: llm-costs publishes no
huggingface provider slug at all, so that provider remains unpriced either way.

Behavior change: `step.provider` for an unmapped `ls_provider` now keeps
underscores ("Some_New_Provider" rather than "Some New Provider"). Consumers
displaying this field verbatim will see the separator.

Adds TestProviderCostSlugs, which asserts no map value contains a space and
that known prefixes resolve to real slugs, with already-correct providers
(gemini, vertex_ai, anthropic, bedrock, deepseek) kept as regression canaries.
The space check catches the specific bug but not the general class: a value can
be space-free and still not price anything. Port the invariant openlayer-ts
asserts through PROVIDER_COST_SLUG -- every provider value must lowercase to a
slug published by llm-costs, or be an explicitly listed display-only vendor.

The slug set is vendored so the suite stays offline. huggingface is the sole
unpriced entry: llm-costs publishes no hugging* provider at all.

Verified red/green: against pristine main this fails with
  LS_PROVIDER_TO_OPENLAYER_MAP['fireworks'] = 'Fireworks AI' lowercases to
  'fireworks ai', which is neither a published cost slug nor a known-unpriced
  vendor: every row would cost $0
@viniciusdsmello

Copy link
Copy Markdown
Contributor Author

Added a second commit that strengthens the guard, after comparing against openlayer-ts.

openlayer-ts already solves this class of bug: langchainCallback.ts declares PROVIDER_COST_SLUG (canonical display name → cost slug, with null for vendors that have no pricing upstream) and its test suite asserts every mapped provider resolves a price. Notably it does not split the field at runtime — step.provider still gets the canonical name; the map exists purely as a test-enforced invariant. That's why every TS canonical name is single-word: multi-word ones can't price.

This PR now ports that invariant to Python. The original space check catches the specific bug but not the general class — a value can be space-free and still price nothing. test_every_provider_value_resolves_a_cost_slug asserts every value in all three maps lowercases to a slug published by llm-costs, or is an explicitly listed display-only vendor (huggingface is the only one).

Red/green verified — against pristine main it fails with:

LS_PROVIDER_TO_OPENLAYER_MAP['fireworks'] = 'Fireworks AI' lowercases to
'fireworks ai', which is neither a published cost slug nor a known-unpriced
vendor: every row would cost $0

One deliberate divergence from TS: TS sidesteps together_ai/fireworks_ai by simply not mapping them, so a TS user behind a LiteLLM proxy to those vendors gets no provider resolution at all. Python maps them and now prices them (Together_AItogether_ai), which is why underscores appear here and not in TS.

Slug set is vendored rather than fetched so the suite stays offline; refresh with:

curl -s https://llm-costs.openlayer.com/v1/costs | jq -r '.costs[].provider' | sort -u

@viniciusdsmello
viniciusdsmello merged commit dc418a2 into main Aug 20, 2026
5 checks passed
@viniciusdsmello
viniciusdsmello deleted the fix/provider-cost-slugs branch August 20, 2026 23:44
@stainless-app stainless-app Bot mentioned this pull request Aug 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants