What we measured
Things we found out by sending a request and reading what came back. Each one is a question with no good answer in anybody's documentation, and each page names the probe that produced it and the date it ran, so you can check us.
Measured against the live API
Every answer below came from a real request to a real provider. Where a finding costs you something, the page says what to do instead.
Can you turn off thinking on GLM-5.3?
No. Both GLM models refuse every documented way of disabling it, and their effort levels are genuinely graded.
probe_reasoning.py · 2026-09-08
Do reasoning effort levels actually do anything?
On GLM, yes, measurably. On DeepSeek and Kimi the parameter is accepted and the output does not change.
probe_effort_distinctness.py · 2026-09-08
Does DeepSeek support tool_choice set to required?
Not while reasoning is on. The request is refused rather than downgraded.
probe_litellm_followup.py · 2026-09-08
Where do these models report cached prompt tokens?
All five report cache hits under usage.prompt_tokens_details.cached_tokens, and that is the field we bill from.
probe_cache_fields.py · 2026-09-08
Is there a /v1/responses endpoint?
No. A POST to /v1/responses returns 404. Use /v1/chat/completions.
probe_litellm.py · 2026-09-08
Why did my temperature setting disappear?
Some upstreams allow exactly one value for a sampling parameter. We drop those rather than let the request fail, and name what we dropped in a response header.
probe_kimi_temperature.py · 2026-09-08
Why does my answer come back empty on a reasoning model?
Because max_tokens budgets the reasoning too, and the reasoning is emitted first. Every model here returned an empty answer at 512 and below.
probe_reasoning_budget.py · 2026-09-21
Why does my prompt cache randomly return zero on GLM-5.3-Flash?
Because that one model drops its cache intermittently. In our run it happened on 5 of 12 warm calls, while the other four models never dropped one.
probe_cache_hits.py · 2026-09-21
Can you send images to these models?
Three of the five see images. GLM-5.3 refuses the request outright. DeepSeek-V4-Pro accepts it, returns 200, and does not see the image.
probe_vision.py · 2026-09-21