Fix Broken Tool Calls in vLLM, SGLang and llama.cpp

Short answer: log the raw completion next to the parsed tool call, then disable speculative decoding. Those two steps cost one restart and about twenty minutes, and between them they identify or eliminate the two most common causes. Do not start by swapping models or engines — most reported corruption traces to the engine-side parser or to optimizations that ship enabled, not to the model.

Symptoms

• The agent stops calling tools entirely rather than calling them badly — the first tool call never happens and the workflow halts.
• Tool arguments arrive as malformed JSON, truncated objects, or empty strings.
• Generation stalls or returns finish_reason=length before any tool call.
• Output degenerates into repeated tokens or filler reasoning.
• Throughput benchmarks still look healthy.

That last point is why this goes unnoticed. A tokens-per-second number cannot tell you whether the tokens were the right ones.

Three causes, one symptom

Issue numbers below were re-verified against live issue bodies on 2026-08-22 — bodies, not titles. Several citations were dropped from an earlier draft of this research when the bodies turned out to describe a VRAM leak, a throughput complaint, or a config question rather than tool-call corruption. These are user reports and open issues, not confirmed vendor behaviour.

Cause	Where it lives	Reported on
Parser bugs	Engine-side extraction of the call from raw output	vLLM #53246, #53227, #45167, #44676; SGLang #35565, #35564; llama.cpp #26987, #27363; Ollama #17638, #16648, #16686
Speculative decoding	Draft-and-verify decode path	vLLM #46249, #36872, #34449, #43221; SGLang #32038, #9187
Quantization	Weight precision	vLLM #13530
Both stacked	Interaction, not sum	vLLM #36872’s AWQ-4bit note; SGLang #4351, #35324

The first row is the underestimated one. Eleven reports across four engines that share no parser code say the same thing: extracting a structured call from raw model output is done by the engine, and it is harder than it looks.

The last row is the argument for bisecting rather than reasoning. Quantization and speculative decoding stacked are reported to produce three different failure shapes — malformed JSON, stalled generation, and a hard crash — from two optimizations engines commonly ship enabled.

Why the two optimizations interact

Speculative decoding proposes several future tokens with a cheap draft path, verifies them against the target model, and keeps the ones that match. The acceptance decision compares distributions.

Quantization shifts those distributions slightly. That is the entire trade: less precision, nearly the same output. But “nearly the same” is load-bearing when another mechanism is making accept-or-reject decisions by comparing exactly those numbers.

So they compound. The combination can misbehave when neither part misbehaves alone, which is why reading changelogs is a poor substitute for changing one thing at a time.

The bisect, in cost order

Each step is one restart and one change. Do not combine them.

1. Log raw output next to the parsed call

This single change tells you which half of the problem you have.

# vLLM: capture the raw completion before tool-call parsing.
# Prerequisite: your client sends tools; VLLM_URL points at the server.
curl -s "$VLLM_URL/v1/chat/completions" \
  -H 'Content-Type: application/json' \
  -d '{"model":"'"$MODEL"'","messages":[{"role":"user","content":"What is the weather in Paris?"}],
       "tools":[{"type":"function","function":{"name":"get_weather",
                 "parameters":{"type":"object","properties":{"city":{"type":"string"}}}}}],
       "tool_choice":"auto","max_tokens":128}' \
| python3 -m json.tool

Read choices[0].message. If content holds a well-formed call and tool_calls is empty or wrong, the parser is at fault. If content itself is malformed, the problem is upstream in decoding or the model.

2. Disable speculative decoding

Restart the server without the draft/MTP configuration. On vLLM that means removing --speculative-config (or the older --speculative-model / --num-speculative-tokens flags, depending on version); on SGLang, the --speculative-algorithm family.

If tool calls recover, you have your cause and a throughput decision to make.

3. Drop to unquantized weights

If you have the VRAM. A fix here puts you in vLLM #13530 territory, and the question becomes which quantization format rather than whether to quantize.

4. Change the parser, not the engine

Most engines expose a tool-call parser choice — on vLLM, --tool-call-parser with a matching --chat-template. Trying a different parser is far cheaper than migrating engines and resolves a surprising share of cases.

5. Only then consider a different engine or model

By this point you have eliminated three cheaper causes. Jumping here first is what turns a bad afternoon into a bad week.

A dated example, and its limits

On vLLM 0.24.0 with Qwen3.8-27B BF16 and MTP=3, concurrent long-context tool work degenerated into repeated ! reasoning, empty assistant messages, and finish_reason=length before a single tool call. The same prompt worked after a restart with MTP removed.

Two days earlier the same feature delivered a genuine throughput win on the same hardware. Full receipts, including the benchmark that looked healthy while the agent was already broken, are in the field report on vLLM MTP quietly breaking tool calls.

That is one dated failure class on one configuration, not a verdict that speculative decoding is unsafe.

Traps in the diagnosis itself

The failure is concurrency-dependent. The example above appeared under concurrent long-context work and not in single-request testing. A clean bisect at concurrency one can be entirely misleading, so reproduce at the concurrency your agents actually use.

Changing two things at once destroys the evidence. Disabling speculative decoding and switching quantization in one restart tells you nothing about which mattered.

Version drift re-enables the cause. These are optimizations engines enable by default. An upgrade can restore a flag you deliberately turned off, so pin the version or assert the setting at startup.

Your load test cannot see it. Add an assertion that counts successful, well-formed tool invocations out of attempts, and run it at realistic concurrency. Without that number, a fast benchmark will keep reporting success on a broken lane.

When it’s actually something else

If the raw output contains a well-formed call and your client is mishandling it, the fault is in the client’s tool-call handling, not the serving stack. Step 1 distinguishes these.

If you rent inference from a hosted API, this is not yours to debug — you have neither the flags nor the visibility. Report the behaviour and switch models.

If tool calls succeed but the agent chooses wrong tools, that is model capability or prompt design. Nothing in this guide applies.

Reference

• Engine-side parser bugs: vLLM #53246, #53227, #45167, #44676; SGLang #35565, #35564; llama.cpp #26987, #27363; Ollama #17638, #16648, #16686
• Speculative decoding correctness: vLLM #46249, #36872, #34449, #43221; SGLang #32038, #9187
• Quantization and stacked failures: vLLM #13530; SGLang #4351, #35324
• Retry cost when this goes unnoticed: why your LLM spend limit doesn’t actually stop spending