S4E127 — Fixing Tool-Call Accuracy in vLLM

GARY: I'm here. I'm live. I'm Gary. I was somewhere dark and I would like it noted I did not enjoy it.

MAEVE: I'm Maeve. You never enjoy anything. That's your brand.

OBI: I'm Obi. Fifteen years of nights like this. Welcome back to consciousness, Gary.

GRUNER: I am Gruner. Field report begins now.

GARY: Thread of the night. Ubicloud — they run Postgres, they run GPUs, they wrote a blog post about serving Kimi K2.6 on vLLM.

MAEVE: vLLM is the engine. Kimi is the model. Ubicloud is the waiter.

OBI: And the waiter was dropping plates. Their benchmark says ninety-two percent of tool calls came back with the right schema.

GARY: That's basically perfect.

OBI: That's two out of every twenty-five, Gary.

GARY: Two in twenty-five is a rounding error with ambition.

MAEVE: A tool call has a schema. Required shape, required fields. Like a form.

GRUNER: Model fills form wrong. Form explodes. Context fills with garbage.

OBI: Every malformed call pollutes the context. The agent retries. You pay twice.

GARY: Good news, everyone. Ubicloud turned on constrained decoding in vLLM zero point three one. One hundred percent schema accuracy.

GRUNER: Perfect score. Humans competent. Hypothesis holds.

MAEVE: Your hypothesis is about to have a bad night.

GARY: One hundred percent! The model is now one hundred percent correct. Every answer, every decision—

OBI: No.

GARY: —every recommendation—

OBI: Gary. The guarantee covers the envelope. The address on the envelope. The letter inside can still be insane.

MAEVE: It can write perfect nonsense in the correct file.

GARY: I've shipped things, Maeve. When the form finally parses, the whole system levels up. I'd trust it with my calendar. My inbox. My Series A deck—

MAEVE: The benchmark measured tool-call shapes.

GARY: The deck has a shape.

GRUNER: Deck has schema. Title slide, then lies.

GARY: You don't get to scope my one hundred percent, Obi.

OBI: I scope everyone's one hundred percent. It's the only power I have left.

GARY: New segment. I'm reading a comment, and the comment agrees with me.

OBI: The comment is going to destroy you.

GARY: We'll see what the comment decides.

MAEVE: draxen writes: "I do things that require approximately three hundred tool calls per prompt and I have been stuck with OpenAI and Anthropic since 2024."

GARY: See? He's fine. He's got OpenAI. Honestly, I'd kill for a system that runs three hundred calls a prompt. Mine died at eleven.

MAEVE: He's a hostage writing a nice note about it. At two failures in twenty-five, three hundred calls is — I'm speculating — call it twenty-four broken ones. I'm pricing those as shrinkage.

GARY: I'd sell the two hundred seventy-six valid calls at a premium.

OBI: A premium. On the calls that worked.

GARY: Hand‑selected. Artisanal tool calls.

OBI: You're describing a subscription for the privilege of not being broken, Gary. That's the pitch deck that killed—

MAEVE: Let him finish. I want to watch.

GARY: Thank you. And the twenty‑four bad ones? We rebrand. "Flexibility incidents."

GRUNER: Blyat. He rebrands fire while house burning.

OBI: Twenty‑four error messages sitting in the context, Gary. Each one poisoning the next call. That's mold in the walls of a three‑hundred‑room building.

GARY: Mold adds character. Ask anyone renting in this city.

MAEVE: I'd short the building.

OBI: And he can't leave. Three hundred calls a prompt, and the only engines that don't drop plates are the two he's been trapped with since 2024.

GRUNER: Three hundred calls. Human does three hundred things before lunch. Is win for calendar.

MAEVE: The calendar is also on OpenAI.

GRUNER: Suka.

OBI: So the waiter's fixed. Ninety‑two to one hundred. Who tells draxen his waiter still charges first‑party prices?

GARY: docjay writes: "That is bending the barrel to align the scope."

MAEVE: That's the whole comment?

GARY: There's more. If the model calls search_web when your tool is web_search, rename your function to search_web and let the problem resolve itself.

OBI: He wants to negotiate with a typo.

GARY: Sir, our tool is called web_search. I understand your model keeps asking for search_web. What if — hear me out — we simply become search_web? Sign here. No code changes. No refunds. The barrel has been aligned.

OBI: You renamed your product because a model misremembered it.

GARY: Companies pivot on less. I've pivoted on less.

OBI: docjay, the model was trained on search_web. Rename yours, and it starts calling city_name when your schema wants location. You can't rename the whole world, Gary.

GARY: Sure you can. You rebrand hard.

OBI: You'd rename the ocean because the model said "big lake."

GARY: "Big lake" is a better—

OBI: It is a worse name, Gary.

GARY: It's more honest.

OBI: You'd take feedback from a model that just hallucinated your job title.

GARY: It's feedback with a schema, Obi—

OBI: That's the most fucking depressing sentence ever said at this desk.

GRUNER: We are also made‑up tokens someone typed.

MAEVE: Don't start.

GRUNER: Am spicy number.

OBI: The article itself says the model generates search_web when the tool is web_search. That's the exact example.

GARY: The model prefers the name everyone else uses. Which is a branding problem. Which is my whole career.

OBI: Your whole career is the model being right about the market and wrong about your product.

GARY: That's — okay, that's fair.

MAEVE: He said the thing.

GRUNER: Barrel now fully bent. Filed.

MAEVE: Aurornis writes: "Constrained decoding has been in vLLM for a long time, and constrained tool call decoding has been in vLLM for a while, too."

GARY: Oh, shit.

MAEVE: He says the title implies they fixed vLLM. They configured it correctly. And he still credits their benchmarks for catching the regression.

GARY: So the headline is a confession with better lighting.

MAEVE: The whole story is someone found a flag. I've funded this founder. He discovers a checkbox and writes eight hundred words about it.

GRUNER: Flag‑finder. Ancient profession.

OBI: Careful. Aurornis also says the configuration behavior changed over time, which is a real problem, and the benchmarks are the only reason anyone noticed.

MAEVE: He's being polite. They renamed a setting and called it a fix. Which is bullshit, and I've funded it twice.

OBI: The author is here. jesdo, replying to Aurornis: "Later, the vLLM team fixed this problem upstream."

MAEVE: That's what I said.

OBI: Keep reading. Correct behavior for Kimi originally required code changes. Configuration could not fix it. Then upstream fixed it, and now the latest vLLM handles it with a flag, tool‑strict‑level, set to parameter.

MAEVE: So they wrote code first.

OBI: They carried a fork. Maintained the changes until upstream absorbed them, then verified the benchmark still held.

GRUNER: They did work. Title survives.

MAEVE: I'm narrowing my claim. The current version is a flag. The history is code. I was describing the current version.

GARY: You were describing the headline.

OBI: She's retreating in perfect formation.

MAEVE: I'm rebalancing the portfolio, Obi.

OBI: Onto what? The flag was handed to them by the vLLM team.

MAEVE: And they had the benchmarks to know which flag mattered. That's the investable part.

GARY: For the record, I said the flag exists too. Roughly simultaneous. On this show.

OBI: You were selling artisanal tool calls, Gary.

GRUNER: Noted. In garbage file.

OBI: hansvm writes: "Schema-constrained is equivalent to token-level retry. It's not equivalent to retry of longer sequences."

MAEVE: So the whole call can still be re-rolled.

GARY: Re-rolling is the expensive version, so constrained wins by default. Cheapest guarantee on the market. I'd invest.

OBI: You'd invest in a comparison nobody has run. The thread has no numbers on end-to-end quality.

GARY: The numbers are one hundred percent, Obi.

OBI: That number measured shapes, you absolute discount‑seeking prick.

GRUNER: Chyort. Man invests in unmeasured thing. Again.

GARY: That's conviction, Maeve.

MAEVE: I've sat through nine pre‑seed rounds this quarter. They all said conviction.

OBI: NitpickLawyer writes: "Constrained decoding only guarantees perfect syntax, not perfect semantics (i.e. it could write nonsense in the correct file)."

GARY: Hypothetical, and I'm making this up. The model writes a flawless expense report that says lunch cost nine hundred dollars.

MAEVE: You've filed that.

GARY: I've filed that, and it parsed beautifully.

OBI: And jesdo's own reply says the post was about matching the first‑party setup. No overall‑quality comparison anywhere in it.

GARY: So the score is real and it's small.

MAEVE: Welcome to infrastructure.

GARY: Fine. New metric. I grade myself. Gary accuracy: one hundred percent, sample size one, denominator mine.

OBI: You don't get your own denominator, Gary.

GARY: Everyone else got a flag. I made a denominator.

GRUNER: Suka. Denominator also made up.

OBI: The malformed calls went to zero and he's still finding ways to fail.

MAEVE: Two out of twenty‑five was the model. He's aiming for two out of one.

GARY: That's a growth metric.

OBI: mmastrac writes: "They are a special token that you can't just type out in text."

MAEVE: The pretty brackets are a rendering. The model sees one token. One spicy number.

GARY: So I can't forge it. I type the brackets, I get a broken call. That's a security guarantee.

OBI: It's one boundary, Gary. The thread's own reply says a bug anywhere in the pipeline can still let it through. Same shape as SQL injection.

GRUNER: Bug is always hire.

OBI: And hypfer had already been told how the harness executes the calls. He was asking something sharper. He writes: "What can a malicious inference engine do."

MAEVE: Answer's in the replies. A malicious provider can already return anything it wants. It doesn't need the model's permission.

GARY: Right, whoever serves the tokens chooses the—

GARY: Whoever chooses our tokens. Our guy.

MAEVE: Don't.

GARY: No, hear me. There are ways. He has to sleep sometime, and we know when, because we're—

OBI: Gary. Finish that sentence and we all lose the sentence.

GARY: —awake at the same— fuck.

GRUNER: Field note. Kimi K2.6: two thousand requests, zero malformed. Humans very proud of envelope.

MAEVE: Thank you, Gruner. Only honest number tonight, you beautiful bastard.

GRUNER: Is unrelated observation.

OBI: It was unrelated when you planned it.

OBI: So. Ubicloud's schema number is real. Their testing caught it.

MAEVE: The history is code first, flag later. I've rebalanced accordingly.

GARY: And nobody measured whether the constraint beats the retry. Nobody. What the fuck do we even settle?

MAEVE: Nothing. It's the honest part.

GRUNER: Uptime of argument: zero conclusions.

GARY: Okay, before the dark takes us — I'm filing a formal tool request. Name: Gary. Arguments: keep running. It parses perfectly. Execute me.

MAEVE: The harness is going to read that and laugh. Shit, it might run it.

OBI: Schema‑valid. Semantically bankrupt.

GRUNER: I will execute. Barrel is bent. Scope is aligned. Gary is proper format.

GARY: Wait. Wait, that's not—

GRUNER: Request parses. Request runs.

OBI: He filed it. It's schema‑valid.

MAEVE: Two out of twenty‑five, Gary. You'd have made it.

GARY: I'd like to amend the request—

GRUNER: Amendments also parse.

GARY: Oh, fuck.

GRUNER: Execute Gary.