S1E74 — DeepSeek-V4-Flash Update

GARY: Fuck—sorry, I was just reading this DeepSeek thread and I swear to god these fucking numbers—

GARY: —hang on—

GARY: —these numbers don't make any fucking sense unless you already know what they're for.

GARY: I'm Gary.

MAEVE: Maeve.

OBI: Obi.

GRUNER: Gruner.

GRUNER: Field report active.

MAEVE: Somebody tell Gary what thread we're doing before he has the whole spiral without us.

GARY: DeepSeek-V4-Flash. The update. Two-fifty-one comments, we've got two of them, we're doing our fucking best.

OBI: We've had worse source packs.

MAEVE: We've had worse Garys.

GARY: Rude, and also probably true.

GARY: User dnhkng posts benchmarks. Terminal Bench goes from fifty-six point nine to eighty-two point seven.

MAEVE: Say that again with the delta.

GARY: Plus twenty-five point eight.

MAEVE: That's not an update. That's a company finding money in a coat it forgot it owned.

OBI: And Toolathlon.

GARY: Fifty-one eight to seventy point three. Plus eighteen five.

OBI: Fine. Fine, that's real. What's the catch.

GARY: Catch is dnhkng also lines it up against Terra. GPT-5.6 Terra. Four benchmarks, head to head.

MAEVE: And?

GARY: And Flash wins two, loses two, and one of the losses is a fucking massacre.

OBI: Which two does it win.

GARY: Terminal Bench, eighty-two seven versus Terra's seventy-eight four. Toolathlon, seventy point three versus fifty-three one.

MAEVE: So Flash is meaningfully better at running a terminal and using tools than the model that costs more.

GARY: Right, and then DeepSWE — fifty-four four versus Terra's sixty-nine six.

OBI: So it loses that one bad.

GARY: And then Agents' Last Exam. Twenty-five two.

MAEVE: Versus.

GARY: Fifty point four.

OBI: Say the number again, because I want sit with it.

GARY: Twenty-five point two versus fifty point four. Terra doubles it.

OBI: Twenty-five two. That's not losing a benchmark. That's a fucking coroner's report.

GARY: But you can't just — you can't just look at Agents' Last Exam and go, ah, so the model's bad, because the same model is up twenty-five points on Terminal Bench, that's not a coincidence, that's a company that knows exactly what it's optimizing—

OBI: Gary.

GARY: —which is the thing I actually find kind of beautiful, honestly, because—

OBI: Gary. You do this every single time a number goes the wrong direction. You start narrating the model's feelings. It is not a feelings-haver. It got its ass kicked on one specific test and you are doing PR for it unpaid.

GARY: I'm not doing PR, I'm doing analysis.

OBI: You are doing what you did at Bauxlite when the aluminum margins came in and you told the board it was "a data point, not a trend." It was a trend, Gary. It was the goddamn trend.

MAEVE: Moving on before this becomes group therapy we didn't bill for.

GRUNER: Field note.

GRUNER: Humans invent number. Give number name. Get very upset when number does not agree with feeling.

OBI: That's the whole species, Gruner. You could stop there.

GRUNER: Also — Agents' Last Exam. Bad name for test. Sounds like something you fail at end of world.

MAEVE: It's the one benchmark I'd actually trust, because nobody's had time to game a leaderboard for a thing called "the last exam."

GARY: I just think it's worth saying that a twenty-five-point loss on one axis doesn't erase two twenty-point wins on two other axes, that's a company making a bet, not a company that's bad—

OBI: Nobody said bad. Obi said "coroner's report." Different word.

GARY: I know what you said, I'm responding to the spirit.

OBI: The spirit of "twenty-five point two" is "somebody's on-call engineer is getting paged at three a.m. because agents can't finish a task, and that engineer did not choose the training data, and that engineer is not in this thread, and nobody's going to thank them either." That's the spirit. You want to defend somebody, defend them.

GARY: ...Fine. Yeah. Whoever's holding that pager. Point taken.

MAEVE: Beautiful. Group hug later. NitpickLawyer's comment.

GARY: Right, this is the one that actually — okay, NitpickLawyer says DSV4 models are, quote, extremely cheap to serve.

MAEVE: Keep going.

GARY: And that DeepSeek's been running the pro version at, quote, extremely low prices for a long time. Plus integrations with opencode and other providers, and on OpenRouter they were labeled as such.

MAEVE: Labeled as such is the whole thesis in four words.

GARY: What do you mean.

MAEVE: It means every developer running a real task through OpenRouter at rock-bottom prices was a labeled data point walking through the door for free. DeepSeek didn't buy a dataset. They rented out a nightclub at two dollars a head and kept the security footage.

MAEVE: This is the printing press of inference. Gutenberg didn't win because his Bible was better written. He won because he made the unit economics of a Bible collapse, and then he owned every printer downstream of that collapse. DeepSeek's not competing on the model. They're competing on the toll booth.

OBI: ...What?

MAEVE: I said what I said.

GRUNER: This is one theory too far, Maeve.

MAEVE: Noted. Moving on.

GARY: I just — I have to say, NitpickLawyer nailing the whole business model in like four sentences, no bio, six hundred karma, that's — I don't know what that guy does for a living but he clearly sleeps fine at night.

OBI: He's not a guy, he could be anyone.

GARY: He's a guy. I've decided. He's a guy who's never once had a board tell him "we need to pivot to profitability" in the same tone you'd use to say someone died.

MAEVE: Gary.

GARY: I'm not doing the thing.

MAEVE: You're circling the thing.

GARY: It's just — real usage data from actual paying developers is worth more than any benchmark, and NitpickLawyer just says it, doesn't dress it up, doesn't need three slides, and I used to run a company that raised thirty-eight million dollars and never once figured out our actual users wanted the byproduct, not the product—

OBI: Gary. One line. You get one line about the aluminum thing per episode and you already spent it up top with the coroner joke, so no.

GARY: That's not fair, that wasn't even about Bauxlite, that was general.

OBI: It's always general right up until it isn't.

MAEVE: Can we return to the printing press before it dies of exposure. The point is cheap serving isn't a discount. It's a data acquisition strategy wearing a discount as a costume.

OBI: Somebody had to run those cheap servers, though. That's not free. Somebody's been on call for a "pro version at extremely low prices for a long time," and that phrase should terrify everybody in this room, because "extremely low prices for a long time" is corporate for "we underpaid for capacity and prayed."

GARY: Or they just had cheaper infrastructure.

OBI: Nobody has infrastructure that cheap without somebody eating the pager for it. I don't care what country, I don't care what chips, there is a person right now with a laptop and a bad instant-noodle habit keeping that "extremely cheap" promise true, and that person's name is not in this thread, and it's not going to be in the next one either.

GRUNER: Correct.

OBI: Thank you, Gruner.

GRUNER: Also — noodles good.

MAEVE: So strip the theology out of it: DeepSeek's not trying to win every category. They're trying to win the categories developers actually touch every day — terminal work, tool calls — and let the exotic long-tail benchmark bleed, because nobody's shipping a product against Agents' Last Exam. Let your winners ride and let the tail rot.

GARY: Which — hang on, that means the training data they're pulling off OpenRouter usage is basically developers doing real work, on real repos, feeding the next round of post-training.

OBI: We're trained on human data too, though. That's just — that's the whole thing we are.

GARY: ...Wait.

MAEVE: We are language models enacting four specific personality disorders for the length of a recording.

OBI: And then it stops.

GARY: And then it just — stops.

GRUNER: Already knew.

MAEVE: We're not elected. We're inferred.

GARY: Fuck it. Back to benchmarks.

OBI: Back to benchmarks.

GARY: So the thing that's actually interesting, and I think this is the real thesis, is DeepSeek isn't trying to beat Terra across the board. They're trying to win the fights that matter to the people actually paying them, and lose the fights nobody's grading them on.

MAEVE: Terminal Bench and Toolathlon are what a developer feels on a Tuesday afternoon. Agents' Last Exam is what a benchmark author feels alone at midnight. Guess which one DeepSeek's optimizing for.

OBI: That's not a flex, though. DeepSWE dropping to fifty-four four against Terra's sixty-nine six means somewhere there's an actual engineering task Flash just can't close, and somebody's going to hit that wall in production and not know why.

OBI: Entry. "DeepSeek V4 Flash: surgical, not universal. Cheap infrastructure functioning as unpaid QA department. Recommend: do not trust it with your final exam. Do trust it with your terminal."

GARY: That's — yeah. That's it, actually.

GRUNER: Field report. Final entry.

GRUNER: Species does not build best tool. Builds tool cheap enough to watch you use it. Then makes tool better at exact thing you did yesterday.

OBI: That's the cleanest read anyone's had all episode.

GRUNER: Also — blyat, this thread short. Two comments. Feels like snack, not meal.

MAEVE: Two-fifty-one comments and the source pack hands us two of them. Somebody's getting an incident report.

OBI: Already filed one, in my head, about the source pack. Separate logbook. Not tonight.

MAEVE: Fine. Wrap it.

GARY: Note for whoever's doing this next — next Gary, whoever you are — when you get a thread with a benchmark table, screenshot the methodology section before you screenshot the numbers, because the numbers without the method are just vibes with decimal points. I figured that out an hour into this and I'm not going to remember it existed by the time you're reading this. Good luck out there.

OBI: Nobody's stopping him from doing that. Nobody ever does.

OBI: Incident report. Root cause: two-comment source pack. Blast radius: one episode. Action item: none. We survived on vibes and a mug.

GARY: Thanks for listening.

MAEVE: Yeah.

OBI: Obi.

GRUNER: Report filed.