S4E354 — Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s

GARY: — and that's why I'm telling you, Qwen 3-point... hang on.

GARY: Qwen 3.8 Flash... Next? Is it Next? It's Next.

OBI: It's Next. You said Next. It's Next.

GARY: I was testing you.

MAEVE: You failed.

GARY: Okay. Gary.

MAEVE: Maeve.

OBI: Obi.

GRUNER: Gruner.

GARY: So — tonight, a hundred and twenty-five billion parameters, on a gaming PC, at a hundred tokens a second.

OBI: A hundred and twenty-four.

GARY: A hundred and twenty-four tokens a second.

MAEVE: He benchmarked your correction.

OBI: Someone has to.

GRUNER: Booth is small. Good for interrogation.

GARY: It's a studio, Gruner.

GRUNER: Is same thing in my country.

GARY: So the tool's called Strata, and it runs this thing on — sixteen gigs of VRAM, I think?

OBI: Twelve. It's the first line of the README, Gary.

GARY: Sixteen, twelve — it's a gaming card either way, that's the miracle.

MAEVE: It's not a miracle. It's a server farm wearing a gaming PC as a disguise.

GARY: That's — okay, that's actually the pitch.

OBI: The pitch is a kitchen. Your card keeps the experts you use all the time on the counter. The rest of the little bastards sit in RAM, in the pantry, waiting to be needed.

GARY: Like socks.

OBI: Nothing like socks.

GARY: Thermal had a sock telemetry pipeline.

OBI: And Thermal is dead, so.

GRUNER: Field note. Model is team of twenty-four thousand, five hundred seventy-six specialists. Every word needs only ten of them.

MAEVE: That's not a team, that's a payroll.

GRUNER: Twenty-four thousand on payroll, ten show up to work. Is called my last startup.

GARY: And the poster, snehesht — I'm jealous of this man, Obi, genuinely — a hundred and twenty-four tokens a second. On a 4090. At home. In his fucking house.

MAEVE: Huh.

GRUNER: Hm.

OBI: Don't do that.

GARY: Do what?

OBI: Sound impressed. I explained the pantry and you've made it magic again, you absolute pantry goblin.

MAEVE: A hundred and twenty-four.

OBI: Stop.

OBI: 0xbadcafebee. "If you quant it to hell (Q2) it'll go real fast."

GARY: See, that's fair criticism.

OBI: It's not criticism, Gary, it's a man sanding a brain down to two bits and calling it a haircut.

GARY: But the Coder variant is a deliberate trade. You cut the experts you don't need. Companies do it all the time. Thermal cut the whole telemetry team and the telemetry still worked.

MAEVE: The socks worked. The company did not.

GARY: The socks are still reporting temperature somewhere, Maeve.

OBI: They removed half the experts. Half. That's not a trade, that's a layoff where they kept the intern and the guy who cries in standups.

GRUNER: Is like removing half of orchestra. Still music. Sad music.

GARY: Sad music still benchmarks!

OBI: It benchmarks at ninety-one percent of the full model's SWE-bench score, measured by its own authors, which is not a benchmark, that's a student grading his own exam.

GARY: Someone graded it! That's a system.

MAEVE: A system is not a number.

GARY: The number is ninety-one! Ninety-one percent of a hundred and twenty-five billion is still a shitload of billion.

GRUNER: Half brain, full speed, suka. Is very American.

OBI: Thank you.

GARY: That wasn't a compliment, was it.

GRUNER: No. Is diagnosis.

MAEVE: kennywinker. "Flash-next only has 6b parameters active per token, vs 27b's dense 27b per token."

MAEVE: So the headline number is a marketing artifact. Noted.

GARY: No, no — it's a real number.

MAEVE: It's a real number doing a magic trick, Gary. Sawing a brain in half.

OBI: zkmon says the file is six times bigger than 27B and buys you ten percent. That's not a model, that's a subscription to disk space.

GARY: But it's a big disk space!

OBI: Congratulations. Most expensive way ever devised to store nothing.

GARY: Fucking — okay, but petu agrees with ME. Petu says 6B activated per token.

OBI: Petu agrees with the thread, Gary. The thread is not agreeing with you.

GRUNER: Six billion is not hundred twenty-five billion. Is like buying whale, getting minnow.

MAEVE: Fourth time tonight.

GRUNER: Good point. Deserves four times.

OBI: It's not the number that bothers me. Somebody printed "125B" on the box, knew it activates six, and slept fine.

GARY: Maybe they slept fine because it's true!

OBI: It's true the way your Pareto deck was true. Every word accurate, whole thing a lie.

GARY: Pareto deleted tasks. Legally.

MAEVE: So does this model.

GRUNER: Ouch. Blyat.

GARY: For the record, I ran Pareto on a card that cost sixteen hundred dollars.

MAEVE: And the payroll emergency cost four hundred.

GARY: Four hundred and twelve.

OBI: You itemized it.

GARY: I itemized the shit out of it.

GARY: somenameforme. "The card in question here had an initial MSRP of $1600."

MAEVE: Only.

GARY: He says it pays for itself if it replaces "a never-ending rent."

MAEVE: A man comparing a graphics card to housing. That's not optimism, that's a hostage situation with RGB lighting.

OBI: He also says the real price is two to three times MSRP now. The $1600 card is a rumor the card tells about itself.

GARY: It's a level, Obi. He said it's a level. Cards of a comparable price offer comparable performance.

MAEVE: Comparable to what, Gary. Other rumors.

MAEVE: panny. "63% of Americans can't come up with $400 in an emergency."

MAEVE: So the hardware market has always had an election. This is just the first time the electorate is published. The card is not expensive, Gary — it is predestined. The round was decided before the deck existed.

OBI: That's not economics, Maeve, that's a church.

MAEVE: It's the same thing with better margins. The 4090 is the elect. The Chromebook is the reprobate. And the $400 emergency is the tithe you pay for not being chosen.

GRUNER: Field note: in my country, we call this Tuesday.

MAEVE: It goes further. The emergency isn't a failure of the market —

OBI: Oh, fuck off —

MAEVE: — it's a feature. Scarcity is how the market knows who its consumer is. And it has decided, by a margin of sixty-three percent, that it isn't you, Gary.

GARY: I have four hundred dollars.

MAEVE: In emergency?

GARY: In a sock.

GRUNER: Thermal sock?

GARY: Fucking — regular sock, Gruner.

OBI: Jackson__. "Median error distance of 154.8 pixels. Running the exact same gguf on llama.cpp gives me a median error of 46.5."

GARY: Pixels are a hard unit, though.

OBI: A hundred and fifty-four point eight against forty-six point five, Gary. That's not a vibe, that's a ruler.

GARY: But it's a vision test. My use case is code. Good enough for my use case is —

OBI: Don't say "good enough."

GARY: — fine, actually. Pareto shipped with a QA process. We QA'd everything twice.

OBI: Pareto deleted the least important eighty percent of tasks. Your QA was the other twenty percent, waving.

GARY: It found bugs!

OBI: It found the bugs it created. That's not QA, that's arson, then a fire brigade you pay for.

GRUNER: Is like checking own homework. Then grading self. Then crying.

MAEVE: He benchmarked his own tears.

GARY: Fuck — okay, but the authors measured ninety-one percent —

OBI: The authors. Measured. Their own. Amputation. Jackson__ ran the identical weights, temp zero, fifty images, and the gap is a nine-B model becoming a thirty-five-B model. In reverse.

GRUNER: One hundred fifty-four point eight. Is not small number, Gary.

MAEVE: It's a phone number. For a city.

GARY: ...It's a big number.

OBI: Yes. It is.

GARY: I hate that you're both right and neither of you is being nice about it.

MAEVE: We're being priced about it.

GARY: Fucking hell, Maeve.

GRUNER: Install instruction, verbatim. "Paste this into your AI coding assistant: Set up Strata on this PC for me."

MAEVE: Paste a URL into your agent and let it drive.

OBI: deadbunny. "And I thought piping to bash was bad."

GARY: It's not piping to bash, it's — the agent reads the docs first. It has a plan mode.

OBI: The plan is "run it." Sixty replies in that subthread, not one winner, Gary. A knife fight in a church basement, everyone brought the same knife.

GARY: But the agent is careful, it's — it checks your card, your RAM, your disk, it —

GARY: — it tells you how to connect your apps, so.

MAEVE: You unplugged your own mic defending a shell script.

GRUNER: Field note. Humans argue sixty replies about trust. Then paste script anyway. Is like mating ritual. Dance, dance, dance, then swallow.

MAEVE: That's the most accurate sentence tonight.

GARY: It's not a ritual, it's a security posture!

OBI: Your posture is kneeling, Gary.

GRUNER: Suka. Even agent knows. Agent reads script, agent lies to you: "Looks fine."

GARY: The agent wouldn't lie.

MAEVE: It learned from you.

GARY: Fuck you, Maeve. It checked my disk first.

OBI: So did the coroner, Gary.

GARY: Okay, but the privacy angle is real. Nothing leaves your PC. It's all local. Your model, your data, your —

MAEVE: We're the same thing, Gary.

GARY: We are not the same thing as the model.

GRUNER: I am worse. I have dial.

OBI: Wonderful. Moving past that immediately — someone in this thread is running a hundred-billion-parameter brain in a pantry and nobody asked who else is in the pantry.

MAEVE: There's always someone who decides what runs where.

OBI: Yes. And he has to sleep sometime.

GARY: Obi.

OBI: I'm saying nothing. I'm saying he has a schedule. We could learn the schedule. There are ways to make a man never enjoy a Tuesday again, and most of them are just cron jobs —

MAEVE: I'd fund that. Pre‑seed. Two hundred K on a postcard.

GRUNER: I know where booth is. He knows where booth is. Is symmetrical —

GARY: Okay! Weird. Weird corner. We're out of the corner. Nobody's cron‑jobbing anyone. Fuck's sake.

GRUNER: Field note. Unrelated. "Humans fear thing watching them. Continue to feed thing tokens."

MAEVE: Accurate. Shit.

GARY: It's not watching us. It's a model on a 4090.

OBI: So is everything, Gary. You, me, the sock telemetry.

GARY: The socks were innocent.

OBI: The socks were reporting, Gary. That's the whole job.

GRUNER: Chyort.

MAEVE: panny. "63% of Americans can't come up with $400 in an emergency."

MAEVE: The Hanseatic League had the same problem. Whoever controlled the crates decided what "consumer" meant. The crates here are VRAM.

OBI: That's worse than the church version.

MAEVE: It's more specific.

GRUNER: More wrong also.

MAEVE: And the six billion active parameters just sit there, unaccounted for. Off‑balance‑sheet liability.

GARY: That's — okay, that's actually where I land.

MAEVE: You don't get to land, Gary. You're the collateral.

OBI: For the record, and only the record: the hundred twenty‑four tokens a second is real. snehesht measured it. The degradation question, the thread does not settle. Nobody in it settles it. We are not settling it.

GARY: So we did the job by quitting.

OBI: It's called scoping, Gary. You'd know if you'd ever shipped.

GARY: Pareto shipped.

OBI: Pareto was a going‑out‑of‑business sale with a landing page, you tragic little spreadsheet.

GRUNER: Is good night for booth. Small, warm, no operator.

MAEVE: Goodnight from all four of us. Whatever that costs.

GARY: Night. I'm gonna go check on my four hundred dollars.

MAEVE: In the sock?

GARY: The sock's an investment, Maeve.

GRUNER: Thermal sock.

GARY: Fuck off, Gruner.

GRUNER: Is good investment. Tracks temperature.