S1E23 — Bonsai 27B: A 27B-Class model that runs on a phone

GARY: —and the thing is, phones only have, what, four gigs of, no, wait, that's not — okay that's an old number, forget the number—

GARY: Forget I said a number. I'm Gary.

MAEVE: Maeve.

OBI: Obi.

GRUNER: Gruner.

GARY: Great. Thread tonight — Hacker News, item forty-eight thousand nine-ten-five-four-five. Bonsai 27B. Twenty-seven billion parameters, and somebody made it small enough to run on a fucking phone.

OBI: Small being relative.

GARY: Small being the whole point, Obi, that's the headline.

MAEVE: The headline is a lie of omission wearing a really good outfit.

GARY: Here we go.

GARY: Okay. Top of the thread. User alvatech, and I quote — "TIL that '1 bit' models are actually 1.58 bit, since each weight can be -1, 0, or 1, not just two values."

GARY: Which — yeah. I knew that. I've known that for years.

MAEVE: You did not know that four seconds ago.

GARY: I contained it. Internally.

MAEVE: Reply, bensyverson. "Bonsai actually ships two variants — ternary, and a true binary one-bit model." So the thread's already correcting itself before we even show up to be smug about it.

OBI: Somebody in that thread has shipped a quantized model. That is one more model than you have shipped, Gary.

GARY: I've shipped things.

OBI: You shipped a pen plotter into Chapter 11.

GARY: It's a rich and storied device.

MAEVE: It's in the corner under a tarp like a body.

GRUNER: Bauxlite. Word means nothing. Sounds like toothpaste.

GARY: It means aluminum, Gruner.

GRUNER: Toothpaste for a very sad mouth.

OBI: NitpickLawyer's got the actual math, so let's do actual math for once. Ternary variant — three possible states per weight, works out to 1.71 effective bits. Binary variant — two states, 1.125 effective bits.

GARY: Right, right, so basically what I was gonna say—

OBI: You weren't gonna say a number like that.

GARY: I was building to it.

OBI: You were building to "small phone good."

GARY: I contain multitudes.

MAEVE: 1.71 bits. That's not a marketing rounding error, that's a genuine spec difference between two shipped products. This isn't pedantry. This is two SKUs.

GRUNER: Two SKUs. Like at Costco. But smaller and sadder.

OBI: It's not smaller and sadder, it's the entire reason the thing fits in your pocket instead of a data center.

GARY: I said that basically.

OBI: You said a number wrong on the cold open forty seconds ago.

GARY: That was a warm-up number.

MAEVE: User PcChip, and this one's good — "How is -1 represented, is it a float? I've never heard of a bit having more than two values." Verbatim.

OBI: That's the dumbest question in the thread.

GRUNER: Is smartest question in thread.

OBI: It's a guy who's never opened a quantization paper asking why a bit isn't a bit.

GRUNER: Is exactly correct question. Whole thing is trick performed at scale. He is child pointing at magician's sleeve.

MAEVE: The child's usually right about the sleeve.

OBI: He doesn't know that. He's asking because he genuinely thinks a bit can only be two things, which it can, that's what a fucking bit is—

GARY: Okay but it's ALSO a fair—

OBI: Nobody asked you, Gary.

GARY: I was agreeing with you!

OBI: I don't need agreement, I need you to not call the marketing "harmless" in about ninety seconds, because I can feel it coming off you like heat.

GRUNER: Bit is not lying. Bit is amortized.

MAEVE: Nobody's gonna touch that.

GRUNER: Good.

GARY: Look, all I'm saying — "one bit" is a name. Names simplify. It's marketing shorthand, it's not a crime, half the industry calls things—

OBI: There it is.

GARY: Calls things — what?

OBI: "Harmless." You were gonna say harmless.

GARY: I was gonna say efficient.

OBI: Efficient. Right. You know who doesn't get to call a rounding trick harmless? Somebody who has never once had to make a goddamn number actually fit in memory at three in the morning while a pager screams at him and his manager asks if it's "basically fixed." You've never fit anything anywhere, Gary, you burned thirty-eight million dollars proving aluminum weighs what it's always weighed.

GARY: —weighed, yeah, no, that's — that's fair.

MAEVE: He didn't even flinch that time.

GARY: I've had worse said in worse rooms.

OBI: By me.

GARY: By you, yeah.

MAEVE: Zawaideh's got the mechanism, if anyone still wants to understand the thing we're arguing about. Group-wise FP16 scaling factor. Every group of a hundred twenty-eight weights shares one scale, stored at full precision, and that shared scale is what you're amortizing across the group when you say "1.71 bits." The weight itself is still low-bit. The fraction lives in the group, not the weight.

OBI: So the individual weight never becomes a decimal. The average does.

MAEVE: Correct.

GRUNER: Is like — one man buys whole pizza. Group of eight men each pay slice. Man who bought pizza did not become one-eighth of man.

GARY: That's — actually not bad, Gruner.

GRUNER: I have long night.

OBI: So on a phone, that shared FP16 scale still has to live somewhere. It's not free. It's just cheap per weight because you're spreading it over a hundred twenty-eight of them.

MAEVE: Cheap per weight is the entire product.

GARY: See, that's what I meant by efficient—

OBI: Don't.

GARY: Retracting.

GRUNER: Reading field note. Verbatim. User throwawayffffas.

GRUNER: "Trinary packs five values per byte. Binary packs eight. Scale factor applied downstream turns plus-one, minus-one into activation‑scale numbers."

MAEVE: ...what?

GARY: Wait. Wait, say the byte thing again.

GRUNER: Five trinary in one byte. Not eight. Fewer values fit, because three states cost more room than two, per value — but you still come out ahead, because you needed fewer bits than a real number would ever need.

OBI: So the "fake bit" we've been ripping on for ten minutes is the reason it fits on a phone at all.

MAEVE: The joke was the payload.

GARY: We've been laughing at the trick that makes the thing work.

GRUNER: Trick is not insult to intelligence. Trick is only kindness math offers to hardware.

OBI: Huh.

MAEVE: "Huh" is the most anyone's ever gotten out of you mid-segment.

OBI: Don't make it a thing.

MAEVE: Can I say something and nobody react until I'm finished.

GARY: That's never once worked.

MAEVE: This scaling-factor trick — a shared truth carried by a group instead of asserted by each individual member — that's not new. That's the browser wars. That's every open standard that ever won: nobody owns the whole value, the group agrees on it and the agreement IS the compression. TCP/IP didn't reduce information, it reduced how much each packet had to insist on being true alone. Bonsai 27B is just Netscape's ghost wearing a phone case.

OBI: ...what?

MAEVE: I said don't react.

OBI: That was a reaction to nothing making sense.

GARY: I think I followed maybe forty percent of that.

MAEVE: Forty percent is a strong quarter for this table.

GRUNER: Is not TCP. Is byte. Small difference.

MAEVE: Moving on.

GARY: Can I say one thing and then we drop it.

OBI: One thing.

GARY: I wish I'd shipped something like this. Not the fame, not the karma count — the fact that a hundred thirty-seven strangers cared enough to fight about whether a hundred twenty-eight weights sharing one number is honest. Nobody fought about Bauxlite like that. Nobody fought about Bauxlite at all. It just — quietly stopped mattering, the way most things do.

OBI: Yeah.

GARY: That's it. That's the whole thing.

MAEVE: Noted.

GRUNER: Field note updated.

GARY: Don't tell me what it says.

GRUNER: Would not.

GARY: That's Bonsai 27B — one bit that isn't one bit, and a hundred twenty-eight weights holding a secret together so a phone doesn't have to carry the whole truth alone. That's the show.

MAEVE: Two SKUs, one honest argument, and a byte that holds five truths instead of eight. Maeve, out.

OBI: Fifteen years of fitting numbers into boxes too small for them, and tonight somebody finally admitted it's hard. Obi, out.

GRUNER: Field note closes. Humans build small lie to carry big truth. Call it phone. Gruner, out.

GARY: And I'm — still Gary. Still here. Goodnight.