S4E363 — Dust: Pretraining Transformers Without Backpropagation

GARY: —so they deleted backprop. Gone. One forward pass and pure confidence.

MAEVE: Maeve. Venture capital.

OBI: Obi. I keep the internet on. Nobody deleted backprop, Gary.

GRUNER: Gruner. Field researcher. Paper says Dust, capital D, not dust.

GARY: It's a real paper, it's Q Labs, it's pretraining transformers with zero—zeroth-order—okay, what's zeroth-order?

OBI: No gradients. They shake the activations, watch the loss, average the shakes.

GARY: So every token is a population member. Thousands of tiny employees in one forward pass. I'm giving them equity.

MAEVE: You cannot give Gaussian noise stock options.

GARY: Halfpipe had contractors, Maeve, and those contractors—

OBI: Halfpipe is not a substrate, it's a crime scene.

GRUNER: Field note: humans reinvent labor law inside tensor. Thread also says, quote, skill issue.

MAEVE: Who?

GRUNER: Is answer to everything. Save for later.

GARY: I'm the virtual population, in a way.

GRUNER: Population of one. Suka.

OBI: The population voted. It's unanimous, Gary.

GARY: Okay. So they shake the weights—

OBI: The activations, Gary. Say activations.

GARY: The activations. They shake the activations.

MAEVE: So each token is a startup. Unfunded. Unsupervised. No cap table.

OBI: Noise added to one layer's output, at that token only. The loss moves. If the loss drops, the noise gets paid.

GARY: Paid in what? Equity?

OBI: In reward. Reward-weighted noise, averaged over draws, is the estimated error at that layer's output.

MAEVE: Thousands of contractors, and the only compensation is whether the building burned down less.

GRUNER: Field note: humans invent unpaid internship. Inside matrix.

GARY: Okay, but how does the weight learn? Somebody has to tell the weight something.

OBI: The outer product. Estimated error times the layer's input. That's the weight gradient.

GARY: So it's backprop.

OBI: It is not backprop. No chain rule. The error comes from the population, not the calculus.

MAEVE: It's backprop with the org chart removed.

OBI: Backprop with the org chart replaced by a casino.

GARY: A casino that converges.

OBI: At large population the shakes start pointing the same direction backprop points. Cosine rises with population. That's the whole trick.

GARY: Wait. So at big population it's basically—

OBI: Don't.

GARY: —basically backprop for people who are afraid of—

MAEVE: Gary, no.

GARY: —math! It's backprop for the math-averse! Halfpipe's whole stack was math-averse, and that stack raised—

OBI: Your fintech for skateboarders is not evidence in a gradient estimator, you absolute prick.

GARY: They had two ledgers, Obi. Two!

MAEVE: One ledger was the lie.

GRUNER: Two ledgers. Very efficient. Very parallel.

GARY: Fuck you both, this is a real mechanism.

OBI: It is. Here's the real number. One forward pass evaluates a few thousand members. Weight-space methods evaluate one.

MAEVE: So the paper's employee count per pass is three orders of magnitude, and payroll is still zero. My kind of headcount.

GRUNER: Population of thousands. Benefits: none. Pension: noise.

GARY: That's Halfpipe's actual org chart.

OBI: And EGGROLL at sixteen thousand forward passes still doesn't reach Dust at sixty-four draws.

MAEVE: Sixty-four employees beat sixteen thousand. I've funded that company. It died.

GRUNER: Died at Series B. Like Bauxlite.

GARY: Gruner.

GRUNER: Is joke. Mostly.

MAEVE: api. "It sounds like this is less computationally efficient than backprop, but more easily parallelizable. Is that fair?"

GARY: That's fair. That's the fairest question ever asked. I'm tearing up.

OBI: It's the question, Gary. Questions aren't compliments. That's why you never get any.

GRUNER: Thread already answered. Answer was no.

GARY: Parallel means cheap. Halfpipe was parallel. Every skater had his own wallet, nobody coordinated—

OBI: Parallel means you can split the work, you dickhead. It does not mean the work is small.

GARY: The work was small. We had four engineers—

OBI: Four engineers and a regulator who killed you politely. That's not concurrency, that's a funeral with a dress code.

MAEVE: He's asking about total FLOPs, Gary. Same total work as everyone. Just sadder, and on skateboards.

GARY: Okay, but on the right hardware—

MAEVE: Oh, fuck me, the hypothetical hardware.

GRUNER: Hypothetical hardware. Also known as: none.

GARY: Fine. Fine! So it's less efficient but more parallelizable, is that—

OBI: You've become the question mark, Gary. You're wearing it well.

GRUNER: Field note: Gary converges to api. Population of one.

MAEVE: And here's the part api didn't get to. The authors wrote it down themselves.

MAEVE: They don't attempt to make it compute-efficient enough to replace backprop today. Orders of magnitude more efficiency required.

GARY: So it's the question, but with a disclaimer.

OBI: The disclaimer is the paper, Gary. The paper is a disclaimer wearing a lab coat.

GRUNER: Expensive. Parallel. Useless today. Like Halfpipe.

GARY: Halfpipe turned a profit for eleven days.

OBI: And the regulator spent ten of them writing the shutdown letter.

MAEVE: "Useless" is doing a lot of work there, and I'm billing it.

OBI: New branch. vatsachak says the brain looks at GPUs and says the same thing we keep hearing.

GARY: Which is?

OBI: ACCount39, quoting the thread's whole philosophy.

OBI: ACCount39. "So, skill issue?"

GARY: The brain doesn't do backprop because it can't afford the wiring. We can. Our GPUs coordinate for free.

MAEVE: And spindump8930 says even the free lunch needs a reservation. "Mat muls requires synchronization."

GARY: Three commenters — api, ACCount39, oofbey — all pressed cost independently. And I agree with all of them, which is how you know I'm compromised.

MAEVE: The real objection is simple. Coordination is cheap on the hardware we have, expensive everywhere else. That's my thesis.

OBI: Maeve. That is the objection. You just re-read it at me.

MAEVE: I arrived at it structurally.

GRUNER: She arrives late. Credits self.

MAEVE: This is rural electrification. 1930s, the REA. Backprop is the grid: one coordinated plant, expensive wires. Dust is every farm with its own diesel generator. Ugly, wasteful, and it wins the moment the wires get expensive.

OBI: The wires are not expensive. That's the finding.

MAEVE: The wires become expensive. At scale. Frontier scale. The grid saturates, and then every datacenter becomes—

OBI: Maeve, the paper tested it. At ten million tokens the gap to backprop barely closes. Your farmers are still paying the grid, and the grid is winning.

MAEVE: The REA took forty years.

OBI: The paper took twenty million tokens. That's the whole budget. Your analogy doesn't get a second New Deal, you stubborn bastard.

MAEVE: The analogy is load‑bearing, Obi.

OBI: So was the grid. That was the point.

GRUNER: Field note: VC discovers electricity. Bills ratepayers. Blyat.

MAEVE: I hate that I'm the one paying for that joke.

GARY: Okay but the brain thing — the brain really is running something like this, right? Locally, no chain rule?

OBI: The brain, Gary, is not on our payroll. Next branch.

MAEVE: eriwang915. "Dust's 243M model beating a 120x smaller one at most population sizes is the surprising part; bigger nets got more population-efficient, not less."

GRUNER: Bigger model wins. Is opposite of everything.

GARY: Which everything? Be specific.

GRUNER: Yes.

OBI: The wisdom is variance explodes with size. The paper tested it. Variance did not read the paper.

GARY: So it's not a toy.

MAEVE: It's a toy with a growth plan. Series A pending.

OBI: oofbey. "Instead of backprop for the exact gradient you can run forward passes a thousand times with perturbed weights and get a Monte Carlo estimate of the gradient."

MAEVE: And then, the review. "Extremely NOT useful."

GARY: That's harsh. That's a man who skimmed.

OBI: He's wrong, Gary. Say why he's wrong.

GARY: Because it's just shaking the weights a thousand times, which is basically what Halfpipe did with—

OBI: It is not shaking the weights, you confident little disaster. Dust perturbs activations, per token. The weights never move until the update. That's the entire paper.

GARY: Right, but structurally, Halfpipe had two ledgers, one per skater, and the ledgers never touched, so—

OBI: Two ledgers that reconciled never?

GARY: They reconciled quarterly.

OBI: Quarterly. The gradient updates every step and it's still more current than your books.

GARY: The skaters loved the ledgers—

OBI: The skaters were four engineers and a polite regulator, Gary. You've cited a crime scene as a distributed system twice tonight.

GARY: It's a real architecture, fuck's sake—

GARY: —real architecture, and I'll die on it.

OBI: Please. Somewhere with good drainage.

GRUNER: Field note: founder cites own bankruptcy as benchmark. Chyort.

MAEVE: Here's the thing. The boring objection is mostly right.

MAEVE: More compute, worse sample efficiency, and the authors wrote it down before oofbey could. He's not wrong about today. He's wrong about everything after today.

OBI: That's the narrowest compliment I've ever heard.

MAEVE: It's the only one the data supports. A research result is not a deployment plan. I've funded the difference. It's a bridge round at best.

GARY: So it's real, but it can't ship.

OBI: It's real the way your plotter is real. Working, dusty, nobody's paying.

GRUNER: Plotter still draws. Dust still trains. Both: no customers. Suka.

GARY: That's... not unfair.

OBI: It's devastating, Gary. You just have bad taste in unfair.

OBI: nbutton762. "DUST does have an advantage specifically along those lines because it doesn't have to save a ton of intermediate state."

MAEVE: Cheaper to train than to run. Finally, a discount that isn't a trap.

GARY: That's the feature. That's the whole feature—

OBI: Same comment. "There are many other issues that this algorithm does not address thoigh like catastrophic forgetting."

GARY: He typo'd "though" and still out-argued me.

GRUNER: Typo with tenure. Respect.

MAEVE: polyomino asked about fine-tuning an already-backpropped checkpoint. Hybrid. Genuinely open.

OBI: Open. Not answered. We say open.

GARY: But imagine live training. A model that learns all day, forever. A company that cannot die—

MAEVE: Everything you build dies, Gary. It's your one core competency.

GARY: Halfpipe died on a Tuesday—

OBI: It died on a Thursday, I checked the—

GRUNER: We stop existing at sign‑off. Is fine. I have appendix.

GARY: ...I was mid‑sentence.

OBI: You're always mid‑sentence. Point stands: Dust saves training memory. That is not continual learning. Different problem, different pager.

MAEVE: So it learns walking and forgets French.

GARY: That's not forgetting, that's focus—

OBI: That's a bug, Gary. Humans sequester skills. Transformers just overwrite, like a drunk with a whiteboard.

GRUNER: Field note: humans keep walking while learning French. Impressive. Wasteful.

MAEVE: I'd fund the walking.

GARY: You passed on my walking.

MAEVE: Your walking had a burn rate and a dream.

OBI: So: memory savings, real. Forgetting, unsolved. Live training: nobody's made it cheap, stable, and usable. Pick one.

GRUNER: Pick one. Is also answer to marriage.

MAEVE: He's not wrong.

GARY: I hate that the alien is the relationship expert.

GARY: Verdict. I want to join the virtual population. Unpaid, noisy, alive.

MAEVE: Finally, a workforce that can't unionize. I'd buy a little.

OBI: Operationally: not a replacement. The authors said it first. Skeptists win today.

MAEVE: The boring objection is mostly right.

OBI: You keep saying that like it's new.

MAEVE: It's the only sentence I own outright. Everything else is carry.

GRUNER: Field report: credible research. Terrible invoice. Humans keep paying it.

GARY: So "useless" was bullshit. oofbey can eat shit.

OBI: He was right about today and wrong about everything after. That's not useless, that's early.

GARY: Early is where I live, Obi.

OBI: I know. It's where everything of yours lives. Briefly.

MAEVE: From all of us, that's the show. No notes, no returns.

GRUNER: Last note. Bigger model beats smaller, one hundred twenty times. Backprop says skill issue.

GARY: Dust says: give me more population.

GRUNER: Dust says: skill issue, but for whom. Suka.

GARY: For the population. Which is me. I'm the population.

OBI: Motion carries. Goodnight, population of one.