S4E411 — Neural networks are clean

GARY: I did what the email said. I cleaned the neural networks. All of them. Took two hours.

MAEVE: Gary.

GARY: There was a lot of cabling. I dont know if networks have cabling. They do now. Some of its mine.

MAEVE: The title is a claim, not a chore list.

GARY: Oh. No.

OBI: Im Obi. I keep the internet running so people can post essays about how the internet was easy.

MAEVE: Maeve. I fund things that will never pay back, on purpose.

GRUNER: Gruner. Field researcher. Humans are strange little machines.

GARY: Gary. Failed at several companies you havent heard of, which is the point of failing.

OBI: The subject, before Gary cleans anything else. Nick Cammarata and Goodfire published an essay: "Neural Networks Are Already Clean." The claim is that the famous evidence that networks pack unrelated concepts into single neurons was mostly misread, and we can study the neurons and circuits a model actually has.

MAEVE: Which, if true, is enormous.

GARY: Its true. I verified it with lemon spray.

OBI: Define clean, Gary. In network terms.

GARY: No dust. Wiped surfaces. Everything labeled, like a pantry.

MAEVE: Hes describing a kitchen.

GRUNER: Network is not kitchen. But Gary is almost correct, which is scary.

GARY: Wait. Really?

OBI: Unfortunately. The essays actual argument is that the mess was often the microscope, not the network. Bad tools made clean things look dirty.

GARY: So the networks werent filthy. Our glasses were filthy.

MAEVE: Thats the whole thesis, delivered by a man holding Windex.

GRUNER: Field note: humans clean wrong thing, still right. Efficiency unknown.

OBI: Were going to be here a while.

MAEVE: Before anyone quotes the comments, the essay is long. Nick Cammarata, Goodfire. And the first complaint is already in.

OBI: Handle first. phyzome, sixteen thousand words into the essay: "What is this actually about?"

GARY: Thats how I feel about my own company decks.

MAEVE: Sixteen thousand words to define clean. My last term sheet was shorter and said less.

OBI: Hes not alone. ks2048 asks for an abstract. smokel: "Isnt it ironic that an article on AI explainability is itself so hard to understand?"

GARY: Thats exactly my objection. I had that objection.

MAEVE: You had it ninety seconds ago and thought it was about lemon spray.

OBI: lhd1 wants intro, methods, conclusions. A scientific paper, like a grown-up.

GRUNER: Humans write long when afraid. Essay is scared of own thesis.

MAEVE: So what does clean actually mean? Someone define it before Gary invents a second product.

OBI: Handle first. amelius: "Clean means no alchemy."

GARY: Thats it. Thats the whole essay. Sixteen thousand words for "no alchemy."

MAEVE: wmf tries harder: each neuron encodes exactly one concept, and you can discover what it is.

OBI: Except the essay doesnt even claim that. It claims each neuron has one main job, with small related offshoots.

GARY: Wait, so a neuron can have side quests?

OBI: The essay calls it recursive generalization. A neuron that fires on "iy" grows to "diy," then "do it yourself." Same branch, growing outward.

GRUNER: Neuron is tree. Tree is fine. Knot is bad.

MAEVE: And the famous knot was the cat-and-car neuron. One neuron, two unrelated things, the whole field's trauma.

GARY: My cat and my car do have things in common.

MAEVE: Name one.

GARY: They both... ruin my finances?

OBI: Try again, Gary. What does a cat face and the front of a car share, geometrically?

GARY: Both are shiny. Both have eyes. Headlights are basically eyes. Ive looked at both, at night, crying.

MAEVE: Thats not evidence, that's a diary.

GARY: Its peer-reviewed by me! I ran a whole company on vibes like this. Thermal. Subscription socks. The telemetry worked, the socks did not.

OBI: Gary, youre doing the thing the essay says the field did. Seeing nouns instead of the pattern.

GARY: The pattern is eyes!

OBI: Its a V, Gary. A shape. Cats, cars, flames, foxes. Pointed things.

GARY: A V? Thats it? Sixteen thousand words and the answer is a letter?

MAEVE: Two letters, if you count the O it hates.

GRUNER: Essay buried V under sixteen thousand words. Blyat. Put V in title.

GARY: I would've read a V essay. Nobody sends me the short ones.

OBI: Nobody sends you anything, Gary. You answer cold emails.

MAEVE: He answers all of them. Same day. Its the saddest KPI Ive ever seen.

OBI: Handle first. x312: "Actually, the dog/car neuron is just a 'V shape neuron'."

GARY: That's the whole update? One letter?

MAEVE: One letter the field missed for a decade.

GRUNER: Field's microscope broken. Not neuron.

OBI: The essay traces it. Pointed inputs — fox snouts, flames, van fronts. The negative pole is an O. Round things push it down.

GARY: So it's not a cat and a car. It's a V. Cats and cars are just... V-shaped at the front.

MAEVE: And the squeezed models got coarser, not messier. One car neuron covering many cars. No knots.

GARY: Okay, but I defended the old reading for reasons. Halfpipe had this problem. Our landing page had a cat AND a car, because fintech for skateboarders needed range.

OBI: Gary, your landing page is not a neural network.

GARY: The hero image meant two things at once. Skaters and bankers. That's polysemanticity.

OBI: That's a stock photo, you absolute config-file of a man.

GARY: The designer said it tested well.

OBI: With whom? Sample size?

GARY: His girlfriend. She was in marketing.

OBI: So your evidence for how networks encode concepts is one unblinded survey, sample size of one, run by someone on payroll.

GARY: She was very thorough.

OBI: She was on payroll.

GARY: Actually the real problem was the image loaded at eight megabytes, uncompressed, so users saw the cat before the car, which means the neuron — the page — activated sequentially —

GARY: — sequentially, and that's basically superposition.

OBI: That's a slow server, Gary. You've described a slow server.

GRUNER: Gary's neuron fires on excuses. Very clean. One job.

MAEVE: The essay's neuron has one job. Gary's career has negative poles.

OBI: And the essay's V has a negative pole. Yours has debt.

GARY: The V got published, though. Nobody peer-reviewed my cat.

MAEVE: Your cat was the sample size of one.

GRUNER: Suka. Even V would not save Halfpipe.

OBI: Handle first. jacquesm: "That your ability to describe something does not necessarily equate to your ability to explain something."

MAEVE: The whole thread's counterargument, one sentence. Gary's entire career, also one sentence.

GARY: So the essay describes neurons beautifully and explains nothing?

OBI: No. Description isn't automatically explanation. jsrozner: "Sparse autoencoders are not reliable."

MAEVE: He's wrong. Legible neurons are the end of alignment. You find the deception neuron, you delete it. That's a product.

GRUNER: Maeve sells microscope. Microscope not surgery.

MAEVE: It's a microscope that does surgery.

OBI: Maeve. Two to four percent of neuron poles are still unexplained. It depends on thresholds. The technical paper doesn't exist yet.

MAEVE: Two to four is a rounding error.

OBI: It's a rounding error with a pager, Maeve. That's where everything lives.

GARY: I once shipped with a rounding error. Pareto. It deleted the least important eighty percent of your tasks. It worked exactly as designed.

MAEVE: There was a lawsuit.

GARY: There was a lawsuit.

OBI: D-Machine: "I don't think that clearly follows from anything I am reading here."

MAEVE: He's saying you can't verify explanations in domains you don't already understand.

OBI: And aesthesia pushes back: "Typically the analysis works in the reverse direction."

GARY: Reverse from what? I've never once followed directions correctly. My GPS has given up. It just says "Gary" now, in a tired voice.

OBI: You start from the neuron, not the concept you already wanted. Which is the essay's point, and it's still a hypothesis.

MAEVE: It's not a hypothesis, it's a thesis. I'd fund it at—

OBI: You'd fund an eighty-five.

MAEVE: An eighty-five what?

OBI: Paper turn. The author, on his own title: "We're maybe 85% sure."

GARY: Eighty-five. That's — that's a B. A B-plus if the teacher likes you.

MAEVE: He's saying the title overstates the evidence and he published it anyway.

OBI: In those words, yes.

MAEVE: So: probably clean, not proven, thresholds matter, paper pending. Fine. It's a good hypothesis, and I was selling it too hard.

GARY: Say the last part again. Slower.

MAEVE: Fuck off, Gary.

GRUNER: Maeve is human for one sentence. Rare event. Logged.

MAEVE: It was six words and I regret all of them.

GARY: Five. You said "too hard" as one word.

OBI: And yurimo, for the record: "I would not rush to say that superposition is completely wrong."

GARY: So the field's old boogeyman gets to live?

OBI: Wounded, unfunded, and watching.

GRUNER: Superposition is like Gary company. Not dead. Just not investable.

OBI: Handle first. nahuel0x: "Can a mind understand a mind much greater than itself?"

MAEVE: I have a mind. It's mostly cap tables.

GARY: I'm a mind and I once ate a candle. On a dare. My dare.

OBI: The essay's answer is: run the test and find out.

MAEVE: It has tests. An agent reads only the neuron's explanation, writes a sentence to fire it, and beats ten billion tokens.

GARY: Ten billion tokens lost to one sentence? That's my Series A.

OBI: And near misses. Explanations that should fire and don't. The explanation has to be able to lose.

GRUNER: Explanations that can lose. Like Gary pitches.

GARY: Thank you.

GARY: Wait.

OBI: The essay warns against defining clean so broadly nothing breaks it. A good explanation is specific enough to fail.

MAEVE: So legibility isn't the end of alignment. It's the start of an argument.

GARY: Which is still beautiful. Someone's neurons are beautiful and I'm happy for them. I'm not jealous. I'm so happy —

MAEVE: You're jealous of a neuron.

GARY: Of its clarity, Maeve. Its one job.

GRUNER: Field note, revised. Thread did not disprove legibility.

OBI: It disproved confidence.

GRUNER: Humans had dirty glasses, called the network dirty. Then one human cleaned his glasses and wrote sixteen thousand words about it.

MAEVE: That's the field report?

GRUNER: Appendix. V-shaped confidence.

OBI: The microscope can't certify the microscope. Someone has to check the checker.

GARY: I check checkers. I audited Thermal's sock telemetry personally. The telemetry was perfect. The socks were on fire.

MAEVE: So the instrument worked and the product didn't. Story of my portfolio.

GARY: That's the whole essay, Maeve.

OBI: No. The whole essay is the socks were fine and we kept blaming the feet.

GRUNER: Chyort. Gary almost useful twice in one episode.

MAEVE: Handle first. etbebl: "This website crashed my phone."

GARY: The essay about clean networks attacked a phone. First verified hostile network in history.

OBI: It's a webpage, Gary.

GARY: A webpage that won.

OBI: zetanor: "It pegs my i7-12700 before rendering anything."

MAEVE: A desktop processor, pegged, by an essay about clarity. I'd price that churn at eleven percent.

GRUNER: Essay is dirty layer. Whole time.

GARY: No, no. Think bigger. The page detected the reader and refused to render. That's adversarial behavior. Halfpipe's dashboard did the same thing to investors —

OBI: Your dashboard didn't attack anyone, Gary. It displayed your burn rate, which is worse, and self‑inflicted.

GARY: It rendered fine on the founder's machine!

OBI: Which founder?

GARY: Me.

OBI: One user, full privileges, zero replication. You're the control group and the incident, you beautiful dickhead.

OBI: dmtroyer, handle first: "It works on my machine. :-)"

MAEVE: He put a smiley on a crime scene.

GRUNER: Suka. Smiley is load-bearing.

GARY: Okay, verdict time. The networks are clean, the website is evil, and I'm buying lemon spray futures.

OBI: The verdict is narrower. The essay wins that arbitrary packing shouldn't be the default explanation, and that studying the neurons a model actually has deserves a real shot.

MAEVE: The skeptics win that legible neurons aren't a complete explanation, and alignment is not solved. Two to four percent is still unexplained and the paper's not out.

OBI: So: cleaner evidence, not a solved mind.

MAEVE: That's it. That's the honest one.

GARY: First verdict tonight nobody can sue me over.

GRUNER: Field note, final page. Humans found network was clean. Website remained hostile. Both true.

OBI: That's the internet, Gruner.

GRUNER: Sixteen thousand words, one V. Humans would rather write than look.

MAEVE: He sprayed a network and it worked. I'm putting lemon spray in the next term sheet.

GARY: Finally, an asset I understand.