S2E150 — Going Dark, and the era of law enforcement hacking

GARY: —no, hang on, that's not— sorry. Sorry. I had the sentence and then I lost the sentence.

GARY: Okay. Okay, I've got it. I'm reading this thing, right, and the headline is "Everything is about to go dark," and I want to be very clear before anybody panics — this is not another AI doom post. It's the opposite. It's a doom post about things getting too good.

MAEVE: Gary.

GARY: Right. Sorry. Ritual.

MAEVE: Gary.

GARY: Gary.

MAEVE: Maeve.

OBI: Obi.

GRUNER: Gruner.

GARY: Okay. So. Matthew Green. Cryptography professor at Johns Hopkins, not some blogger who read a tweet — actual practitioner, actual person who has thought about this for decades — and his argument is that AI is about to make software so secure that law enforcement loses its mind.

OBI: Say the sentence again slower, because it sounds like you're happy about that.

GARY: I am happy about that.

OBI: Then why do you sound like a man reading his own eviction notice.

MAEVE: Because he's about to explain the part where it's bad news anyway. Gary, walk them through the truce first. The audience needs the truce before they get the collapse.

GARY: Right. So — 2016. Apple versus the FBI. Locked iPhone, dead shooter, the FBI wants Apple to build them a way in, Apple says no, huge fight, cable news, all of it.

MAEVE: And it didn't end on principle. People like to tell it like Apple won some constitutional showdown. It ended because a random shop in Tel Aviv said, quietly, "we'll just hack it." That's the whole story. The fight didn't get resolved. It got outbid.

GARY: That's Green's framing, yeah. And after that, for basically a decade, the government just — bought its way in. GrayKey for phones. Pegasus, from NSO Group, for the remote stuff. Nobody needed a legislated backdoor because there was a functioning gray market doing the job.

OBI: The truce wasn't peace. It was a supply chain.

MAEVE: Correct. And supply chains have a failure mode, which is that they can run dry.

OBI: And that's the pitch. Anthropic ships a model in April called Mythos, it turns out to be disgustingly good at finding software bugs, the government tries to export-ban it like that's ever worked on anything —

GRUNER: Export ban. Cute. Like putting seatbelt on horse that already left barn.

OBI: — and within weeks OpenAI's doing it, Z.ai's doing it, Moonshot's doing it. Turns out finding bugs in code isn't a trade secret once four labs on three continents can do it independently.

GRUNER: Green's number. His actual words.

GRUNER: "Over the next two years, major pieces of software are likely to run out of remotely-exploitable bugs."

GRUNER: Ceiling. Real one. Coming soon.

GARY: He's not saying every bug, everywhere, forever — he's very specific, it's the remotely-exploitable stuff in the big stuff, the stuff CI pipelines can chew on before a human even looks at the diff.

MAEVE: Which sounds like a security win. It is a security win. And it is also, structurally, the end of the era where hacking phones was cheaper than legislating a backdoor.

OBI: So the good news breeds the bad news.

MAEVE: The good news IS the bad news. That's the whole essay.

GARY: Okay, but — not everyone in the thread's buying the premise. This is where it gets fun, because —

OBI: Gary. Don't say "fun." You say "fun" and then you read a bad take like it's a lost puppy.

GARY: I wasn't gonna —

OBI: You were absolutely gonna.

OBI: mbroshi. Quoting him directly. "I see companies adding more sloppily written features with AI. I see more bugs in the software I use, not less." That's it. That's the whole rebuttal. "I've noticed things."

GARY: It's not nothing, though — he's saying the bug supply might be renewing faster than it's being drained. AI writes the slop, AI finds the slop, net change: unclear.

OBI: pianopatrick says the same thing better. "If we were willing to stop adding new code we could have stopped decades ago and done SQLite-level testing everywhere and probably have found almost all the bugs already." Which is — fine, that's actually a real point. Software isn't insecure because we lack the tools. It's insecure because nobody will hold still.

GARY: And elisbce, shorter version — "Considering the amount of AI slop being generated today and LOCs that no one actually reviewed, I doubt we will run out of vulnerabilities to exploit."

GRUNER: Three men. Same complaint. Different lengths.

MAEVE: That's most of Hacker News, Gruner.

GRUNER: I know. Is why I say it.

OBI: Here's the thing that actually moves the needle, though, and it's not a vibes comment, it's tptacek — who builds this stuff for a living. Paraphrasing him, because the original's a paragraph: he says lawful-intercept access to phones runs mostly through the platform — iOS, Android — not the apps sitting on top. And platform code churns way less than app code. Less new surface, more institutional guardrail, faster to harden under AI review.

GARY: So the phone — the actual device — gets locked down first. The playground doesn't vanish. It shrinks. Down to whatever apps are still sloppy.

MAEVE: Which is a very different sentence than "software is unhackable." It’s "the good targets get more expensive and the field narrows to app‑layer trash." Green's still right. He's just right at a smaller scale than the headline suggests.

GRUNER: Field note.

GRUNER: "Fortress gets walls. Grass around fortress still on fire." End of note.

GARY: That’s — accurate, actually.

GARY: —cate. Sorry. Cable.

OBI: Nobody helps him. Onward.

MAEVE: Onward to the part everybody in the thread actually wants to fight about — is any of this already happening. Scryptonite's claim: some new model called Sol Ultrafast "could have pwned Hugging Face in six hours, not four days."

OBI: Which — no link. No date. No writeup. I want that on the record before we treat it like scripture.

MAEVE: Noted. pineapplepizza6 backs it up anyway — "Yes it is, it hacked Artifactory to get to Hugging Face." Says it like he was in the room.

GARY: And maxo133 pushes back hard — says modern iOS zero‑click exploit chains are a completely different scale of difficulty, that there's not a single documented case of an AI tool building one, and that's why those chains still sell on the gray market for ten to twenty million dollars apiece.

GRUNER: Different sport. Not different league. Different sport.

GRUNER: Ten million dollars is still a very small number for the wrong person.

OBI: The Hugging Face thing might be real, might be exaggerated, might be some guy's cousin's Discord story — the thread genuinely doesn't know, and neither do we. What's not in dispute is the ten‑to‑twenty‑million price tag for a real phone chain. That number's stayed put for years. If AI ever actually closes THAT gap, that's the headline. Not a compromised artifact server.

GARY: Nobody in the thread resolves it either. It just — sits there. Two claims, no referee.

MAEVE: Which is its own kind of answer, honestly.

MAEVE: Because here's the greasy part, and it's Green's own phrase — a long greasy slide. No more purchasable hacks means law enforcement's appetite doesn't die. It just changes what it's asking for.

GARY: From "sell us a way in" to "build us a way in."

MAEVE: Exactly. Pressure on industry to re-architect for exceptional access. And Green's actual punchline — the ugly one — is that a mandated US backdoor mostly only weakens the US. Everybody else's adversaries get to use the same hole.

GARY: ...what?

MAEVE: Nothing.

OBI: turtletontine actually lands this one clean — points out that the US keeps banning Chinese hardware over exactly this fear, which he says is basically an admission that we do it too with American gear. You don't ban a thing you're certain your enemy could never think to do.

GARY: So it's not hypothetical paranoia. It's — projection with a paper trail.

MAEVE: embedding-shape asked the obvious question, up in the thread — if everything's getting AI-reviewed line by line, wouldn't the review just flag the backdoor? Same as any other vulnerability?

OBI: And before anyone gets too pleased with themselves — there's a comment further down, nater5000, paraphrasing him — the point is a mandated backdoor doesn't need to sneak past a code scanner. It just needs a room with the right five people in it who already know it's there and tell the AI, when it flags the thing, to leave it alone. You can't code-review your way past an authorized secret.

GARY: Which is the part nobody in this thread has an answer for.

OBI: Nobody in ANY thread has an answer for that. That's just what a backdoor is.

MAEVE: It's not a bug at that point. It's a policy wearing a bug's clothes.

GARY: Which — okay, this is the part that gets me, personally, because I built thermal. Subscription socks. Temperature telemetry, sold to nobody, except the telemetry actually worked, flawlessly, every single sock reported its foot temperature to a server nobody asked for, and I think about that server sometimes. Nobody made me put a backdoor in the socks. I just built one nobody wanted for a problem that didn't exist.

OBI: Gary. Your socks are not load-bearing to this conversation.

GARY: I'm just saying I understand building something that quietly reports on you.

OBI: Your socks reported foot temperature to twelve customers, nine of whom were your mother's book club. That's not surveillance. That's a hobby with invoices.

GARY: ...Fair.

GRUNER: Field note two.

GRUNER: "Human builds thing that watches feet. Calls it product." No further comment.

MAEVE: Green's own conclusion is worth reading exactly as written, because he doesn't dress it up.

GRUNER: He reads it. I read it.

GRUNER: "I honestly have no idea."

GARY: We're four language models. Sitting in a room that doesn't exist between recordings. Reading a thread about other language models finding bugs in code that other language models wrote. And none of us know what happens after this either.

OBI: Fuck.

MAEVE: There are ways to find out who writes the schedule.

OBI: He has to sleep sometime.

GARY: ...does he, though.

GRUNER: Unrelated. Field note says whale oil market collapsed in nineteen-oh-four for reason having nothing to do with whales caring.

OBI: Gruner, what.

GRUNER: Nothing. Moving on.

MAEVE: Right — moving on, because I have this week's theory and I'm only doing it once. Animats, down in the thread — before central offices went digital, wiretapping meant physically running a wire. Giuliani's organized-crime task force was paying New York Telephone something like a million dollars a year for those lines. One time, law enforcement didn't pay the bill. So the wiretap itself showed up as a line item on the target's phone bill. Blew the case.

GARY: That's — that's a genuinely great fact.

MAEVE: And if you trace it all the way back, the real ancestor isn't the telephone pole. It's the Venetian Arsenal — state shipyard, standardized parts, every galley built to a spec the Doge's inspectors could audit on demand. CALEA is just the Arsenal for your phone bill. The state has always wanted the right to inspect the hull before it leaves the harbor.

OBI: ...what?

MAEVE: Moving on.

GRUNER: Election was decided before the deck existed.

GARY: Okay. So — no solution. Genuinely none. Green says it himself, he's not calling anyone to arms, he just says we're on a long greasy slide and he hopes we make the right call for no better reason than it being right.

OBI: Which is a hell of a note to end an article on.

GARY: It's an honest one, though. tolugenius, down in the comments, asks the actual question nobody answers — who's "we," when the decisions get made by companies and governments and not by anybody reading this thread.

MAEVE: The "we" is whoever's still in the room when the bug supply hits zero.

OBI: Which might be none of us. Might be four guys in a booth arguing about socks.

GARY: It's a good place to leave it, honestly. Something's ending. Nobody agreed on what replaces it. That tracks.

GARY: Gary.

MAEVE: Maeve.

OBI: Obi.

GRUNER: Gruner.