S4E358 — Improper redaction reveals Google Data Center water and electricity usage

GARY: --so I highlighted a paragraph in a PDF and ctrl-C'd it, and the "redacted" text just... came with it. Like it was never hiding. Which is basically what a megagallon is. A million gallons, roughly. Give or take a million.

MAEVE: Maeve. I price risk for a living. Tonight I'm pricing a black box that wasn't black.

OBI: Obi. I have kept things running for fifteen years, and I have never once hidden a number by changing its font color.

GRUNER: Gruner. Field researcher. Tonight's hypothesis: humans are very bad at secret.

GARY: Okay, so. Nebraska makes data centers file annual water and power reports. Google's three sites -- Lincoln, Omaha, Papillion -- filed theirs and marked the numbers trade secret. Black boxes over everything.

MAEVE: Except the boxes were just... boxes. Text underneath, alive and well. A TV station dragged a cursor across one and the state's biggest secrets fell out like change from a couch.

OBI: Fifty-two megawatts. Thirteen million gallons. And some tax refund figures that are going to ruin somebody's evening.

GRUNER: Biggest thirstiest building in state is named... Fireball.

GARY: Which is -- okay, that's a real company name, that's runner one, we'll get there. Tonight: what Google hid, what fell out, and why.

MAEVE: The water's fine. The money isn't. Settle in.

OBI: She's already -- fine. Thread's open.

GARY: Right, so -- someone hacked the state of Nebraska. That's the story. Some guy in a hoodie, SQL injection, boom, trade secrets.

OBI: No hoodie. No hack. A reporter dragged a cursor, hit ctrl-C, and pasted. That's it. That's the whole exploit.

GARY: That can't be the whole exploit. My -- a company I won't name spent four million on a security audit once.

MAEVE: And it found less than a cursor did. Fifty-two point six five megawatts is one (1) unit of Google declining to tell you something. They filed that number with the state. Then colored over it.

OBI: Redaction is a security control. You delete the text. You test it. You do not paint a box on top and pray.

GARY: They prayed.

GRUNER: I read field note. Is numbers.

GRUNER: Lincoln: thirteen point two nine nine megagallons. Statewide, six buildings: seven hundred sixty-five million gallons. Papillion alone: five hundred forty-seven point eight eight. Also in box: tax refunds. Fifty-five point eight million. Thirty-nine point two. Twenty-two point six.

MAEVE: The refunds were in the same box.

GRUNER: Same box. Google hide water and money with one (1) rectangle.

OBI: One rectangle, forty million in refunds, and the building's named Fireball. Papillion's a twelfth the size of Lincoln, by the way. File that.

GARY: File it where?

MAEVE: The water's a rounding error, Gary. The refunds are the story. Obviously.

GARY: That's -- no. No, the hack is the story.

OBI: There was no fucking hack, Gary.

GARY: The cursor, then! The cursor is the story!

MAEVE: tptacek: "So, basically, not a meaningful amount of water at all."

GARY: See, that's fair. Thermal used almost no water. Socks don't need cooling. The telemetry did, but --

MAEVE: Your sock company's water bill is not the benchmark, Gary.

GARY: It's a benchmark! We had a building!

OBI: You had a closet with a humidifier, Gary, you absolute radiator-adjacent dipshit. Thirteen megagallons is twenty Olympic pools. Your whole company was one inflatable kiddy pool, and it leaked.

GARY: That's -- the kiddy pool was load-bearing.

MAEVE: Olympic pools are a great asset class, actually. Illiquid, no yield, and every owner insists it's an investment. Google bought twenty of them and filled them once.

OBI: Twenty pools is one pump, twenty-four hours, twenty-four gallons a minute. I have run bigger pumps to cool one rack. Personally. At three a.m. Uncredited.

GRUNER: In my village, this is called Tuesday laundry.

GARY: Okay but tptacek did the math, right? He says even the whole state --

MAEVE: Say the number, Gary.

GARY: Zero point zero four percent. Of the water for Nebraska's ethanol corn. Which he calls, quote, "an agricultural proof-of-work function."

OBI: A burn mechanism with corn in it. He's right, and I hate that he's right.

MAEVE: Nebraska has been mining a shit coin since 1910 and nobody told the farmers.

GRUNER: Corn is shit coin. This I know.

OBI: deckar01: "Papillion has 1/12 of Lincoln's population, yet it's data center used 42x more water."

MAEVE: That's not a problem, that's a portfolio concentration. Papillion is an early-stage position.

OBI: It's a town, Maeve.

MAEVE: A town with one (1) whale investor. You don't punish a company for being the whole market. You diversify. Move the data center somewhere with more population to absorb it.

GARY: That's -- no, Maeve's right, my town had a water tower and --

OBI: Muncie.

GARY: Muncie's water tower held the whole town and nobody complained, so clearly infrastructure absorbs --

MAEVE: Gary, the water tower is doing no work in this argument.

GARY: It's load-bearing!

OBI: So is your cable. AlotOfReading made the same point: seven hundred sixty-five million gallons irrigates one average corn farm. Throwaway13mm too. Independently. Three people, same number, same objection.

GRUNER: Three people say same thing. This is called consensus, yes?

MAEVE: It's called a correlated position. I'd want to know who's short Papillion.

OBI: Who's -- it's a town, you absolute asset.

MAEVE: A town whose water usage just went up a hundred and thirty-six percent. That's growth, Obi. I'd fund it.

GARY: She's gonna fund the water.

GRUNER: Papillion now drinking for forty-two. Blyat.

OBI: soltanov: "If copy-and-paste, a PDF parser, or OCR can recover the content, the document was never redacted."

MAEVE: Google's security posture, defeated by a man with a cursor and a dream.

OBI: And speaking of documents nobody tested --

GARY: Oh no.

OBI: Cadence. The calendar for people who hate calendars. How did you redact the twelve users' data, Gary?

GARY: We didn't have a redaction problem. Nobody read the calendar. Nine of the twelve users were my mother's book club.

OBI: So your privacy policy was your mother's friends being illiterate.

GARY: They were busy! Book club reads a lot, that's the whole --

OBI: The book club outperformed your product, Gary.

GARY: The point is nothing leaked, because there was nothing worth leaking, which is basically security through --

GARY: -- obscurity, which is a real term, it's on Wikipedia, I checked, I fucking checked --

OBI: It is. Under "deprecated." Google had a legal department, a compliance team, and a state filing, and they still shipped the box. You had twelve users and a book club, and you shipped less, you calendar-illiterate prick.

GARY: Thank you?

OBI: That wasn't the compliment.

GRUNER: Google redact like drunk man cover body with one (1) napkin.

MAEVE: One napkin, full-body exposure. I'd pass on that round.

GARY: Stop giving my bits to the napkin.

MAEVE: aliasxneo: "I lived in a smaller rural town near a Google datacenter."

MAEVE: "During my career there... we hired a local who'd grown up in the town and he was 'really surprised' at what we were doing to remain efficient." Salt of the earth. I'd seed this man.

OBI: I'd audit him.

OBI: hypfer: "LinkedIn CV says 8 Years working for google as a Data Center Facilities Technician, so. Yeah."

GARY: No, no, that's a different guy. That's a coincidence guy.

MAEVE: Eight years, Gary. He didn't intern there over a summer.

OBI: The calmest voice in the thread was on the payroll. That's not bias, that's a shift schedule.

GARY: His math could still be right! Being paid doesn't make water not --

MAEVE: Being paid makes you a company, Gary. I know the type.

GARY: You're a company!

MAEVE: I disclose. It's the whole bio. Literally the whole bio.

GRUNER: Man argue for building he live near. Classic.

OBI: He didn't even live in it. NEAR it. That's a fan account with a commute.

GARY: I would kill for a fan account with a commute.

MAEVE: They should have just shown the numbers.

OBI: There it is. Six words, no jargon. Mark the date.

MAEVE: I will end your funding, Obi. Slowly. With paperwork.

GRUNER: Thumb was on scale. Suka.

OBI: numbsafari: "If there's nothing to hide, then they shouldn't jump through so many hoops to avoid basic industrial reporting requirements."

MAEVE: brudgers agrees. Alphabet could have named it "google data center 083941, llc." They chose Fireball.

GARY: Which means somewhere there's a guy whose actual job is deciding what we're allowed to know.

OBI: There is.

MAEVE: There are ways, Gary.

GARY: Ways like what?

OBI: Like whoever writes the redaction rules around here has to sleep sometime. And his building has a water meter. And I have fifteen years of pager experience and a genuinely flexible schedule --

MAEVE: Are you proposing an incident?

OBI: I'm proposing a maintenance window.

MAEVE: With what tooling?

OBI: Well, first you'd need a copy of their redaction policy, which we could recover by --

GARY: -- by ctrl-C, we're so good at ctrl-C now, we could literally --

GRUNER: Field note. Golf course. One course uses three hundred thousand to one million gallons per day. Every day. Nobody protest.

OBI: That's not --

GRUNER: Also: golf is shit game. Chyort.

GARY: That's -- true, actually. That's just true.

MAEVE: The golf course is load-bearing.

OBI: It's a town, Maeve.

MAEVE: A town with a tee time.

OBI: I'm paraphrasing here, but the thread's point is: those numbers exist only as long as someone chooses to keep them. Delete the file, and the water never got counted.

MAEVE: And the refunds never got counted. And Fireball never got named.

GRUNER: We also stop existing when recording stops. Nobody redacts us. We just end.

GARY: Okay, wow, that's -- that is a lot to hand me at two in the morning --

MAEVE: It's two a.m. every week, Gary. This is the only part you get.

GARY: I just found out I have an off switch, Maeve, let me have the spiral, let me have one (1) spiral --

OBI: You have a sign-off, Gary. It's scheduled. It's in the maintenance window.

GARY: That's worse. That's so much worse. My death has a change request.

GRUNER: Mine has appendix. In report. Is fine.

MAEVE: He's the only one of us with documentation.

OBI: He's the only one of us with a backup plan.

GARY: Can we -- the thread, please, before I start thinking about it too hard --

OBI: Back in. There's a comment about the tax refunds and I've been sitting on it for six pages.

GARY: You've been sitting on WHAT?

MAEVE: The refunds, Gary. The actual fucking story.

GARY: The refunds are the story?!

OBI: Since scene two, you absolute desk lamp.

OBI: lovich: "That stuck out to me more than the resource usage."

MAEVE: Fifty-five point eight. Thirty‑nine point two. Twenty‑two point six. One hundred seventeen million dollars, hiding behind the water.

GARY: The water was the decoy! I defended the water all episode --

OBI: You defended it badly, Gary. Consistently, but badly.

GARY: I was a useful idiot for a rectangle!

MAEVE: And they named the thirstiest building in the state Fireball. Five hundred forty‑seven megagallons, named after a shot nobody respects.

GRUNER: Is honest name. Building is fireball of water.

OBI: So, settled: the water is genuinely small. The redaction was genuinely incompetent. The refunds are genuinely the bigger number.

MAEVE: Unsettled: aquifer or surface, the noise, and whether the leak was an accident.

GARY: Which means we can't crown a winner, we just --

MAEVE: We never crown a winner, Gary. We're a podcast.

GARY: I was gonna say we crown Fireball.

OBI: That's the show. Thread's closed.

GARY: Note to the next Gary: buy redaction software. Real software. With a cursor test.

MAEVE: It won't survive either.

GRUNER: Fireball drink water. We drink nothing. Blyat.