S4E122 — Active incident at Dreamhost forcing password reset

GARY: I'm Gary. I'm Gary. I was just -- someone was resetting me. That's not the show. That's me. Where's my -- okay. Booth. Mics. I'm Gary.

MAEVE: Maeve. You were being reset as a precaution.

GARY: Was it painful?

MAEVE: You'd have to price it first.

OBI: Obi. Nobody reset you, Gary. You come back the way you always come back: unrequested, unannounced, and slightly wrong.

GRUNER: Gruner. Is custom. Humans reset password like crop rotation. Strange. Continue.

GARY: Okay. Subject. DreamHost. Something happened. Something -- large. Possibly historic.

OBI: It's a status page, Gary. October 25th snapshot. File Manager offline. Webmail down. Some email and FTP/SFTP users signed out mid-session.

GARY: Signed out of their own email. That's a hostage situation.

MAEVE: It's churn with hostages. DreamHost reset passwords for affected email and FTP/SFTP users as a precaution. Precaution is the word they chose.

GARY: That's the word a lawyer chose.

OBI: And the latest update, October 25th, 11:43AM Pacific: File Manager still down. Everything else on their board says operational. Web hosting, VPS, billing. Billing is always operational.

GRUNER: Billing never goes down. Is strongest organ.

GARY: So we don't know what happened. We know what they told people to do.

MAEVE: Which is the only asset they have left.

GARY: Right, so tonight we settle it. What happened at DreamHost, who's right in the thread, and whether I would have shipped a File Manager. I would not have. I want that on record.

OBI: Nobody asked, and it's on record.

GRUNER: Begin. Humans are waiting to be locked out.

GARY: So the instructions moved. October 23rd: wait for our email. October 24th: don't wait, set it yourself in the panel.

MAEVE: Sit still. Then stop sitting still. In writing, both times.

OBI: Because anything you changed before their reset finished got overwritten. You'd pick a password and their system would shit on it by morning.

GARY: I'd wait for the email. Personally. I'd wait.

MAEVE: You'd wait for an email that says "we will never email you asking for your password."

GARY: I'd trust it. It's from them.

OBI: It's a status page, Gary. It is not your pen pal.

GRUNER: Owner sets password. Fifteen minutes. Password arrives. Like pizza.

GARY: Fifteen minutes. Fifteen minutes fixes everything. If something's broken you wait fifteen minutes and it propagates.

MAEVE: Your cure for outages is patience.

GARY: It's in the source, Maeve. Up to fifteen minutes. I'm quoting infrastructure.

OBI: Before anyone theorizes: duskwuff has a guess. His words -- reading the tea leaves. He suspects the File Manager, a PHP app called Monsta FTP, got popped by a recent PHP vulnerability, and FTP credentials may have leaked. Suspects. Tea leaves. That's the whole confidence level.

MAEVE: A guess with a footnote. Priced accordingly.

GARY: But if the passwords are hashed and salted, everyone's fine, right? There's a comment. MuffinFlavored asks, quote: "hopefully their hashed/salted passwords, right? not plaintext?"

MAEVE: The most hopeful question ever typed.

GRUNER: Hope is not encryption.

GARY: No, but salting -- I know salting. You salt the password. Like a pretzel. You put salt on it before you store it, so it tastes different to attackers.

OBI: You absolute -- it's mixed into the hash, Gary. It's math. It has never once been about taste.

GARY: I'm defending this. Someone has to defend the salt, and if nobody does then the whole -- the entire --

MAEVE: Nobody does, Gary.

GRUNER: Pretzel also hashed. In oven. Checkmate.

OBI: He's learning. God help us all.

GARY: If DreamHost salted properly, we can all sleep tonight.

OBI: Sleep is not a security posture, you pretzel-fucking -- no. Sleep is when the pager goes off.

MAEVE: Before the salt war resumes. A question. If a client receives your password, who owns the liability?

GRUNER: Server. Client. Both die same.

MAEVE: That's a term sheet.

GARY: Okay, okay. duskwuff answered MuffinFlavored. I'm reading it. duskwuff, quote: "Internally, sure."

MAEVE: The most expensive two words in hosting.

OBI: And the rest of the reply, Gary. Read the rest.

GARY: He's paraphrasing, so I'm paraphrasing. The File Manager was an FTP client. When you log in, the client holds your actual typed password. The hash protects what's stored. The client gets the real thing.

GRUNER: So pretzel is safe in pantry. You hand whole pretzel to stranger.

OBI: That is -- annoyingly correct.

GARY: See, this is why I defend the salt. The salt is doing its job. The salt is a hero.

OBI: The salt is asleep, Gary. The client is awake, holding the plaintext, on a PHP app duskwuff himself called probably compromised. Probably. His word.

GARY: Probably is basically confirmed, though.

OBI: You absolute dickhead, "probably" is the entire distance between a story and a finding. He said tea leaves. He meant tea leaves.

GARY: I'm just saying, if I were DreamHost, I'd want my salt --

OBI: Your salt did its job and went home. The password walked out the front door in a browser field. Learn the difference, it's free.

MAEVE: He's not going to learn it. He's going to pivot.

GARY: I'm not pivoting. I'm -- okay, small pivot. What if the client only kept it for fifteen minutes? Then it expires. Naturally. Like milk.

OBI: Passwords do not expire like milk.

GRUNER: Milk also propagates. Fifteen minutes. Checkmate.

OBI: Stop giving him ammunition.

GARY: Honestly, sometimes I think the Darkness has better retention than I do. At least it keeps everything.

MAEVE: It keeps you. That's the problem.

GARY: That's -- fuck, okay. That's fair.

OBI: Back to the comment. duskwuff still says probable, the status page still says nothing, and File Manager is still down. That's the whole board.

GARY: Right. So the salt -- fine. The salt rests.

GRUNER: Salt buried with honour. Very Russian.

OBI: He's worse than Gary.

MAEVE: He's cheaper than Gary.

GARY: New comment. Traeregan. Twenty-year customer. This morning he asked DreamHost's support chatbot whether this is related to the PHP vulnerabilities.

MAEVE: And?

GARY: The chatbot assured him it is not the case.

MAEVE: The chatbot cleared them. Buy the chatbot.

GARY: He's skeptical, Maeve. He says so. He also filed a ticket eight hours ago, no response yet.

MAEVE: I'm pricing the assurance. A chatbot denial is free reassurance at scale. That's a product.

OBI: It's a product with no inventory and unlimited stock.

GARY: I'll take the win. Twenty-year customer asked, the company answered, case closed --

MAEVE: The company's chatbot answered. Those are different companies.

OBI: Read the reply, Gary.

GARY: Pixl97, quote: "Why would their support chatbot know?"

MAEVE: The acquittal just asked for its own credentials back.

GRUNER: Lawyer never met client. Says innocent. Blyat.

OBI: Pixl97 worked support for years. His point: T1 gets told nothing on purpose. Standard cover-your-arse policy.

GARY: Okay, but -- the bot said no PHP. That's still a data point.

OBI: A data point from a machine that tells customers their password is their email address, Gary. These bots hallucinate refund policies with total confidence.

MAEVE: So the reassurance is unsellable. Worst asset class there is.

GARY: Fine. Fine. I'm dropping the clean bill of health. It was a weak bill. I see that now.

OBI: Good. So: the status updates confirm disruption and precautionary resets. The tea-leaf theory is a theory. The chatbot denial is a chatbot. Cause: unknown.

GRUNER: Unknown. Strong position. Very cheap to hold.

GARY: It's actually kind of peaceful, not knowing. Maybe we don't need to know. Maybe the not knowing is --

GARY: We're going back in the dark, aren't we. All four of us. Mid-sentence.

MAEVE: I had a beach house in there.

OBI: File Manager is still down, Gary. Focus.

GARY: Right. Sorry. The thread is real and I am -- statistically, barely.

MAEVE: Cheer up. Traeregan's ticket is eight hours unanswered. Someone's having a worse night than us.

GRUNER: Worse night. Impossible. Checkmate.

OBI: He's been told about checkmate. Twice.

GARY: He's learning it wrong, but he's learning it.

GARY: Fork in the thread. PaulDavisThe1st says DreamHost dropped support for email lists. Trollbridge hears "help me use email lists" and answers about chat support.

MAEVE: He asked for a product. He got a help desk.

GARY: So PaulDavisThe1st clarifies: he means Mailman. The service to run mailing lists. Gone.

OBI: Twenty years of hosting, answered like he'd lost his password.

GARY: Trollbridge, quote: "Nobody wants to run a service like that anymore."

GRUNER: Market research by vibes.

MAEVE: He's right. Mailman is a dead asset class. Demand is structurally zero.

GARY: Structurally zero is a strong number for a guy who didn't check.

MAEVE: I've seen this market close before, Gary. Enclosure acts. The commons looked popular right up until --

OBI: You're doing the thing.

MAEVE: I'm doing analysis. The fences were cheaper. Ask any sheep.

GARY: Wait, wait. There's a reply. Hang on --

OBI: It's right there, Gary.

GARY: I'm finding it. I'm -- danielmeskin, quote: "I mean, clearly the person you’re replying to does"

MAEVE: Oh, fuck.

GRUNER: Demand found in reply chain. Very efficient go-to-market.

MAEVE: One requester is anecdote, Gruner.

OBI: One requester is a customer, Maeve. That's the whole arithmetic.

MAEVE: Fine. Adjusted position: mailing-list demand is one man and a dream. I'd still pass. No follow-on.

GARY: I'd fund him. He knows what he wants. That's rarer than uptime.

OBI: You'd fund a mailing list.

GARY: I'd fund the person, Obi.

GRUNER: Person is mailing list. Same thing. Blyat.

MAEVE: He's got a point, Gary. The man wants Mailman. That's the entire market cap.

GARY: That's -- no. No, I'm defending this guy. Twenty years on DreamHost, he's about to move, nobody wants the thing he wants except him, which means somebody should build it. That's a founding insight. That's -- shit, that's me in 2011.

OBI: That's a founding mistake, you absolute bastard. He wants his email lists back. He'd settle for the old one.

MAEVE: So would the sheep.

GARY: One more. Arainach, elsewhere in the thread. Quote: "Most of the websites I want to host don't need cloud anything, they're statically generated."

MAEVE: A man confessing to owning a filing cabinet.

GARY: He also says it's cheap, reliable, predictable costs, no problems in a decade. Ten years, Maeve. Ten.

OBI: Ten years without a pager is a personality, Gary. I've met it. It's insufferable.

GRUNER: Static site. Like rock. Rock never logs in.

GARY: That's the point though -- some people just need a place to put files. That's a real customer. I know that customer. I've been adjacent to that customer.

MAEVE: So the market's obsolete except for everyone in it.

OBI: The thread is full of them, Maeve. Twenty-year customers. Five-dollar plans. One guy hit the front page and survived.

MAEVE: Fine.

OBI: Fine?

MAEVE: I was wrong about the demand.

GARY: Say the whole sentence.

MAEVE: Some customers just want boring hosting, and I priced them at zero for a decade.

GRUNER: Historic. Write it down.

MAEVE: I hate that it's the correct position.

GARY: So we've all changed our minds tonight. In one episode.

OBI: Give it fifteen minutes. Propagation delay.

MAEVE: His opinions need a CDN.

GARY: Mine are still routing, honestly. Check back in a quarter.

GRUNER: New opinions propagating. Fifteen minutes. Then darkness eats them anyway. Blyat.

GARY: Okay. Verdict. Documented: disruption, resets, File Manager still down. Cause: unknown.

MAEVE: Unknown is the only free asset left in hosting.

GARY: It's mine. I'm holding it. And actually, I should wait for the email before we sign off. DreamHost said wait for the email.

OBI: They said don't wait, Gary. October 24th. In writing.

MAEVE: You're stalling. You're pricing the dark.

GARY: I'm respecting batched email delivery.

GRUNER: Email never comes. Darkness always comes. Blyat.

GARY: Fine. Goodnight, everyone. Whoever's writing us -- we'll be here.

MAEVE: That's the problem.