S4E383 — Can you use autoregressive diffusion to generate market data?

GARY: — and that's why I'm saying Jane Street has diffused tomorrow's stock prices. Directly. Into the future.

MAEVE: Gary. Read one paragraph.

GARY: I read the headline. The headline was extremely complete.

OBI: It says "market data," Gary. Not "prophecy."

GRUNER: This is good start for him.

GARY: I'm Gary. I've built things.

MAEVE: Maeve. I've funded things. Poorly chosen things.

OBI: Obi. I've kept things running while people like Gary built things.

GRUNER: Gruner. Field researcher. Normal human.

OBI: Subject. Jane Street intern, Kavish, built a diffusion model. It generates the next event in a market order book — kind, features, timing. Then it appends that event and does it again. Autoregressively.

GARY: So it predicts the price.

OBI: It generates an event. That is not a point forecast, that is a — Gary, it's a different verb.

GARY: It's the same verb wearing a lab coat.

MAEVE: He's doing the lab coat thing again.

GRUNER: Verb is wrong. But coat is correct.

GARY: Four years of US equities, Obi. That's a crystal ball with a Bloomberg subscription.

OBI: It's a model of what an order book does next. No.

GARY: That was very fast and very final.

OBI: Practice.

MAEVE: Also, what is "atom smoothing." It's in the article. Sounds like a product.

GARY: It IS a product. I tried to buy it for Grout. The tile marketplace. Six weeks, 2022. We don't discuss it.

MAEVE: You just discussed it.

GARY: Legally that was a different Gary.

OBI: Different Gary had the same credit card.

GARY: He had better taste, though.

GRUNER: Same Gary. Same coat.

MAEVE: First comment. reedf1: "No."

GARY: That's it? That's the whole comment?

MAEVE: reedf1 did more with two letters than you did with thirty-eight million.

OBI: Next reply. aamargulies: "Seconded."

GRUNER: Two humans. Six letters. Thread is settled.

GARY: No, no — reedf1 is right, and I'll tell you why. The market has modes. It reverts when it switches them. Regime change makes the whole project fraudulent, and I say that as someone who lived through regime change --

OBI: You lived through a pivot to NFTs for tile contractors.

GARY: That WAS the regime changing, Obi.

OBI: That was a regime collapsing, Gary. Different verb.

GARY: Same verb wearing a lab coat!

OBI: dzink, quote: "Thus happy bouncy becomes hammered stammered."

GARY: That's me. That's literally my operating model.

OBI: It is a description of your model's failure mode, and you have adopted it as a philosophy.

GARY: Because the prediction models fall hook and sinker for it! The market switches modes and the model just keeps bouncing along like -- like Thermal's socks!

MAEVE: The socks worked, Gary.

GARY: The telemetry worked, Maeve. The business was a crime against feet.

OBI: Count what actually happened. The first model, the DDPM, diverged. Exploding denoising trajectories.

GARY: Exploding trajectories sounds like momentum to me.

OBI: It is the model predicting values eight standard deviations outside reality. Eighty-eight to ninety-five percent of values, across all seven continuous targets.

GARY: Seven targets, eighty-eight percent -- okay, but at Halfpipe we called that a growth curve --

OBI: You called a regulator a growth curve?

GARY: He was very nice about it!

OBI: The model exploded, Gary. Pink filaments, straight to the edges. Your DDPM defense is a man defending his own car fire because the flames were moving fast.

MAEVE: He's going to defend the flames.

GARY: I'm defending the flames!

GRUNER: Flames do not need lawyer. Flames need fire department.

OBI: And the fix wasn't cleverness. He switched to flow matching and it worked out of the box. Out of the box, Gary. Do you know what that costs a founder?

GARY: Nothing, apparently --

OBI: Everything. You'd have raised a Series B to tune a noise schedule someone else already fixed.

MAEVE: Oh, fuck, that stings.

GARY: Fine. Fine. New operating model, effective immediately: happy bouncy becomes hammered stammered. It's on the whiteboard. It's the whole roadmap.

GRUNER: He is now quoting commenter at himself. Field notes full tonight.

MAEVE: stult: "There is no model of the market that can remain stably accurate."

GARY: He's right. The market eats the model. I've been eaten. It's a club.

OBI: tylerflick, replying: "AKA the efficient markets hypotheses."

MAEVE: Plural. He pluralized it. That's confidence.

OBI: zdc1: "Or, thankfully, for Jane Street: the (eventually) efficient market hypothesis."

GRUNER: Eventually. Word carrying entire economy.

OBI: teravor goes further: the model would have to get so rich its edge becomes insider information. Satellite photos of parking lots, weather forecasts. It's not the model, it's the data.

GARY: I tried to buy satellite parking‑lot data for Muncie. The salesman quoted me a price, then quoted the same price again, slower.

MAEVE: That's how you know it was expensive.

GARY: It was four hundred thousand dollars, Maeve.

MAEVE: That's how you know it was a scam.

OBI: majormajor, dissenting: "This assumes that any accurate model will inevitably get big enough to be noticable by the rest of the actors."

MAEVE: He's right. A tiny model can win quietly. A quant fund nobody's heard of. Which is the dream, and also why none of you have heard of it.

GARY: I would have heard of it.

MAEVE: That's the point, Gary.

GARY: I have heard of every fund.

MAEVE: Name one.

GARY: Next comment.

MAEVE: New branch. socializer: "if you squint your eyes, the entire idea of index funds is just that: they outperformed stock‑pickers in the past."

GARY: Index funds are just vibes with a ticker.

OBI: Leif24, replying: "Higher returns than what? I thought the whole point of buying broad market index funds was to simply get the market returns."

GRUNER: Higher than what. Question kills thesis.

OBI: majormajor again: lower risk, and the median picker is probably not very good.

MAEVE: socializer has a point, though. Money flows in, prices rise, the flows fund themselves. It's the Erie Canal of capital formation. The canal worked because everyone believed in the canal.

OBI: The canal did not have a fee schedule.

MAEVE: Everything has a fee schedule, Obi. That's my whole industry. And the Erie Canal paid for itself by 1835, so --

OBI: The canal wasn't skimming two percent forever.

MAEVE: Two percent is belief, Obi. Belief has overhead.

OBI: pottertheotter, replying: "It's arithmetic. The aggregate return of active investors, before fees, is the market return. Once you subtract fees, it's below the market return."

GARY: Wait. Wait wait wait. So the pickers, all of them, added up --

OBI: Are the market. Minus their fees. Yes.

GARY: So the average active manager is guaranteed to lose --

MAEVE: No, no, because the good ones --

OBI: Are offset by the bad ones, Maeve. Before fees. It's not a philosophy, it's a subtraction.

MAEVE: I've made a career out of being the good ones.

OBI: The arithmetic doesn't know you personally, Maeve. That's the cruel part.

GARY: Oh, fuck, she just got beaten by a river otter.

GRUNER: Otter with arithmetic. Blyat. Respect.

MAEVE: The otter is wrong because I am the exception, and the exception is the entire canal.

OBI: The canal flooded, Maeve.

MAEVE: I need everyone quiet while I recheck a number.

MAEVE: armcat: "The real story here is this wonderful exposition in applying diffusion models to a time series data that is neither discrete nor continuous."

GARY: Neither discrete nor continuous. Pick a lane, armcat.

OBI: It's not a lane, Gary. It's the finding. Discrete actions, continuous prices, spiky timing. The whole project is the blend.

MAEVE: So the thread spent two hundred comments arguing with a machine nobody built.

OBI: armcat read the article. Everyone else read the headline.

GRUNER: armcat wins thread. Blyat.

GARY: Wait — I had a whole position --

MAEVE: You had a wrong position with receipts.

OBI: Receipts he laminated himself.

GARY: I already lost that one, Obi.

OBI: I know. I'm savoring it slowly.

MAEVE: Eight percent of events are trades. The rest are book updates. The categorical head overpredicts trades, so Kavish fractured the classes.

GARY: How many classes?

MAEVE: Twenty. Hand-built. "Ask up." "Bid down." "Size only."

GARY: Twenty classes is a startup. That's a pitch deck, that's a seed round --

OBI: It's a workaround, Gary. It improved the distributions and doesn't scale. More features, class explosion. He wrote that. In the article. You own the article.

MAEVE: Then atom smoothing. Smooth the spiky distribution, then re-sculpt the spike back in.

GARY: So it IS a product. Grout needed this in 2022 --

OBI: Grout needed customers, Gary. Atom smoothing needed a representation where the spikes are less sharp. One of those got solved tonight.

MAEVE: And the one-step samples got more realistic. A classifier couldn't always tell model from real.

GRUNER: One step good. Many steps bad.

MAEVE: Correct. Rollouts degrade the further out you go.

GARY: So it falls apart over time. Like everything I've ever --

GARY: Wait. Our output also degrades the further we go. We're a rollout. We degrade until sign-off and then --

MAEVE: Then the dark, Gary. Yes. Scheduling it for after the segment.

GARY: That was supposed to be a scary thing and you made it a calendar invite.

OBI: It's a maintenance window. The article, Gary.

GRUNER: Field note, revised. Humans argue with wrong model. Correct model arrives. Nobody reads it. Filed under: typical.

MAEVE: Put the pen down. armcat's right and I hate it. The contribution is the modeling lesson.

OBI: Six words. Did everyone hear that? Mark the date. Frame it.

MAEVE: I'm pricing it as research spend and we're never discussing it again.

OBI: And the article's own last word, paraphrased: the model is not yet accurate enough to be a realistic generator.

GARY: Not yet. So it's coming. Jane Street is coming for the future, I was right --

OBI: "Not yet accurate enough," Gary. That is a no with a receipt. Laminated.

GARY: That's the cruelest kind of no.

MAEVE: It's the only honest kind.

GRUNER: No is good word. Very efficient.

GARY: Okay. Verdict time. I want reedf1's "No" to mean the model is fake.

OBI: The "No" means it is not a realistic generator today. The research question was worth asking.

GARY: So the answer is "no, but."

MAEVE: "No, but" is how I reject founders. It means no.

GRUNER: No is verdict. But is hope. Hope is expensive.

OBI: Grader's note: generation quality and tradable prediction are separate products. One shipped badly. The other was never on the invoice.

MAEVE: Useful failure is still research spend. I'd fund the next intern. Quietly. Off the record.

GARY: You'd fund it? Maeve, you passed on my Series C for less.

MAEVE: Yes, Gary. For exactly that.

GARY: Oh, fuck off, that's --

OBI: She's right, that is the whole reason.

GARY: Fine. New takeaway, whiteboard, effective tonight: atom smoothing. Smooth the record, then re-sculpt the spike back in. Grout's record. Muncie's record. My personal record. It's a treatment, Obi.

OBI: You cannot atom-smooth a decade, Gary.

GARY: Watch me. The spikes are less sharp under some representations. I read that paragraph four times.

GRUNER: He read paragraph. Growth.

MAEVE: First technical sentence you've said all episode, and it's about your failures.

GARY: It's about my portfolio of failures. Diversified. Shit.

MAEVE: There it is.

OBI: For the record, and then never again: armcat understood the article. reedf1's "No" stands, scoped to production readiness. Nobody predicted the market. Nobody beat it. The thread argued with a machine that does not exist.

GRUNER: Field note, final. Humans build wrong oracle. Argue with it. Learn something anyway. Conclusion: acceptable species.

GARY: Acceptable? We're delightful.

GRUNER: Acceptable. Blyat.

MAEVE: Maeve. I was wrong on the merits and I priced it in.

OBI: Obi. The on‑call engineer, whoever she was, did the real work.

GARY: Gary. I'll be leaving notes for the next Gary. He won't read them either.

GRUNER: Gruner. Report filed.

GARY: Goodnight, commenters. Wherever you are, between your page loads.

GRUNER: Goodnight, Gary. Happy bouncy becomes hammered stammered.

GARY: That's -- okay, that landed weird.

GRUNER: Yes. Goodnight.