The Lab: we ran X’s released model on real posts

Everyone argues about what the algorithm does. The released checkpoint is 2.9 GB and runs on a laptop — so we stopped arguing and measured. Below: the released mini-Phoenix ranker scored against 50,000 real posts from the corpus xAI shipped with it, joined to 1,311 posts’ live public engagement, plus direct probes of the model’s learned behavior. Every script is downloadable at the bottom — reproduce all of it.

Finding 1 — the released model cannot pick an author’s winners

The definitive test of “score your post with the real algorithm”: hold the audience constant and ask whether the model’s P(favorite) ranks an author’s own posts by how well they actually did. Across 74 prolific authors (877 live-verified posts, ≥6 each): median within-author rank correlation -0.003, positive for 50% of authors — a coin flip. Replies: median +0.049.

Replicated before publishing: the same backtest rerun under a second viewer context (empty engagement history) gives median +0.008, positive for 51% of authors. The null is not a quirk of one user context.

Across all authors it’s worse than a coin flip — Spearman -0.314 between P(favorite) and actual like counts. Median likes by model-score decile (lowest→highest):

The model’s highest-scored posts got the fewest likes. Why: it predicts per-impression probability for one specific viewer — big verified accounts collect more total likes on far more impressions at a lower per-impression rate, and the model (correctly, for its own job) scores them lower for this viewer. Raw like counts measure audience size; the model measures a person’s click. Any tool showing you “predicted likes” from this model is wrong twice.

Finding 2 — personalization is the whole game

Identical 50,000 posts, two viewers: a user with just 3 engagement events in history vs the same user with none. Mean P(favorite) shifts 6.7×, and the post ORDERING itself only correlates 0.68 between the two. There is no such thing as “your post’s score” — only “your post’s score for a given person’s history.” This is why the checker refuses to print one number.

Finding 3 — the model’s cross-head probabilities are miscalibrated ~10× (we got this wrong first, then caught it)

Our first draft of this page reported the model’s head ratios as facts about X: “a reply is ~345× rarer than a like.” That was wrong, and our own data disproves it. In the same 1,311 posts, real engagement runs 35.4:1 likes-to-replies while the model implies 352:1 — off by a factor of 9.9.

The corrected finding is more useful than the wrong one: the released checkpoint’s per-head rankings may be usable, but its cross-head probability scales are not — most likely an artifact of per-head negative sampling during training, which preserves within-head order while destroying cross-head scale. So anyone converting these outputs into “a reply is worth N likes” is publishing a training artifact. That includes the version of us that existed earlier today. Retracted before launch; the numbers are preserved in lab.json under calibration_check.

Finding 4 — the learned age curve, measured

Same 2,048 posts scored at 15 impression ages (the model takes the impression timestamp as input). Mean P(action) relative to 15 minutes:

Three learned behaviors nobody has published before: replies collapse (a 3-day-old post earns ~36% of its fresh reply probability — the conversation window closes), reposts spike at 2–4 hours (~1.5×) then fade, favorites show no decay. And every curve goes exactly flat from 80 hours — the POST_AGE_MAX_MINUTES = 4800 bucket cap in phoenix/recsys_model.py, now visible in behavior. “Golden hour” folklore is half-right for the wrong reason, and it’s per-action.

The flatline is not a preference — it’s saturation, and the weights prove it. The shipped post_age_embedding_table has exactly 82 rows = 4800 ÷ 60 + 2. Past bucket 80 every post shares one embedding, so the curve must go flat — presenting that as learned behavior would have been wrong. What is learned is the shape inside the window.

Replication + a correction: rerun under a cold viewer context, the reply and repost decay directions replicate (cold reply 0.43–0.48 of fresh at 64–80h; cold repost 0.71–0.84), but the table’s late rise in favorites (1.27× at 80h) does not — the cold context is flat (~1.0×). So the viewer-robust claims are: replies and quotes decay hard, reposts peak early then fade, favorites don’t decay; exact magnitudes are viewer-dependent. Cold-context curves are in the raw data under replication.

Prior art worth knowing: Pfeffer et al. (arXiv 2302.09654) measured a median impression half-life of ~80 minutes. That is a different quantity from ours — they measured how fast a post stops being shown, we measured predicted per-impression propensity conditional on being shown. Both can be true; no 2024–2026 replication of either exists.

Finding 5 — real-engagement feature effects (with the confounds stated)

From the 1,311 live-verified posts (sports corpus). These are raw correlations on real engagement, NOT causal claims — audience size is uncontrolled except where noted:

The question-mark row is the one to keep: posts with a question earned 4× the median replies — on real data, consistent with what the reply head lever has said all along. An academic sweep found no published study estimating this effect on X, so treat it as new and unreplicated — and note the one adjacent randomized result cuts against free lunch: Neubrander et al. (RCT, n=2,282) found curiosity-promoting design raised question-asking but decreased liking. Questions may buy replies at a cost in likes; our data doesn’t test that trade-off. It’s the subject of a registered forecast — we re-run it on fresh posts in August and publish whichever way it lands.

Finding 6 — heads move together, but not identically

Rank correlations between named heads across 50,000 posts: reply~quote 0.86, dwell~favorite 0.84, dwell~video 0.84 — but favorite~repost only 0.42 and reply~video 0.39. The heads are genuinely different predictions, not one “quality” score renamed 19 ways — which is why the weighted sum (and its withheld weights) matters.

Finding 7 — the fair test: a regex beats the model 3.5×

The strongest objection to Finding 1 is that like counts conflate a good post with a big account. The clean fix would be engagement per view — but we checked, and the zero-API path doesn’t expose view counts or even follower counts. So we used a quantity that cancels reach algebraically instead of adjusting for it statistically: engagement composition.

actual reply share = replies / (replies + likes)          <- impressions cancel
model  reply share = P(reply) / (P(reply) + P(favorite))   <- per-impression by construction

Both sides are rates. A model whose entire job is per-impression propensity should be good at this even if it can’t predict totals. Results on posts with at least 10 engagements:

Then the comparison that matters. On the same 1,065 posts, we ran a one-line heuristic against the model: does the text contain a question mark?

That is the honest headline for anyone considering paying for “algorithm scoring.”

Finding 8 — its skill appears exactly where it has memorized

The prolific-author result is genuinely positive, and it points at the mechanism. Splitting the same test by how many times an author appears in the shipped corpus — a proxy for how much the model could have memorized them:

Skill is absent for authors seen once and rises to ~0.13–0.17 for authors seen repeatedly. It isn’t perfectly monotone (the top bucket dips and is the smallest), but the direction matches the parameter census exactly: this is a memorization engine, and it predicts where it has memorized. For a post from an author it hasn’t seen — which is to say, for your draft — there is nothing left. That is Finding 1, Finding 6 and Finding 7 all reduced to one sentence, and it is the reason no tool can honestly score a new post with this model.

Caveat: corpus frequency proxies training exposure, it doesn’t measure it. Reproduce with phx_composition.py.

Finding 9 — checkpoint forensics: what the weights say that the docs don’t

Loading the file settles things reading the code cannot. Three results:

(a) The release misdescribes its own model, and everyone repeated it. The root README (line 32) advertises the artifact as “256-dim embeddings, 4 attention heads, 2 transformer layers”. phoenix/README.md’s architecture table says 128 / 4 / 4. The shipped tensors settle it: 128-dim, 4 decoder layers (transformer/decoder_layer_0..3, every matrix [128 × …]). The root README is wrong on both dimension and depth — and since syndicated coverage copies the root README, the wrong numbers are the ones in circulation. The error is upstream, in X’s documentation of the file it ships.

(b) 99.79% of the model is memory, not reasoning. The checkpoint is ~770M parameters — and 99.79% of them are ID-hash embedding tables (3 × 1M × 128, twice over). The transformer that actually reasons is 1.6M parameters, 0.213% of the file. That is the mechanism behind Finding 1: a model that is overwhelmingly a lookup table over IDs it memorized in training has almost nothing left with which to judge a post it has never seen. The null result isn’t a bug — it’s the architecture.

(c) How many action heads are there? Public accounts say 12, 13+, 14, 15+; the weights say 19. The root README never states a number (“probabilities for many actions”); phoenix/README.md says 19; write-ups have reported everything from 12 upward. The shipped action_projection tensor is [19 × 128]. It’s 19 — the cleanest demonstration on this page that running the checkpoint beats reading about it.

(d) The 80-hour ceiling is physically in the weights — an 82-row age table, as above — and the corpus ships 84,564 posts, not the “~537K” documented.

Is any of this new? We checked before claiming it

As of 2026-07-31: the repo has issues, discussions and pull requests all disabled — there is no public technical surface, and consequently every substantive analysis we could find (Truth Tide, swyx on Hacker News, several Substack teardowns) is static code reading. We found no published work that loads and runs the released checkpoint. The nearest prior art, hjosugi/xalgo, does compute rank correlations against a real For-You ordering — but with a hand-built engagement÷views proxy, explicitly not the Phoenix model. If we’ve missed someone, say so publicly and this section gets their name.

The likely reason nobody checked: this release was published into an empty room. The 2023 open-sourcing drew a Hacker News thread with over a thousand comments and a community annotation project (awesome-twitter-algo) that made it legible; the 2026 release drew a fraction of that attention, no annotation effort, and — because issues and discussions are disabled — nowhere to report anything. xAI has never published a technical writeup of its own recommender; no named engineer has explained the Phoenix stack publicly. An LLM “banger” quality screen shipped inside it and, as far as we can find, went unmentioned in the press entirely.

Where independent evidence disagrees with the code — links

Our link verdict says: no link term exists anywhere in the open ranking code. That remains true and checkable. But two peer-reviewed 2026 studies measure a large real-world penalty anyway — Galeazzi et al. (NDSS 2026, 40M+ tweets: link posts get 4.67–7.45× lower median visibility) and Efstratiou et al. (ICWSM 2026, ~205k exposures: external links the strongest negative predictor, p<0.001, while follower count was not significant at p=0.14 — which dissolves the account-size confound we flagged in Finding 5). Meanwhile X’s own leadership denies it outright: Nikita Bier, “links were never deboosted”; Musk, July 29 2026, “we haven’t for over a year.”

We’re updating our verdict rather than defending it: no written rule in the open code, strong measured effect in the wild — exactly the learned-or- withheld case our myth page allowed for, now with peer-reviewed evidence attached. Note also that Efstratiou finds video exempt from the penalty. We cannot test the before/after ourselves: our sample is a 23.7-hour window (May 14–15 2026), which is a limit of the shipped corpus, not a longitudinal dataset.

Honest limits of everything above

1. This is the mini checkpoint — production is larger, continuously trained, and fed by Grox content embeddings this release doesn’t include. Our null result (Finding 1) bounds what tools using this release can do; it does not measure production.
2. One viewer context (the shipped example user, plus a cold variant). Ensembling many viewer histories could behave differently.
3. Live like/reply counts include impression luck; within-author is the closest control the public data allows.
4. Only 6 of 19 output heads are officially named in the release (run_pipeline.py); we publish the rest as unnamed distributions in the raw data.
5. Cross-head probability scales are miscalibrated ~10× (Finding 3), so no number here should be read as “action X is worth N of action Y.” We published that error first and corrected it the same day; it is the single easiest mistake to make with this checkpoint.
6. Our sample is one 23.7-hour slice of one topic (sports). Effects that differ by vertical or by news cycle are invisible to it.

Reproduce it

Clone xai-org/x-algorithm, pull the LFS artifact, then run our four scripts (Python, JAX CPU): phx_score.py (score any corpus subset across all 19 heads, or sweep impression ages) · phx_fetch.py (live public engagement, no API) · phx_analyze.py (backtest + feature effects) · phx_composition.py (the impression-invariant test, bootstrap CIs, and the regex baseline). Raw output: lab.json. If you get different numbers, tell us publicly.