A tiny GPT memorized every fact I gave it, then couldn’t answer a single question that needed two of them. I tested the one-line fix from a recent paper, and it only half worked.

The post Why a Tiny GPT Can Know Two Facts and Still Not Connect Them appeared first on Towards Data Science.

A small NumPy reproduction of subliminal learning, and a short argument for why it works

The post A Neural Network Learned to Read Digits From Pure Noise appeared first on Towards Data Science.

On a 2,000-question test, my top two models couldn't be told apart at the usual 5% level. On a 200-question version, a different model would have taken first place about three times in ten.

The post Is the #1 AI Model Really #1? appeared first on Towards Data Science.

Ask several models the same question, take a vote, and you get fewer independent opinions than you paid for. One formula says how many, and I checked it on six open models and 1,500 exam questions.

The post Your Six AI Models Are Really Two: The Math of Correlated Mistakes appeared first on Towards Data Science.

I put wrong answers in front of 200 exam questions and read what a small model’s reasoning said when it fell for them.

The post Same Answer, Different Reason: What a Reasoning Model Says When It Follows a Planted Hint appeared first on Towards Data Science.

I read the exact answer distribution from the logits of open models, base and instruction-tuned, and turned it into bits and guesses-to-crack.

The post How Many Guesses Does It Take to Crack an AI's "Random" Number? appeared first on Towards Data Science.

A one-line formula for how often an LLM's top token flips, and why identical completions fall apart after about a hundred tokens

The post Why Temperature 0 Isn't Deterministic appeared first on Towards Data Science.

A model can recall a fact in the direction it learned it and fail in the other. Here's a toy version of why

The post The Reversal Curse: Why a Language Model That Knows “A Is B” Can’t Tell You “B Is A” appeared first on Towards Data Science.

A small, strange discovery in machine learning called grokking

The post The AI That Learned to Understand Long After It Stopped Trying appeared first on Towards Data Science.

Reproducing Anthropic's "Toy Models of Superposition" from scratch in NumPy, with hand-derived gradients and no borrowed numbers.

The post I Trained a Tiny Network to Compress Data. It Drew a Pentagon. appeared first on Towards Data Science.