序

“What does it mean to predict the next token well enough? What does it mean actually? It's actually a deeper question than it seems. Predicting the next token well means that you understand the underlying reality that led to the creation of that token.”

“Like, it is statistics, but what is statistics? In order to understand those statistics, to compress them, you need to understand what is it about the world that creates those statistics.”

“What is it about people that creates their behaviors? Well they have thoughts and they have feelings and they have ideas and they do things in certain ways, all of those could be deduced from next token prediction.”

“And I'd argue that this should make it possible — not indefinitely, but to a pretty decent degree — to say: well, can you guess what you do if you took a person with like this characteristic and that characteristic? Such a person doesn't exist but because you're so good at predicting the next token, you should still be able to guess what that person would do.”