Eponymous

Hi, friends!

I'm back in Idol, and I'm very glad to see a lot of familiar faces, and just as happy to see a lot of unfamiliar faces that I look forward to getting to know and read.

I'm usually a speculative fiction short story writer, but I figured I'd start off this season with something a little bit different, both in terms of topic and approach. Usually, I write an entry, post a link to it at exactly (or after) the deadline, and then walk away from it forever. For this entry, though, I wanted to do something different, and invite comments and questions, because I think dialogue and discussion are important, and I think that people are approaching it from very different angles and understandings. I don't expect everyone to agree with me - if anything, I would be shocked if that happens - but I do hope that people learn something from it. So without further ado, let's bring forth (with thanks to John Scalzi) a fictional interlocutor for this conversation:

Oh hey. So you're writing for Idol, cool. What are you thinking of?

I was thinking of writing about AI, actually.

Oh. That's definitely not a controversial topic that people have extremely strong feelings about. Why this instead of literally anything else?

I think it's a pretty important topic, and I think it's going to impact all of our lives soon. I am not unbiased: I recently started at one of the frontier AI companies, in fact. However, I'm not here to tell you to use it, or here to defend it, exactly. But I think in order to have meaningful conversations, reasonable minds should be working on the same facts.

Okay, I'm sure someone is thinking this right now: was this post written with AI?

No, and I'll go into a bit more of that later, but I think it's an important question. None of my entries will be written with or edited with AI.

Probably. Maybe we should start with the basics: what is AI?

This feels like a weirdly hard question, which is maybe why I feel like society doesn't do it well. We live in a time when context is really hard and nuance is nonexistent, and trying to communicate what the technology is feels both like describing an elephant whilst blind and, I dunno, impossibly reductive, at the same time.

I'm going to keep this as high level as possible, but dive a bit more into the specifics if people ask for it in comments. The highest level (while trying to keep it useful) is that what we talk about when we say 'AI' refers to large mathematical models that have been given a staggering amount of information (we call this 'pretraining'), and then trained to produce outputs that we want (this is fine-tuning/post-training). In pretraining, the goal is to give it all the information it can, and then let the model establish/identify connections between the things that it's given, primarily by having it guess the next word that comes in a sequence, trillions of times.

That's pretty abstract; to ground it a bit, the simplest version of a model thinks of sun as a concept as probably close to warm and light, and is probably not very related to wisdom or algebra... but possibly has a slight association due to some mythological stories about sun gods and wisdom. But it's more complex than this: it's not just about word associations, because with enough data, the model is not just making word maps but also can parse/weigh structures like grammar or patterns of argument. Concepts - words, grammar, ideas - are encoded mathematically with 'weights', and those weights symbolize their associations with other words. The more you feed it, the more 'accurate' the weights are.

Pretraining means a model gets weights, sure. And then it's done?

Not quite. After the pretraining phase, you have a model that has weights... but that, it turns out, is not very helpful to actually solving problems or doing anything. If you ask it 'what is the Senate'? A model that has only been through pretraining may answer any number of ways - it might give you an answer, or it might respond with 'what is the House?', because those might be equally likely outputs. After pretraining, there's then a process to effectively 'teach' the model how it should respond, both by giving it samples of good questions and answers ('supervised learning'), as well as by going through tests and grading it as it gets closer to good outcomes ('reinforcement learning'). So you can imagine giving it examples of what a person might ask and good answers to those questions, as well as having it take tests and grading its responses better as they get closer to the answer.

After you have done this post-training process, the model is 'complete', and will respond the way that you trained it to. So to summarize: A model gets pretrained on as much data as possible, generates weights for everything it's seen, and then is post-trained to do what people want it to do. AI models are everywhere, most of them are specialized for their tasks: spam filtering, anti-fraud from your bank, ad-targeting, machine vision for self-driving cars, etc. But most of what dominates the conversation are these 'general purpose' AI models, or an AI that can do anything you ask it to.

So... AI is fancy autocomplete?

I think this is a basic misunderstanding. In some sense, yes, you could probably describe a model that is only pretrained as 'fancy autocomplete', but the general AI models in the world are far more advanced than that. If there's one takeaway, I think it's that philosophically, saying 'oh AI is just math' is kind of like saying 'oh, human sentience is just neurons firing chemicals at each other' - it's strictly true but does not describe or provide any understanding of what happens at the aggregate level.

Why doesn't it understand how many rs are in strawberry, or that you can't walk to a car wash if you want to get your car cleaned? Why does it hallucinate?

The strawberry example is a great example of how it processes information (by tokenizing words), and how those errors seem extremely simple to us... but are a poor reflection of its capabilities. It doesn't 'think' the way we do, and it doesn't do what a person does in terms of reasoning. And so it makes what looks to us are really dumb mistakes, because the way that it gets to answers is closer to 'this sort of looks like the past' instead of 'this is logic' - the way that we would logic it out. It's trained to produce plausible text, not reason absolutely like we are.

Here's another way of thinking about it: all of the answers it gives are probabilistic instead of deterministic - it doesn't know anything for sure, but instead is guessing based on its picture of the world.

Wait, so no one should use it for anything at all! How is it making strides in solving previously unsolved math problems ... why is anyone using it if it's just a guessing machine? There seems to be a lot of use in software engineering.

For three reasons. First of all, it turns out that if your models are good enough, your outputs are going to be pretty good. The obviously scifi thought experiment is this: if you could simulate every single atom in the entire universe, you could just run a simulation of the universe, and as long as you had access to that simulation, you would be able to predict and control every event. We're obviously not there yet, but the models are good enough to give reasonable outputs to a large variety of things. But more importantly, for many things (like, say, writing emails) a good enough guess is more than good enough.

The second answer is that fundamentally, the models are better when they create output that can be checked, and so unsurprisingly they're extremely efficient and effective at code, because there are quantitative ways of judging code: does it run? Does each individual element do what you want it to (unit tests)? Does the overall outcome look like what you want it to look like? For math, proofs can be checked similarly, and so every time it comes up with something potentially right but wrong, it can be checked and rejected.

The third answer is a matter of resources: if one AI has a 5% chance of getting it wrong, that's pretty bad! But you can assign another agent to error check the first one - take the first agent's outputs and ask it whether it's correct - you catch a number of those errors. Increasingly, especially for complex tasks, it's no longer one agent doing everything, but a team of agents that splits up the task, error checks at multiple levels, and then returns the outcome to you.

Wait, you didn't address that thing you were going to come back to - why aren't you writing entries with AI?

Because I want to communicate with intent, and because (perhaps this entry aside), I want to create art. I think that art is about human intent. To me, AI should be used to push forward the boundaries of math and science (and especially if it can cure diseases, or compress research, or make our lives better). I think AI should do what AI is good at. But AI - at least in its current state - does not 'think'. It does not create by itself with the goal of communicating. I believe that the value of art is that a human being wants to communicate something to the world. And I think the line to be drawn is one of choice: an artist's choices are what make art the way it is, whether it is the arrangement of words in a poem or the brushstrokes on a canvas or the brush size in digital art or the framing and shutter speed of a photograph. In the way it's typically used, there are no such choices in AI generated content, and that makes it Not Art to me. It's not to say that it can't be beautiful, or even evoke feelings like the best art, but that doesn't make it art to me, and that's not what I want to present to you.

Okay, that was kind of a lot.

Yeah, sorry. And this is the short version - I haven't talked about alignment (everything from Skynet to making paperclips, to AI hacking other companies). I haven't talked about the wider concerns about geopolitics that exist right now. I haven't talked about the commercialization, or the potential societal impact, or any of that. There's so much to talk about, everything from philosophy of mind to socioeconomics, from how technology advances to what we should do about it. But here's where I want to ask you what you think, what you'd like to know, and what you're scared about.

Let's talk about it.