What is a vector database, anyway?
Longread: How machines learned that meaning is distance, and why your brain already knew.
Here is something your brain does so casually that you’ve never once thanked it for the effort.
You hear a song you have never heard before. Only a few seconds in, a thought arrives: this sounds like that one band. You couldn’t tell me which feature gave it away. You weren’t comparing waveforms or counting beats per minute. You didn’t run down a checklist. It just somehow felt close to something you’ve heard before, and so your brain handed it to you.
Or take the more annoying version. You’re reaching for a word and it isn’t there. You know it’s near “happy,” but happy isn’t quite it. It’s warmer and more backward-looking than that, a little wistful, the bittersweet feeling you get looking at an old photograph. You can feel the shape of where it lives. The neighborhood, the street, almost the house. You just can’t ring the doorbell. And then twenty minutes later, in the shower, it hits you. Nostalgia. There it was the whole time, exactly where you knew it would be, filed not by spelling but by feeling.
Notice what’s happening in both cases, because it’s the entire subject of this post. Memories aren’t retrieved by their address, file path or phone number. You never think “fetch thought number 4,812”, or try to remember a word by going through every word you know alphabetically. You fetch thoughts by resemblance; you hear something, see something, smell something, and a thought simply arrives, and others with it. If you pay attention to it, you can almost feel how one thing in your mind is so close to other things, you can’t help but invite them into your consciousness.
Now, this is definitely not how a computer remembers anything.
A computer’s memory is a filing cabinet with numbered drawers, and it is really good at exactly one thing: if you know the number, you get the contents, instantly, perfectly, a billion times in a row. What it can’t do, natively, is the thing you just did in the shower. Ask a traditional database for “documents like this one” and it will stare at you blankly, because it has no idea what like means. It can match the label on the drawer, and that’s it.
This is why keyword search, for its entire long life, has been great at finding an exact thing, but lousy at finding similar things. Search for «car», and it will find records that mention cars, but skip records that say «automobile» instead. It’s a different set of letters meaning the same thing, and the machine, bless it, reads labels, not meaning.
So we’re on a quest, and it’s older than you might think. For more than half a century, a line of people in very different disciplines have been trying to teach the filing cabinet to do the shower trick. Librarians. Linguists. A few cognitive scientists who thought they were studying pigeons. And, eventually, a generation of AI engineers. The goal was always the same: set up a system that retrieves by resemblance instead of by address, something that works less like a filing cabinet and more like the lump of tissue currently reading this sentence.
It’s surprisingly difficult. What does a word really mean? A dictionary offers a precise definition, but that’s not what the word truly means. Take Christmas: it’s “an annual festival commemorating the birth of Jesus Christ, observed primarily on December 25 as a religious and cultural celebration.” Every word of that is true, and none of it is what the word means to you. Christmas means family. Food. Smells. Familiar songs. Gifts. Stockings. Santa Claus. Try wishing someone “a joyous annual festival commemorating the birth of Jesus Christ, observed primarily on December 25”, and you would sound like an alien trying to pass as a human.
This is the cloud of association carried by even the smallest unit of language. A single word is already a dense knot of meaning, most of it unspoken. And if one word holds that much, consider what waits inside a sentence, or a paragraph, or a question someone types into a search box at midnight.
That’s the size of the problem. Not “store the word Christmas,” but store everything Christmas drags along with it, and do the same for every word, every sentence, every paragraph ever written, and then somehow find the ones that mean nearly the same thing. For forty years it was closer to a dream than a plan. And then, recently, it started working.
The thing they built is called a vector database, and over the past few years it has become one of the most important brain parts inside almost every AI product you’ve touched. It is the reason a chatbot can answer questions about your company’s documentation. It’s underneath the “you might also like” on Amazon and Spotify. It is, increasingly, how artificial intelligence is given a memory more like a human brain than a file system.
I’ve built apps for more than a decade, with both no-code tools and later having AI work out code for me. And throughout my entire career, I’ve suffered from a particular compulsion: I cannot leave a tool unexamined. Picking it into pieces, holding each one up to the light to see how it works, reassembling the whole thing, taking notes all the while. And then the moment, when the mechanism clicks into understanding. I have chased that click through hundreds of articles and two books.
When vectors started showing up in my databases about a year ago, it felt like a different kind of beast. I’d open an explainer, read the words “high-dimensional embedding space,” and feel my brain politely excuse itself from the room. Every description I found was either four words long and useless or forty pages long and worse. The thing clearly worked, but I couldn’t assemble even a wrong picture of what it was, and a wrong picture is usually where I start. How could I possibly even begin to understand this thing?
So I did what I do: I wrote. I tried to pick that “high-dimensional embedding space” apart, and see if I could simplify it, to the point where I could get, if nothing else, an inkling of an idea of how it worked. And after a few days of trying to construct some kind of framework, what emerged was, quite simply, the most elegant and beautiful piece of technology I’ve seen. Once I’d started, I couldn’t stop.
This is that journey, my notes, if you will, with the rough edges polished. What looked impossibly complex turned out to rest on an idea of surprising simplicity, and there is a trick to feeling it click. First, you need to visualize it. See everything, in one, two and three dimensions. Build the picture fully. Then, at a moment I’ll point out, you have to force that eye shut, because the idea keeps going where pictures can’t follow. From that point on, trust the mathematics. And believe me, I’m as surprised as you are to be saying that, but by the time we get there, so will you.
But we won’t start with the database. We won’t start with vectors. We’ll start with one of the first attempts to connect content with meaning – in the library.
Meaning as a place
Let’s begin with one of these everyday things you’ve seen many times without giving it much thought: the numbers on the spine of a library book.
In the 1870s, librarian Melvil Dewey had an idea: that you could assign every book a number such that books about the same thing end up physically beside each other.
The 500s are the sciences. Slide along the shelf: 530 is physics, and a few steps further, 539, the physics of the very small, atoms and particles and the subatomic. Stop anywhere and the books to your left and right will be about roughly what you’re standing in front of. No single book covers the entire subject, but you can be sure its content is connected to the very place you’re currently standing. At a specific location in the library, you are in the science section, and you can walk, with your actual legs, toward a topic and watch the books change title and content as you move. The location is the meaning. Where a book sits tells you what it’s about, and books to the left and right of it share a similarity with it.
The Dewey Decimal System is a beautiful idea, and has held up for more than 150 years in libraries all over the world. But it has an obvious limitation, which is that a shelf is a line, and a line only has one direction.
Think about it: Dewey gives each book a single number, representing one meaning, one position on one axis. But things don’t resemble each other in only one way. A book on the physics of music is about physics and about music, and the shelf forces it to pick a side. It cannot sit on the shelf next to “The physics of sound waves” and “How to tune a piano” at the same time, because on a single line, everything has exactly two neighbors and no more. To organize the world along one dimension is to declare that there is only one way for two things to be alike, but the world is alike in a thousand directions at once.
So, one thing, one number. Physics. 530. But if we want to move it closer to how the human brain works, we need more. What if we didn’t give each thing just one number, but a few hundred?
Before we do that, and move it too into a visual space, let’s visualize how that might work. Imagine describing anything in the world by turning a bank of little dials. Each dial assigns a specific trait or quality, and you can assign many dials to a single thing. How formal is it? Dial. How cheerful? Dial. How much does it have to do with food, with money, with the ocean, with being old? Dial, dial, dial, dial. Every dial gets a setting, some of them with a high value and many with a low one. The full row of settings is that thing’s complete description.
A cappuccino and an espresso would land on nearly identical dials: high on “warm,” high on “caffeine,” high on “Italian,” lower on “soda.” A cappuccino and a penguin would agree on almost nothing. And that’s the whole definition of similar, sitting right there: two things are alike to the exact degree that their dials match.
A library shelf is a control panel with one dial. Carl Sagan’s “Cosmos“ has the dial set to 520: “Astronomy and allied sciences,” but nothing else. It is a book on astronomy, but Dewey’s system suggests it has no other quality. In a library that’s a good enough system, but in our mind we can and do assign lots of other qualities to it, with different values. High for “astronomy,” a little lower for “sense of wonder,” lower still for “adventure,” down near zero for “romance.” Those qualities give us a list of numbers, one per quality, and each number is a coordinate on its own axis.
Let’s look at how “qualities across axes” would be organized in the library. The first thing we’ll do is to assign a dial to the X axis. The dial is “scientific”, and the metric shows how scientific it is on a scale. This is as simple a ranking system as we can get. On Dewey’s shelf, books on the left side of it would be not scientific at all, and books on the right side would be as scientific as it gets.
Add a second number and you get a Y axis too. We’ll label that dial “inspiring”. On Dewey’s shelf, that would place books above and below each other.
Add a third and you pop off the paper into the depth of a room, a Z axis, where books can also sit in front of or behind one another.
And here’s the magic Dewey could never fit onto a shelf:
the same book keeps different company in each direction.
Cosmos sits among the science books along the X axis, among the inspiring ones along the Y axis, and among the poetic ones along the Z axis. And each of those rows is its own little world, gathered around one quality, and uninterested in the other qualities. The inspiring row doesn’t care about science. Along it you’ll find an inspiring book about the cosmos shelved right next to an inspiring book about Michael Jordan. On the poetic row, Cosmos sits among other non-poetic books, covering potentially wildly different subjects, ranked by their poetic-ness (or lack thereof). Nothing forces Cosmos to choose a single shelf.
In these illustrations, we’ve used library shelves to make the idea visible. But you may have noticed a small cheat: we kept Cosmos in the dead center of every image, with the other books arranged around it, as if the library existed for its benefit. In reality, no book gets that honor. Cosmos is one book among tens of thousands, every one of them sitting in its own spot, every one of them the center of its own little neighborhood.
So drop the cheat, and a question appears: if Cosmos isn’t “the middle,” where is it? The shelves themselves answer it. Its position along the X axis is a coordinate, set by the value we assigned it with the “scientific” dial. Same for the “inspiring” Y axis, same for the “poetic” Z axis. And the moment you write those down as numbers, say, scientific 9, inspiring 8, poetic 2, you no longer need the picture at all. The three numbers give the book its position. Let’s walk up the dimensions one more time, now with Cosmos’s actual numbers attached.
The first is a one-dimensional score for the “scientific” quality. A single coordinate: 9. This is easy, it could be a slider on a website:
Now move to two dimensions, using two of Cosmos’s dial values:
Scientific: 9 (X axis)
Inspiring: 8 (Y axis)
With one more dimension, Cosmos’s position starts to make sense. We have a 2D map, and Cosmos sits at the exact coordinates (9, 8).
Add a third dimension, the last dial, for a Z coordinate:
Scientific: 9
Inspiring: 8
Poetic: 2
Now Cosmos sits near the front of the room, floating high. Its position tells you what kind of book it is: highly scientific, pretty inspiring, not very poetic. We’ve just reinvented something you’ve known since school: coordinates. X, Y, and Z, except each axis now measures a quality instead of a direction in a room.
That’s the whole translation. A dimension is a quality. A coordinate is how much of that quality a book has. And a book’s full location, its address in this strange library, is just the list of its coordinates, one per quality.
You can still picture this. It’s only three dimensions, a point floating in a room. But three isn’t enough. Real meaning needs hundreds of qualities, maybe thousands, and that’s where we have to leave the room behind. Past the third dimension, there’s nothing left to picture: no up, no left, no forward, just more axes, more directions than the mind was ever built to see. The visualization stops helping here. From now on, we hold it in numbers instead.
None of this was ever really about books, of course. A book was just a convenient thing to place, a physical thing we could move relative to other physical things. The same trick works on anything you can describe, a sentence, a phrase, a single word. So let’s shrink it down. Instead of placing Cosmos in the library, we’ll place individual words in the space.
Picture that bank of dials again, except now there are fifteen hundred of them. You’ve already met a few: scientific, inspiring, poetic. But don’t let that tidy list fool you, a dial can measure anything, “formal” and “cheerful” and “how much it has to do with food,”. Keep going, past every quality you could name, into faint shades of meaning no human ever needed a word for.
You can’t picture fifteen hundred dimensions, so don’t try. Just accept that there are around fifteen hundred axes instead of three. Every dial is one axis; its setting is one coordinate. Set all fifteen hundred and a word is pinned down as a long list of numbers, a single exact spot in an enormous space. A list of numbers, as we saw with Dewey, is just coordinates: (3, 8, 2, 5, 6, 9, 0, 6…) and on and on. Imagining is hard. The math is easy.
And here’s what the dials buy you. Two words land near each other not because they share one tidy category, but because their dials broadly agree, this one high on both, that one low on both, enough of them matching that the math pulls the two together into neighbors.
So in this space, “king” and “queen” are neighbors. So are “king” and “monarch.” “Apple” and “orange” share a block; “apple” and “democracy” sit far apart. Position is meaning, exactly as it was on Dewey’s shelf, except a word can now sit near its science-neighbors along one axis and its music-neighbors along another, at the very same time.
That list of coordinates, that address in meaning-space, has a name. It’s called an embedding. Hold onto the word; a later section is entirely about where these numbers come from. (And if you already know this material: yes, I’m crunching several decades of ideas into one tidy picture. We’ll pull them apart. Bear with me.) For now the only thing that matters is the shape of the idea: turn meaning into a location, and “find me things like this” becomes “find me what’s nearby.” The shower trick, rebuilt out of arithmetic.
And now the part that connects all of it to the brain. We did not invent this idea for computers. We found it in ourselves, long before.
In 1987, a cognitive scientist at Stanford named Roger Shepard published a paper proposing what he believed might be the first true law of psychology. And I mean law in the full, physics-like sense of the word: a regularity he expected to hold for any thinking creature anywhere in the universe, humans included. The law was this: The mind represents things as points in an internal space. Whatever you learn about one thing carries over to another in proportion to how close they sit: strongly for near neighbors, weakly for distant ones, fading smoothly as the gap grows. He called it the universal law of generalization, and the word universal wasn’t modest: he applied it to all thinking things, humans, animals and birds alike.
Take his color experiment, the cleanest version of the idea. Ask people how similar various colors are, feed the answers through the math, and the colors arrange themselves into a space, the familiar wheel, red beside orange, far from blue.
Now teach someone that one particular shade of green means food, and watch what they do with a shade just to one side. They treat it almost the same. Nudge the color further and the response fades, smoothly, exactly in step with how far you’ve moved across the wheel. Close colors, treated alike. Distant colors, treated as unrelated. Run the same setup on pigeons and you get the same curve.
Another one of Shepard’s own examples was a bird that has eaten one earthworm and is offered another one. Does it eat it? It depends on how near the new worm sits to the old, in the bird’s internal space. A different-looking earthworm? Close, probably safe. Yum in a slightly new package. A grasshopper? Further away, maybe, maybe not. A rock? Nope. That same reflex, Shepard argued, runs when a child who’s been stung by one wasp flinches at the next striped insect, or when you trust a stranger because they remind you of someone kind. It runs in any mind, anywhere.
In the decades since, Shepard’s theory has held up under a staggering amount of testing, across species, across senses, and most recently across hundreds of thousands of human judgments about images. The pigeons, it turns out, sort the world by the same geometry we do.
Hearing that song in the beginning of this article, your mind didn’t immediately hand you the artist. Your brain lit up a long list of coordinates in a virtual space. A song that refuses to go where you expect it to. Synth textures mixed with real guitars until you can’t tell which is which. A high, fragile voice singing about something beautiful and anxious at the same time. None of those is a name, but together they occupy one neighborhood, and at its center sat the answer to the question your brain was actually asking: who made this?
It must be Radiohead.
All of this happens so fast and so automatically, we don’t really register it. We may be somewhat aware of what made us recognize Radiohead, but the initial intuition is just a feeling. It’s the song being similar to another song in some ways that we can define, and others that are a complete mystery to us.
Looking at the timeline, it’s interesting how technology is catching up with the psychology of remembering. Decades before anyone shipped a vector database, a psychologist looking at how pigeons and people sort the world wrote down its founding principle: meaning is distance. The engineers who eventually built the machine weren’t just dreaming up a clever new data structure. They were, perhaps unintentionally, reverse-engineering the filing system in your skull. The same one that just handed you nostalgia in the shower.
The rest of this piece is the story of how they pulled it off: how you turn a sentence into a point, how you find its neighbors when there are a billion of them, why this suddenly matters so much in the age of AI, and the number of things people get badly wrong about it along the way.
But the basic idea is already in your hands. Similar things go to similar places. Everything else is just learning to read the map.
The idea that waited
If vector databases feel like something that fell out of the sky in 2023, it helps to know that the core idea is old enough to collect a pension.
Rewind to the 1960s. Computers are the size of refrigerators, and you talk to them by feeding in stacks of cardboard punch cards. Into this world walks a man named Gerard Salton, a German-born researcher who had done his doctorate at Harvard under Howard Aiken, the man who built one of America’s first large-scale automatic computing machines. Salton learned his trade just one generation after the birth of the machines themselves. And he had just the kind of unreasonable ambition we’re looking for: he wanted a computer to read a library and tell you which documents were about the same thing. A system that could work all that out on its own.
As a computer science professor at Cornell, Salton led the group that built a system called SMART, and to make it work he took a mathematical leap that still sits under everything we’re discussing.
Think back to the library. A book’s location gave away exactly one thing about it: stand in the astronomy section and the book is about astronomy, one quality, one position on one line. We already saw how to break past that limit, by giving a thing many coordinates instead of one, so it could sit near different neighbors in many directions at once. That was our thought experiment. Salton built the real machine.
Everyone before him treated a document as what it obviously is: a string of words, one after another. Salton treated it as a place. A single point, sitting somewhere in space, fixed by a list of coordinates.
Start by having a machine read through every document in a library, collecting each unique word onto a master list. What you end up with is the complete vocabulary of the collection, say fifty thousand words. Give each of those words its own dimension: X, Y, Z, and 49,997 more, just like in our thought experiment. Now take a single document, and for every word in the vocabulary, count how many times it appears in that document. Here is the goosebump moment: that count is the document’s coordinate on that word’s axis. Meaning, turned into a precise location by something as plain as counting.
But wait, words like “the,” “and,” and “it” appear constantly in every document while saying nothing about its content. This is the second move. Weigh each word by how much it gives away. A word like “the” is in everything, so it tells you nothing. A word like “photosynthesis” is in almost nothing, so when it turns up, it announces what the document is about. Each raw count is scaled by how rare the word is across the whole library, common words shrunk toward nothing, rare words amplified (a method called inverse document frequency, usually credited to Karen Spärck Jones). The system leans on the revealing words and ignores the filler, and the beauty of it is that not a single one of those judgments is hard-coded. It falls out of the math, much of which Salton worked out, and whose bones still run inside Google today.
(I can’t resist the urge to digress here: If you’ve worked with SEO for more than a few years, this will sound familiar. Early web search engines leaned heavily on term frequency: roughly, the more a page used a query word (tempered by how rare that word was), the more relevant it was assumed to be. So people did the obvious, dishonest thing: they crammed “cheap flights cheap flights cheap flights” into the page, often hidden in white text on a white background or stuffed into meta tags, to spike their frequency score and rocket up the rankings. Later versions of Google brought in signals from the outside, like PageRank, scoring a page by how many other pages linked to it, but the practice of keyword stuffing haunted the web for years. Anyway, back to Salton.)
Every document gets exactly fifty thousand coordinates, no matter its length, because every word in the vocabulary is counted in every document, scoring zero in the ones where it never appears. A five-page memo and a five-hundred-page book are described the very same way, as fifty thousand numbers, most of them zero. The memo just has more empty slots. The book, being far longer, racks up bigger counts across the board, which pushes its coordinates higher everywhere. That lopsidedness causes a problem we’ll have to fix later, so hold the thought.
What you’re left with, for each document, is a long row of numbers. And here is the move that turns those numbers into a place. Take the first one, the document’s “photosynthesis” score. Every document has one, so you can line them all up along a single axis, the same left-to-right axis as Dewey’s shelf: the ones steeped in photosynthesis off to the right, the ones with none of it back to the left. Take the second number and you’ve got a second axis, this one running up and down. A third, and the documents lift off the page into a room. Keep going for every word on the list, and each one ends up pinned at a single exact spot in a space of 50,000 directions, its position set by how much of each word it carries.
A document is no longer a wall of text. It’s a dot, hanging at a precise address, surrounded by other dots. And the instant a document is a location, you can do the one thing this entire field is built on. You can measure the distance between two of them. Close together, they’re probably about the same thing. Far apart, probably not. Meaning, for the first time, had a where.
By 1975 he had written the idea down in full, in a paper with about as much swagger as 1975 academic titles allow, “A Vector Space Model for Automatic Indexing.” Strip away the grey prose of the era and the claim is the one your brain was making in the shower: meaning is distance. Depending on who you ask, he was half a century early.
Regardless, he’s by many considered the Father of Information Retrieval, which makes his contributions to the wave of AI enormous.
There was just one problem, and it would haunt the field for the next forty years. You may have already spotted it. Salton’s machine measured documents by the exact words they contained. So a page that said “the king” and a page that said “the monarch” looked, to SMART, like strangers. Same meaning, different letters, different point in space. The idea was right. The execution could only ever see the surface of language, never the thing underneath. It could see that the words were there, but it couldn’t make out their meaning.
Which raises the question that turns out to be the whole game: how do you get a machine to learn that “king” and “monarch” belong together, without hard-coding anything?
The answer had been sitting in a remark a British linguist made in 1957. J.R. Firth, writing about where words get their meaning, put it about as plainly as it can be put:
“You shall know a word by the company it keeps.”
You don’t need a definition of “king” if you’ve seen the word ten thousand times. You’ve watched it sit next to crown, and throne, and reign, and queen, and you’ve noticed that “monarch” likes to sit next to those same words. They keep the same company. They mean nearly the same thing. A machine, in principle, could notice that too, just by reading enough.
For decades this stayed mostly theoretical, with a few honorable attempts along the way. (In the late 1980s a method called Latent Semantic Analysis got partway there, pulling hidden themes out of raw word counts with some heavy-duty algebra.) The bottleneck never changed: not enough text, not enough computing power, not enough of either to let a machine read its way to meaning.
Then, in 2013, a team at Google led by a researcher named Tomáš Mikolov released a tool called word2vec, and the field changed shape overnight.
Why? Because Firth’s recipe could finally run at scale.
Salton’s coordinates were honest, but literal. They tracked how much a document leaned on specific words, and nothing more. No matter how many words you added to the vocabulary, all you could ever get back were documents with similar counts of the exact same words. The system had no sense of how words related to one another. “King” and “monarch” sat on separate axes that never touched. Every word was its own island in the vast space, and finding all documents related to royalty could require many searches combined.
word2vec found a way to learn the company a word keeps. The basic premise was simple. What it needed was an amount of text and computing power that didn’t exist until it did.
Take a staggering amount of text, around a hundred billion words of Google News articles, and hand a neural network one stupidly simple game to play, over and over: cover a word, look only at the words around it, and guess what’s missing. “The ___ sat on the throne.” “She poured a ___ of coffee.” It’s the intellectual equivalent of a slot machine, and somehow, a hundred billion pulls later, it walks out understanding what “monarch” means.
Now watch what it’s forced to do to win. Salton had given each document a position, fixed by the weighted count of its words. word2vec moves the position from the document to the word: every word gets its own list of numbers, and the network is allowed to adjust those numbers each time it guesses wrong. The machine is assigned 300 dimensions, and the only way to get good at the game is to give words that appear in the same kinds of sentences similar coordinates across those 300 axes. “King” and “monarch” turn up surrounded by the same words, thrones and crowns and kingdoms, so to predict well around either one, the network keeps nudging the two toward the same neighborhood. Nobody told it they were related. It placed them close because they kept the same company. Repeating that game hundreds of times would still give you a seemingly random cloud of words. But do it billions of times, and the positions stop being random and settle into a map of meaning.
That map is an embedding, exactly as we described it in the last section, except now drawn automatically from the raw patterns of human writing. Salton needed no dictionary either, but his coordinates only ever knew how often a word appeared. word2vec’s coordinates know what a word means, in the only sense a machine can: by the company it keeps.
Here is where it stops being clever engineering and starts being a little uncanny. Once every word was a point in space, the points turned out to be arranged with a logic nobody had programmed in.
To understand it, we need to extract something from the premise we just set up: if we can identify two points in a space, that also means we can draw a line between them. With 300 dimensions, that’s a little hard to hold in your head, so let’s spend a couple of minutes on it, down in two dimensions, where we can actually draw.
Longitude and latitude are just other names for X and Y: how far east, how far north. So let’s assign two cities a pair of numbers each. I’ll make up short ones to keep it clean:
Oslo: (1, 1)
Stockholm: (4, 3)
Using the coordinates on these two dimensions (east and north), we could place each city at an exact location on a gridded map. Done.
Finding the components to draw a line between them is simple, but bear with me, because it’s also kind of cool.
Start by pretending the cities sit on a single line, one dimension, just east and west. Oslo at 1, Stockholm at 4. What’s the distance between them? 4-1 = 3. Easy.
But a map has a second dimension, so measure that one the same way. North and south: Oslo at 1, Stockholm at 3. The distance is 3-1 = 2.
That gives us the two numbers (3, 2). This is how you find the mathematical components needed to draw a straight line: start in Oslo, and draw the line toward a location that’s 3 points east and 2 points north. Ok, so now we know how to calculate the shortest possible travel route on a map. Just subtract the coordinates from each other. Nothing spectacular, that’s how you read a map.
But it reveals something interesting: when we added a dimension, the recipe didn’t change. We can keep going. Measure the distance on the first dimension, then the second, then the third, all the way to the three-hundredth, and you’ve got a straight arrow in a 300-dimensional space, pointing from one word to another in a direction we can’t see, but can easily calculate. Just subtract the coordinates. The simple grade school map arithmetic still holds up.
So, that little detour helped us do one thing: set up a straight arrow from one point in our space to another, even across 300 dimensions. Just like Oslo and Stockholm, the relationship between words has become directions and distances you could measure.
And here’s the uncanny part that word2vec’s logic revealed: Take the point for “king.” Subtract the point for “man.” Add the point for “woman.” You land almost exactly on the point for “queen.” Geography hides the same trick. Start at the point for France and look at the arrow that carries you over to the point for Paris. That arrow is a specific direction and distance, and in meaning-space it stands for a relationship: “go from a country to its capital.” Now here’s the cool part. That exact arrow, the one you pulled off France-and-Paris, works on every other country too. Start at Italy, follow the same arrow, and you arrive at Rome. Start at Japan, you land on Tokyo. Nobody had told the model that royalty comes in genders or that countries have capitals. It had read a hundred billion words and, in the act of arranging them by their company, accidentally built a map where these human concepts sat at consistent, measurable distances from one another. Meaning had a geometry, and a machine could learn it on its own, from reading the news.
(One coauthor on the paper that showed this off was a young researcher named Ilya Sutskever, who would go on to cofound OpenAI two years later. The thread running from “doing math on words” to “ChatGPT” is closer to a straight line than you’d expect.)
It would be unfair to let the king-and-queen trick stand without a small confession, because it got so famous that it also got oversold. In the strict version of the test, the model is forbidden from answering with any of the three words you fed in. Remove that guardrail and “king minus man plus woman” will often just hand you “king” right back, sitting stubbornly closest to itself. The party trick is impressive, but for now, it is a trick.
Regardless, word2vec’s impact was huge, but its reign was short, as is often the case with emerging technology. Within a few years it was overtaken by an architecture called the transformer, the engine inside every large language model you’ve used, and the thing that produces the embeddings inside almost every modern vector database. It was, once again, the next logical step in teaching a machine to read meaning, and once again you may already have spotted the problem it fixed.
Salton’s machine struggled with meaning because it handled words in complete isolation. word2vec was the step forward: it explained a word by the company it keeps. But it gave each unique word exactly one location, forever, and that was a weakness: a word can mean different things in different company. You withdraw money from a bank, and you also sit on the bank of a river. word2vec didn’t miss that second meaning so much as get polluted by it: the single point for “bank” sat surrounded by both rivers and money, with no way to tell the two apart. What a transformer does differently is read the whole sentence before it places the word, so “bank” beside “river” lands somewhere different from “bank” beside “money.”
(Quick note: Even though the transformers work with whole sentences or even bigger bodies of text, I’m going to keep saying words to keep it simple. The principle is exactly the same, whether we talk about books, words, or sentences.)
The transformer doesn’t know any of this is a problem. It simply notices that a word turns up among wildly different neighbors in different sentences, and produces a fresh position for each, fitted to those neighbors. Linguists do have names for them: when the meanings are unrelated, like a river bank and a money bank, that’s homonymy; when they’re related, like the head of a person and the head of a company, that’s polysemy. The transformer has no clue about this terminology. It just stops conflating the things.
We’ve come a long way, but we haven’t reached the vector database just yet. A vector database doesn’t generate these numbers, it only stores them and finds the nearby ones. Word2vec and its successor the transformer are both embedding models; the machine that turns text into coordinates.
The line from Salton’s punch cards to the chatbot on your phone is one unbroken idea, idling the whole time while it waited for the world to catch up. The blueprint was drawn in 1975. The math to fill it in arrived in 2013. The machines and the ocean of text big enough to run it at full scale showed up a few years after that. Forty years, give or take, from sketch to skyscraper.
But all of this leaves the question we’ve been dancing around for two sections. We keep saying a machine “turns a word into a few hundred numbers.” Fine. But how? What are those numbers, and where do they actually come from?
Where the numbers come from
I owe you a confession about those dials.
When we were turning knobs for “formal” and “cheerful” and “how much it has to do with food”, I let you believe something cozy and false: that somewhere, some person decided what each dial should measure, and then sat down and dialed in Cosmos by hand. As you may have guessed, there is no such person. No committee naming the dials, no librarian assigning scores. I gave you a comforting little fiction so the idea would go down easy, and now I have to take it back, because the truth is so much more fascinating.
So. The thing that turns the dials is the embedding model, the machine we just built in stages: Salton’s word-counter, then word2vec, then the transformer. You feed it a word, a sentence, a whole rambling paragraph, and it hands back the row of numbers that says where the thing lives. Text in, coordinates out.
It sounds trivial, as if we’re simply converting text to numbers, like a cryptographic algorithm. But the process we’re looking at right now is actually one of the things that people find most spooky with AI, and we’ll get to it in a second.
The code of the guessing game itself is almost insultingly simple, a few hundred lines. What it read was a staggering pile of ordinary human writing, and the only thing it ever practiced was a game so dumb it sounds like a joke: cover up a word and guess what goes in the hole. That’s it. The simplicity of that mindless drill is completely disproportionate to the emergent properties that come out of it: an entire universe of meaning.
You cannot get good at that game without absorbing how the language actually hangs together: which words travel in packs, which ones can swap places without anyone noticing, which ones never turn up within a mile of each other. The model has no choice but to give every word a position and nudge words that keep the same company closer together, little by little. But there’s a key thing we haven’t covered.
In the last section, we focused mostly on the proximity of words and sentences. “King” and “queen” are similar because they are close to each other in our multidimensional space. It’s a useful way to spot the emergence of patterns like gender and capitals, but to understand how the coordinates are set, the logic is backwards: “King” and “queen” aren’t similar because they are close. They’re close because they are similar.
And that begs the question: similar how?
Each dimension is a dial, and each dial is one quality. Left-to-right meant “how scientific” a few sections ago, up-and-down meant “how inspiring.” So if a word now lives in a space of fifteen hundred directions, then fifteen hundred qualities are being measured, fifteen hundred dials, each one some specific way that two things can be alike. Is dial 400 “witty”? Is dial 901 “Italian-ness”? Is there one for “menacing,” for “nostalgic,” for “smells like Christmas”?
And this is the spooky part: We mostly don’t have a clue how it comes up with these qualities and what they are.
I want to be precise about how strange that is, because it’s easy to just rush past it. It’s not a computing problem, or the code being so complex we can’t understand it. It’s that we’ve allowed the machine to simply invent ways in which things can be similar, and they can be absolutely anything. Scientists can occasionally spot patterns, such as one dimension tracking past versus present, and another leaning formal against casual, but mostly we’re completely in the dark as to how the machine has conjured up most of these phantom qualities. All we know is that whatever they are, the map they create is breathtakingly accurate. Somehow, the word “king” and the word “queen” can be described in thousands of different subtle characteristics, with such precision that two minds, yours and the machine’s, reaching independently, mostly agree on what’s near what. It may not be that difficult in theory to imagine describing “king” in that many qualities, but remember the lesson from Salton’s dictionary: Every word is described with the same set of qualities. There aren’t 1,500 “kingly” dials. There are 1,500 dials that, between them, have to capture every word in human language, “king” and “Tuesday” and “photosynthesis” and “grief,” all scored on the very same axes.
Take “sarcasm.” Is there a sarcasm dial in there? Maybe. More likely there isn’t one anywhere, and sarcasm is instead a quality that emerges from many different dials, the way no single instrument in an orchestra is the sadness of a piece of music, and yet the sadness is unmistakably there. The quality is real. It changes where the word sits. But we may never know what it is.
We’ve talked about the number of dimensions quite a bit by now, so let’s inspect where these numbers come from. When you ask one of OpenAI’s standard embedding models to place a scrap of text, it does not return three coordinates, or ten, or fifty. It returns 1,536 of them (Its larger sibling returns 3,072), each one nudged to capture some sliver of what your sentence means.
The machine has invented qualities we have no words for, qualities we likely couldn’t be given words for, because the distinctions are too subtle, too entangled, too far below the resolution at which human language operates. If the machine could somehow explain to you what dial 738 measures, it may not mean anything to you at all.
And then it does the thing that should not be possible. It doesn’t just detect that nameless quality. It measures it. It decides how much of it a given word carries, and writes that down as a number. Whatever ineffable thing dial 738 is responding to, the model will tell you, with a straight face, that “king” scores 0.81 on it and “Tuesday” scores 0.12.
(If you build these for a living: yes, there’s a second stage after the guessing game where models get fine-tuned for specific jobs. It refines the space; it doesn’t create it. This too may be the topic of another piece, but the meaning is already there by the time fine-tuning starts.)
Now, you might object, and you’d be right to. This is all just arithmetic. Couldn’t it be that the dials don’t mean anything at all? The model never set out to measure qualities. It set out to win a guessing game, and 738 is simply a number that, alongside all the others, happened to help it win. And yes, from the machine’s side, that’s exactly true. There is no little “sarcasm” concept living in there, no idea of royalty, the model doesn’t know anything. It didn’t invent fifteen hundred qualities and hide them from us. It encoded meaning as a geometric structure, dimensions with no names or descriptions, no concepts, no categories, only distances and direction. It didn’t build that structure to understand human meaning. It needed that structure to be able to win the game, and so it assembled itself in billions of tiny adjustments.
It’s tempting then, to say that both things are true at the same time. Each dial has no label, no single quality pinned to it. And yet it seems like it must, a little, because when you combine enough of them, words with similar meanings get pulled together, so surely each coordinate is describing that meaning just a little bit. But that’s the paradox. No coordinate describes the thing at all, not even a little. The meaning lives in how the points sit relative to one another, in the distances and directions between them, never in the numbers themselves. You could rotate the whole space, which would change every single coordinate of every single word, and lose nothing, the way sliding a map’s grid lines around doesn’t move a single city.
The map works. We just can’t read the legend.
If this sounds like some glitch in the machine that we’ll have to fix, remember that we do the same thing.
Go back to the song. Four seconds in, you knew it was Radiohead. How? You could maybe scrape together a few obvious clues, like Thom Yorke’s voice. But remove that, and you might still recognize it. You felt a Radiohead-ness, a quality with no name, assembled out of who-knows-how-many impressions too small and too fused to ever pull apart. You named a clue or two, and then ran out of words, and knew anyway.
The same is true for sarcasm. Some sentences seem sarcastic simply because the person who spoke it happens to be sarcastic a lot of the time. Other times, you may be unsure if it was sarcastic or not, and that ambiguity could itself be a quality that’s difficult to name.
That is exactly where the machine lives, its internal space built not from life experience but from the guessing game.
And this is the whole apparatus, finally assembled. Every word, every document, every product and song and grumpy support ticket you own can be handed to the embedding model and given an address. Picture them all poured into the room at once: hundreds of millions of points, each one coming to rest near the things it resembles. There is your map of meaning, drawn at last. And it leaves exactly one question, the one that decides whether any of this is worth the trouble. When someone walks up and asks for “the things near this one,” how do you find them, fast, with a billion points in the room?
Finding what’s nearby
And now, at last, we reach the vector database itself. Everything up to here has been about turning meaning into coordinates, the embedding model that places each word, and the geometry that lets us measure which placements sit close. The coordinates are pinned down. What we haven’t built yet is the thing that holds them, millions of them, and can find the nearest neighbors to any point without checking all million one by one. That’s the database’s real job, and it’s where we’re headed now.
We’ve focused mostly on single words: king, queen, apple, to keep things simple. But the modern machines, as we discussed, don’t stop at words. You hand them a whole sentence, a paragraph, a support ticket, and they place the entire thing as one point, according to what it’s about. So let’s bring some longer strings into this to illustrate just how much meaning can be hidden between the words and lines of a spoken sentence.
What a thing means is a slippery, unspoken business, nothing like a dictionary definition. Take the question “who is the king of rock and roll.” The word king is sitting right there in it, but it has almost nothing to do with the answer. The sentence isn’t pointed at royalty or crowns. It’s pointed at music, and the 1950s, and Memphis, and a curled lip, and a hundred other associations it never says out loud. Elvis is nowhere in the text, yet the sentence is practically screaming his name. That sentence and Elvis share a neighborhood. That implicit, unwritten aboutness, the stuff a sentence evokes without naming, is what these coordinates actually capture. Not what the words say. What they summon.
So before we can find the nearest point, we have to say what “near” even means when your location is a list of fifteen hundred numbers. On Dewey’s shelf it was easy. Near meant a smaller gap between two numbers on a line: two books close to each other. You pick up Cosmos, and other books about cosmology are close to it. Up here with fifteen hundred numbers it’s almost as easy, except for the part where you have to mentally visualize billions of coordinates in a more-than-three dimensional space.
Because of that, we’re gonna tune down the numbers again, and just look at two things.
Remember how we found the move between two points by subtracting their coordinates. We’re about to lean on that same picture, with one shift in how you look at a point.
So far a point has been a dot: Cosmos sits here, “king” sits there. But a point’s coordinates are also a set of instructions for getting to it from the center of the room: go this far along the first axis, this far along the second, and so on.
To picture that in three dimensions, think of the solar system. Freeze everything in place, the sun at the center, the planets hanging motionless around it. Each planet has coordinates that fix its exact spot in that frozen room. But because the sun sits at the center, you can also just point a straight arrow from the sun out to any planet. Earth becomes an arrow. What changed? We used the sun as a reference point; somewhere you could travel from, in a specific direction and a specific distance. Once you have a center, every location becomes a direction and a distance away from it.
Two ideas are similar when their arrows aim the same way. “King” and “queen” are two arrows pointing in nearly the same direction. “King” and “tax form” point off toward different walls. So how do we measure that difference? We measure the angle between two arrows: a sliver of an angle means almost the same direction, and almost the same meaning. A wide angle means the two have little in common.
But wait, why the angle? Why not just measure the plain distance between the two points, the way we have all along? Here’s the catch, and it’s the whole reason this works.
Return for a second to Salton’s word counter, where each dimension is a word like “photosynthesis” and the coordinate is how many times that word appears in a document. Remember the lopsidedness we spotted: a long document might say “photosynthesis” twenty times and a short one twice, but that doesn’t make the long one ten times more about photosynthesis. The raw count is inflated by sheer length. Measure the straight-line distance between the two and you’d call them strangers, punished for nothing but being long or short. But the direction they point stays the same. So we throw away the distance and keep only the direction. Two arrows aiming the same way mean the same thing, no matter how far each one happens to extend.
You might wonder how this squares with word2vec and the newer embedding models. If each dimension is a quality-score, not a count that piles up, then why would one word’s arrow reach farther from the center than another’s at all? Shouldn’t every word land at roughly the same distance?
In most modern models, it does, and not by accident. Before any comparison happens, the model rescales every vector to the exact same length, a step called normalizing. Magnitude (one arrow’s reach from the center) is erased on purpose: every arrow pushed out to the same distance from the center, so the only thing left to tell two words apart is the direction they point. That’s why we reach for the angle. In a 3D space, all the points are scattered on the surface of a sphere, all at equal distance from the center.
But normalizing technically means throwing away data. Magnitude isn’t always meaningless, and once you’ve normalized, whatever it carried is gone. In Salton’s case, normalizing his vectors would erase how long a document is, keeping only the proportions of which words it favors. For comparing topics, that’s exactly what you want. In other tasks the raw magnitude is a signal you care about. For example, if you’re analyzing data from an eCommerce website, two carts could have the same mix, 60% groceries, 40% electronics by value, pointing the same direction, but one totals $50 and the other $5,000:
Are these similar shoppers: direction works.
Who’s the high-value customer: magnitude is the signal.
This is why vector databases often let you choose your measuring stick: cosine for direction, plain distance when size matters. Cosine is the sensible default for meaning, not the only option. Its real advantage is that the same calculation behaves the same way across almost any vector-based system, which is much of why it became the standard.
That measurement, the angle between two arrows, has a name you may have met: cosine similarity. And now you know exactly what it’s doing and why. The name is less intimidating than it sounds. “Cosine” is just the cosine from school trigonometry, that you were promised you’d use one day and then never did. It turns out you use it constantly, because it actually runs half the search on the internet.
One thing that’s easy to miss: the cosine isn’t the angle itself, not “30°.” It’s a single decimal between −1 and +1.
The picture turns that angle into a score, shown at the top. A 30° gap, fairly close, comes out around 0.87 on a scale that runs from 1 at the top down to −1 at the bottom. The three landmarks on that scale are worth holding onto:
Two arrows pointing the same way score near 1, near-identical in meaning
Two arrows at a right angle score exactly 0, meaning unrelated, little to do with each other
Two arrows pointing dead opposite score −1. In practice you almost never see this with real language. Opposites like “hot” and “cold” actually sit close together, because they show up in all the same contexts. The −1 end is more the floor of the scale than a place words actually land.
This is the single measurement underneath everything in this post: a way to ask “how alike are two things?” and get one number back.
So, two arrows starting from the same center, the origin at the bottom: Vector A and Vector B. Think of them as two ideas, say “who is the king of rock and roll” and “Elvis,” each pointing off in its own direction. The only thing that matters is the angle between them, marked θ (”theta”).
That is the whole concept in one line: similarity is direction.
With two things, we can see two tidy lines and measure their angle. With 1500 we are adding arrows in directions that, to a human, don’t exist. But mathematically, they can exist just fine. We can add as many dimensions as we want, with axes pointing towards directions that we have no way to envision or explain. The machine doesn’t try either. It has no idea about the visualization we just explored, it’s slavishly following the math to understand similarity.
So, up until this point we’ve been comparing two arrows, A and B, as if that were the task. It isn’t. Remember what’s actually in the room: every document you loaded, every product, every paragraph of your company’s help center, each one already sitting there as its own arrow. When a question arrives, it becomes one more arrow, and the machine has no idea which of the stored ones it’s closest to. Keep in mind, there’s no actual space to travel, and we have to rely on mathematics. The only way to be sure it found the nearest is to measure the angle to the first stored arrow, then the next, then the next, all the way through. Two arrows was the illustration. A billion arrows is the job.
This is where it all falls apart.
Comparing your query against a single stored point means grinding through all fifteen hundred numbers to work out the angle. Do that against a billion points and you’ve performed well over a trillion tiny arithmetic operations to answer one question. A real service is fielding thousands of questions a second. The exact check-every-point method would leave your user watching a loading spinner long enough to grow a beard. Storing a billion vectors is the easy part; any hard drive will swallow them without complaint. Finding the right handful before the user gives up and closes the tab is the part that turns a pile of vectors into a vector database. The searching is the entire product.
So the natural thought is: don’t check everything, be clever about it, the way a map is clever. When you search for coffee near you, Google Maps does not measure your distance to every café on Earth. It knows roughly where you are and only bothers with the ones nearby. Surely we can chop our room of meaning into neighborhoods the same way and only search the right one.
We can try. But to see why it’s hard, we have to take a short walk into one of the stranger corners of mathematics, the place where our cozy three-dimensional intuitions go to die.
You see, space gets kind of crowded when you add dimensions. This fact, once it’s pointed out, feels like something you already knew. We’ll call it neighbor explosion.
Think back to the single X axis, one dimension: a point has just two neighbors, left and right. Move to a 2D map and the neighbors jump to eight: left, right, up, down, and the four corners. Move to 3D and it leaps to twenty-six, picture all the little cubes packed around the center of a Rubik’s cube. The count isn’t creeping up. It’s exploding, and we’re only at three dimensions. At four each point has 80 neighbors. By ten dimensions a single point has more than fifty thousand neighbors. By the time you reach the fifteen hundred a real embedding uses, the count is a number with hundreds of digits, the kind of number that has no name and no business existing. Every point is surrounded by a crowd larger than the universe has particles. “Neighborhood” has stopped meaning anything.
This is the curse of dimensionality (coined by Richard Bellman back in 1957): a point has neighbors in so many directions at once that you can’t carve the space into a few tidy cells and check only the nearby one. There are too many nearby cells. The shortcut dies, and you’re back to checking everything. Boom. An explosion of neighbors. So the field did what engineers do when perfection is impossible and the deadline is real. It cheated, beautifully.
Do you remember the claim that you’re only six handshakes from anyone alive? That you could reach a fisherman in Portugal or the prime minister of Japan through a chain of just six people, each one knowing the next? It has a name, six degrees of separation, and it came out of an experiment in the 1960s where a psychologist named Stanley Milgram asked people to get a letter to a stranger on the other side of the country, with one rule: you could only mail it to someone you knew personally, who would mail it to someone they knew, and so on. The letters that made it through had passed, on average, through about six pairs of hands. A country of two hundred million people, crossed in six steps.
The exact number has been argued over ever since, and you’ll see smaller ones thrown around now. But the shape of the finding is solid, and it is the entire secret to how a vector database searches.
When the database first loads its billion points, it does something patient and one-time: it connects each point to its most similar points and writes them down. Every point ends up holding a short list of links, usually 16-64 (64 being more accurate, but requiring more memory), pointing to the handful of points that sit closest to it in meaning. “Cosmos” links to a few dozen books nearest it. “King” links to “queen,” “monarch,” “throne,” and a collection of others. Think of it like social media linking friends. You may be familiar with traditional databases building indices to speed up searching, and this is the vector database equivalent. It’s done after the embedding process, and before anyone can search, to solve the neighbor explosion problem.
Now someone searches, and here’s the walk, one step at a time.
You start at one point and look only at its contacts, the few dozen it’s linked to. You measure your query’s angle to each of them. One of them is closer to the query than the rest, so you walk to it. Now you’re standing somewhere new, and you do the identical thing: look at its contacts, find the one closest to your query, walk there. Every hop carries you nearer to where your query belongs, the way each handshake in Milgram’s chain carried the letter closer to the stranger.
You keep hopping, and you stop on a dead-simple signal: when you look at your current point’s contacts and not one of them is any closer to the query than where you’re already standing, you’ve arrived. There’s nowhere better to step.
Tally it up and the walk has touched a few hundred points, maybe a few thousand. Never the billion. You followed a chain of introductions straight to the right neighborhood and never wasted a glance on the overwhelming majority of the database, the same way a letter crossing a country never passes through more than a handful of the people living in it. It just follows the chain.
One question hangs over all this: where do you take the first step? Start somewhere random and you’d waste an age just trudging across the space to reach the right area.
The database handles it the way you’d find a single house in a country you’ve never set foot in. You don’t inspect every house on Earth. You start with the globe and pick the continent, then zoom to the country, then the city, then the street, then the door. Each zoom throws away almost everything and hands you a smaller, truer patch.
The network is built in exactly those layers. A sparse top layer holds only a few points, joined by very long-range links, the globe, a few far-flung places and the great distances between them. Below it, more points and shorter links. Below that, more again. At the very bottom lies every point, joined to its immediate neighbors by the shortest links of all. A search drops in at the top, takes a few giant strides across the whole space, lands in roughly the right region, then falls a layer and repeats with finer steps, then falls again. By the time it reaches the bottom, the floor where every point lives, it’s already on the right street and need only knock on a few doors.
A billion points, searched in a few hundred glances.
The dominant search method, which goes by HNSW, or Hierarchical Navigable Small World, wires the whole room of meaning into exactly that kind of network. It takes the long-range links first to cross the room in a few enormous strides, then the short local ones to home in.
That’s the trick, end to end. Turn meaning into points, measure nearness by the angle between arrows, and lace the points into a small-world social network so you can sprint to any group of friends in a few strides instead of trudging past the entire population.
Which, at long last, is a working machine. We can take almost anything, give it an address in meaning-space, and instantly pull its nearest relatives out of a sea of billions. A marvelous gadget. And for something like forty years, outside of search engines and recommendation systems, it was a marvelous gadget without a moment, a beautifully engineered answer waiting for the right question. Then the question showed up wearing the face of a chatbot, and the modest little nearest-neighbor engine turned out to be the missing organ that could give an artificial intelligence the one thing it had never had: a memory. That’s next.
Giving the machine a memory
A large language model is the most well-read entity that has ever existed, and also one of the most forgetful. It has absorbed a sizable fraction of everything humans have ever written, which is why it can write poetry, debug your code, and explain the French Revolution in the voice of a pirate. But two things are missing, and they happen to be the two that matter most for real work.
It knows nothing that happened after its training was frozen, the cut-off date when its reading stopped. And it knows nothing about you. Not your company, not your customers, not the document you wrote yesterday, not the private wiki your team has been building for six years. None of that was in the books it read, so as far as the model is concerned, it does not exist.
As we all know, the dangerous part is what it does with that ignorance. Ask it about something it doesn’t know, and it will produce a confident, fluent, often entirely plausible answer that happens to be invented. The industry calls this hallucination, which sounds almost whimsical. In practice it means the model will happily instruct your customer to click a button in your app that has never existed, in a tone of perfect authority.
So, you have a brilliant generalist that makes things up, and a pile of private, specific, current knowledge it has never seen. You would like to introduce them.
An obvious move is to tell the model everything it needs by pasting it straight into the question. For a paragraph or two, that works. But your company’s knowledge is not a paragraph. It could be thousands of help articles, support threads, PDFs, and policies, and you can’t staple all of it to the front of every question. It is too much to fit, too slow to chew through, and you would be paying to re-send your entire company each time someone asks where the logout button is.
The other obvious move is to teach the documents to the model permanently, the way the LLM learned everything else. But that kind of training is hugely expensive, complicated, and goes stale the instant you edit a page. Retrain the entire mind of a machine because you changed your refund policy? No.
The fix that actually won is a lot easier. Don’t put everything in the question, and don’t retrain the model. Instead, the moment a question arrives, look up the handful of passages from your own documents that bear on it, and hand just those to the model along with the question. Then let it answer from what it was given. The model supplies the language and the reasoning; your documents supply the facts. The arrangement has a name that sounds like a burlap sack, RAG, or retrieval-augmented generation, and it is the reason a chatbot can abruptly speak fluently about your specific business.
And now the two halves of this piece snap together. That lookup, “find the handful of passages most relevant to this question,” is not a new problem. It is the precise thing we spent the last section building. It is nearest-neighbor search in a room full of meaning. The memory we are handing the machine is a vector database, and the gadget that spent forty years waiting for a reason to exist has just found one.
Watch it work, end to end, with a single question.
Long before any customer shows up, you do the prep, once. You take your help center and chop it into bite-sized pieces, a paragraph or so each, because you want to pull back a precise passage later, not a fifty-page manual. Each piece goes through the embedding model, which turns it into its arrow, its address in meaning-space. You store every arrow in the vector database, the original text tucked alongside it. One of those pieces reads, “To cancel your subscription, open Billing, choose Manage Plan, and select Cancel.” It now sits at one specific point in the room, ringed by its neighbors: other passages about accounts, payments, and plans.
Now the customer arrives, irritated, and types: “how do I stop getting charged?”
Look at those words, because they are the whole game. Not one of them appears in the help passage. No “cancel,” no “subscription,” no “billing.” A keyword search, the typical database search I described in the opening, returns nothing, and the customer leaves angrier than they came. But we’re not matching words anymore. The question runs through the same embedding model, becomes its own arrow, and that arrow points almost exactly where the cancellation passage already sits, because stopping the charges and cancelling the subscription are the same idea wearing different clothes. The search hops across its small-world web, lands in that neighborhood, and lifts out the passage. Meaning found what spelling never could.
Here is the last step, and the one people most often smear together with the rest. The vector database has now done its entire job. It found the relevant passage and handed it over. It cannot write, it cannot reason, it cannot talk to your customer, and it never pretended otherwise. What it returns is a chunk of your own text, nothing more.
That chunk is now placed in front of the model with an instruction roughly like: here is a user’s question, here is a relevant passage from the company’s help center, answer the one using the other. And the model, which a heartbeat ago knew nothing about your billing screen, reads the passage and writes: “No problem at all. To stop the charges, open Billing, choose Manage Plan, and hit Cancel. That ends your subscription right away.” Fluent, warm, and resting entirely on a fact it did not hold until half a second earlier.
And that is the whole of it. Two machines with no knowledge of each other, introduced at the last possible second by a third. The vector database is doing the thing your own memory did in the opening paragraph of this piece, when a song or a smell surfaced the one related thing you needed out of everything you had ever stored. We have handed the amnesiac genius a notebook it can flip through at the speed of thought.
It is worth knowing the notebook was finished before the world urgently wanted it. Pinecone, the company that did the most to turn vector search into something an ordinary builder could switch on without a research team, was founded in 2019, years ahead of the ChatGPT moment that made everyone crave exactly this. The answer was sitting on the shelf, complete, waiting for the question to walk through the door.
That is the clean version of the story. The moment you try to build one of these yourself, you meet the messier one: a whole catalog of things that almost everyone, including a lot of people who should know better, gets wrong. That is next.
The misconceptions
For a technology that comes down to “similar things sit close together,” vector databases generate a remarkable amount of confusion. Most of it traces back to a handful of misunderstandings, and they are worth walking through, partly so you avoid them, and partly because each one, turned over, reveals something true about how the whole thing actually behaves. Here are the ones that cost people the most.
It does not understand anything. This is the big one, the misconception the rest grow out of. After watching the chatbot field a question it had never seen, it is tempting to feel that something in there comprehends. It doesn’t. The system measured proximity between two arrows. That is the entire act. “Stop getting charged” landed near the cancellation passage not because a mind grasped the customer’s intent, but because the patterns in billions of sentences put those phrasings in the same region of space. It is closer to an extraordinarily sophisticated reflex than to thought. This matters because the instant you believe the machine understands, you start trusting it where it has not earned it, which leads directly to the next two.
A vector database is not an LLM. The two are easy to confuse, because they share machinery: both lean on embedding models, both turn text into coordinates, both were trained on language. But they do opposite things with it. A vector database is a retrieval tool. It holds a fixed pile of vectors and, when you ask, hands back the ones already sitting closest to your query. It never invents anything; it finds. An LLM is a generation tool. It doesn’t look up your answer in a stored collection, it writes a new one, word by word, every time you ask, even if you ask the same thing twice. One is a librarian who points you to the right shelf. The other is a writer who composes a fresh reply on the spot. They’re often used together, the database fetches relevant material and the LLM writes a response using it, which is exactly why people blur them. But the database finds; the model writes.
Similar is not the same as true. A vector search returns what is nearest, and nearest is not a synonym for correct. If the closest passage to a question happens to be wrong, outdated, or sarcastic, that is what comes back, ranked first, wearing the same confident face as a right answer. The machine has no notion of truth; it has a notion of proximity, and it cannot tell the difference between “the most relevant thing” and “the most relevant correct thing.” Retrieval narrows the model down to relevant material. It does not, on its own, make that material true.
You cannot mix embeddings from different models. Every embedding model builds its own private space, with its own invented dimensions, in its own arrangement. A coordinate from one model is gibberish to another, the way a grid reference on a map of Paris means nothing on a map of Tokyo. Two consequences follow, and both bite hard. First, your documents and your incoming question must go through the same model, or you are measuring distances between two unrelated spaces. Second, the day you decide to upgrade to a better embedding model, every coordinate you have ever stored becomes worthless, and you have to re-embed your entire archive from scratch. People discover this halfway through a migration, which is the worst possible time.
You very often do not need a dedicated vector database at all. Around 2022 a belief took hold that doing any of this required signing up for a specialized, separate vector database product. For a great many projects, it doesn’t. The plain, boring database you may already be running can very likely do it. Postgres, the workhorse relational database that has run the world for decades, gains full vector search through a free add-on called pgvector, which is what Supabase can use under the hood. If you are storing a few hundred thousand passages, not a few hundred million, bolting vectors onto the database you already have is frequently the wiser move than standing up a whole new piece of infrastructure to maintain. The dedicated products are genuinely excellent at enormous scale. Most projects never reach that scale.
Pure vector search is strangely bad at the literal. This is the one that surprises people most, because it is the exact opposite of the weakness they expected. Semantic search is brilliant at meaning and clumsy at specifics. Ask it for order number 5592-B, or a customer named Mr. Fielding, or the precise term “Section 4.2,” and it may hand you something that feels related while missing the exact string you needed, because to the machine “5592-B” and “5593-B” are nearly the same point in space, near-identical in meaning, catastrophically different to a human trying to track a package. The fix is not to abandon vector search but to pair it with the old keyword search we spent the opening teasing. Run both, the semantic one for meaning and the literal one for exact matches, and blend the results. The industry calls this hybrid search, and the lesson inside it is humbling: the clever new meaning-machine works best with a hand from the dumb old word-matcher it was supposed to replace. Neither is whole without the other.
There is no space. We’ve spent a large chunk of this post building up dimensions, coordinates, arrows, and the room they live in. It’s worth saying plainly, as we’ve hinted along the way: none of that is literally there. There is no room, no arrows, no distance. There are lists of numbers and arithmetic over them, and every spatial word in this post, “near,” “direction,” “corner”, is a picture we painted to make the arithmetic graspable. Here’s the part that keeps it honest, though: the picture isn’t wrong. The numbers behave exactly as they would if there were a space. When we say two ideas point the same way, the machine really is computing a number that rises and falls precisely as an angle in a real space would. So the space and the math agree perfectly. The machine just never gets the picture. There’s a kid in a certain movie who explains that the spoon doesn’t bend because there is no spoon. This is what he was talking about. The machine isn’t picturing meaning laid out in three dimensions, or fifteen hundred; it’s grinding through arrays of numbers, the green rain of code, with no sense that any of it stands for anything at all. The qualities we named, the angles we drew, the neighborhoods we walked, that’s all us, looking at the numbers and finding meaning in them.
Notice the shape of the list. Nearly every error is a version of the same root mistake, granting the machine more mind than it has. It does not understand, it does not know truth, it does not even hold a single universal space that everything can share. It measures distance, narrowly and literally, in a space some other machine built. Keep that deflationary picture in your head and the failures stop being surprises and start being predictions.
Which is the right frame of mind for the last practical question. Suppose you want to build one of these. What does that actually take, and what does nobody tell you until it hurts? That is next.
Building one in the real world
Here is the part that should cheer up anyone who has read this far with a growing sense of dread. You do not build most of this.
The two hardest pieces, the embedding model that turns text into coordinates and the search engine that races through billions of them, are exactly the parts you rent rather than make. They arrive as a service you call or a library you install, built by people with doctorates in precisely this and nothing else. Your job is not to invent the machine. Your job is to feed it well. And the decisions that actually determine whether your system is brilliant or useless are not the glamorous ones. They are the plumbing. There are three worth your attention, and the first decides everything.
Before a single document goes into the room, you have to cut it into pieces. We waved at this back in the walkthrough, “chop it into bite-sized passages,” as if it were obvious. It is not obvious. It is the single biggest lever on whether the whole thing works, and almost everyone gets it wrong on the first try.
Remember that one chunk becomes one arrow, one point in space, so the size of your chunks decides what a point is even able to mean. Cut them too large, a whole ten-page document as a single chunk, and you are asking one arrow to stand for ten pages about a dozen different things. The embedding model does the only thing it can. It averages. The result is a point sitting at the bland center of all those topics, near none of them in particular, the way averaging the map coordinates of every city you visited last year drops a pin in the middle of the ocean. Ask a sharp question and that blurry average never surfaces, because it is not really about anything. It is about everything, which is the same as nothing.
So cut them smaller. But cut too small, down to a lone sentence, and you slice through the context that made the sentence mean something. “It costs nine dollars a month” is a useless chunk. Nine dollars for what? Stranded from the paragraph around it, it points somewhere unhelpful. The craft, and it is a craft, is to cut along the seams of meaning: roughly a passage per idea, often with a little overlap so a thought that straddles the boundary is not severed in half. Get this right and the rest mostly takes care of itself. Get it wrong and no database on earth, however expensive, will rescue you, because you blurred the meaning before it ever reached the map. If you take one practical thing from this piece, take this: the magic is real, but it lives or dies on how you slice the bread. I realize we have now compared this to a library, a control panel, a solar system, a map, a chain of handshakes, and a loaf of bread. Stay with me.
The second decision is what you store beside each arrow. A vector database does not have to hold only coordinates. Next to each point you can tag whatever you like: which document it came from, when it was written, which customer it belongs to, whether it is published or still a draft. Then, when you search, you can demand both things at once: find the passages nearest in meaning, but only the ones from this user, written this year, marked public.
This sounds like a footnote and is in fact load-bearing, for one reason above all. Permissions. A pure meaning-search has no conscience. Ask it for the passages most relevant to “executive salaries” and it will gladly return the confidential ones, because confidential is a human idea, not a geometric one, and the arrows have no notion that they are secret. If your app serves more than one customer, the filter that restricts a search to this user’s own documents is not a nice-to-have. It is the wall between a working product and a data breach with your name on it. Meaning lives in the coordinates. Permission lives in the tags. You need both, and you must never let the first run without the second.
Which leaves the bill and the upkeep, the parts that never make it into the demo. Turning text into embeddings costs money, priced by the amount of text. For a few thousand documents this is lunch-money territory, fractions of a cent per page, but for enormous amounts of content, the cost can be real. Speed, by contrast, is rarely the database’s fault. The nearest-neighbor search is the quick part, milliseconds even across millions of points. When a chatbot feels sluggish it is almost always the language model composing its reply, not the vector search finding the passage. And there is the standing chore of keeping the index honest as the documents drift: every edited help article is a chunk that has to be re-embedded and swapped back in, or the machine will keep confidently serving last year’s answer.
None of this is an argument against building one. It is an argument against building a bigger one than you need. The most common mistake here is not technical at all. It is reaching for a planet-scale, distributed, dedicated system in order to put two hundred PDFs in front of a chatbot, and then drowning in machinery you will never use. Build the smallest version that answers your real question. You can always make the room bigger.
Which brings us, at last, to the honest reckoning. We have a machine that gives meaning an address, finds its neighbors in a sea of billions, and lends an amnesiac genius a memory. It is one of the most useful tricks software has picked up in a decade. So it is worth asking plainly, with the sales pitch switched off, what it still cannot do, and where all of this is headed next.
The map is not the territory
We have spent this whole piece celebrating a map. It is time to remember that a map is not the place.
An embedding is an astonishingly rich description of meaning, and it is still only a description. It captures how things resemble one another, and resemblance, for all its power, is a narrower thing than understanding. This is the deepest limit of the whole technology: it is not intelligent by any means, and doesn’t try to be. All it does, and I mean all, is to take one piece of data, and find data that is similar to it.
A vector database can only ever hand you what looks like your question. That is miraculous when the answer resembles the question, which is a lot of the time. It is useless when the answer doesn’t. Ask “is this contract safe to sign,” and the passage you truly need might be a dry clause about indemnity that doesn’t resemble the question at all. The thing that connects them is reasoning, and reasoning is exactly what the map leaves out. Nearest-neighbor search finds the look-alike. It is blind to the consequence, the implication, the fact that sits three logical steps away and shares not one word with what you asked. Your own mind makes those leaps constantly. The map cannot, because a map only knows what is near, never what follows.
And there is a heavier inheritance. This map of meaning was not drawn from the world. It was drawn from our words about the world, billions of them, scraped from nearly everything we have ever written. So it learned the world exactly as we describe it, prejudices and all. Researchers found early embeddings finishing “man is to computer programmer as woman is to” with “homemaker,” not because it is true, but because that is the shape of how we have written for a century. The space is a mirror, and it makes no attempt to flatter. Every bias folded into human language is folded into the geometry, sitting there as distance, and a machine that retrieves by resemblance will faithfully retrieve our blind spots alongside our knowledge.
So what does the future look like? Two directions, both already underway.
The first is the exaggerated predicted death of vector databases. Every time language models learn to hold more text in mind at once (and the amount has grown enormously), someone announces that retrieval is now obsolete: just hand the model everything and skip the lookup. It never quite takes. Feeding a model your entire company on every question is slower, costlier, and less accurate, since a model swimming in a hundred documents tends to lose the one that mattered somewhere in the middle. The goal of the vector database is to feed the LLM something that counts, instead of feeding it everything. A higher context window simply makes “what counts” bigger: it doesn’t suggest you should feed it an ocean of text where 99% is irrelevant to the user’s prompt. Maybe future technology will reveal an even better way to provide context, but we’re not there yet.
The second direction is the more beautiful one. Nothing about this trick was ever really about words. The same machine that gives a sentence an address can give one to a photograph, a song, a face, a heartbeat traced on a monitor. Models already exist that drop a picture of a sunset and the phrase “a sunset over water” at nearly the same point in space, because they mean the same thing, and meaning was always the coordinate, not language. The map is widening to hold everything the senses take in. We are heading toward a single space where a hummed melody can find the song, a snapshot can find the words for it, and a feeling can find its name.
Which lands us back where we began, in the shower, when the word you wanted walked up on its own.
You found “nostalgia” the same way this machine finds a forgotten billing passage: not by its spelling, but by its neighbors, by everything sitting close to it. We did not invent that trick in a laboratory. We copied it, clumsily and at enormous expense, from the only device that ever ran it natively, the one behind your eyes.
Roger Shepard, watching how pigeons and people sort the world, guessed in 1987 that meaning is distance, and that the law might hold for any mind anywhere in the universe. Decades later we built a mind of our own, out of silicon and arithmetic, and it obeys his law too. Not because we forced it to, but because it seems there may be only one good way to organize meaning, and a hundred million years of evolution and forty years of engineering both groped their way to the same answer. Similar things go to similar places. The brain worked that out slowly, in the dark. We just did it again, on purpose, in the light.
So the next time a chatbot answers a question about your own files, or a song you have never heard aches like one you love, or a lost word ambushes you under the hot water, you will recognize it for the single trick it is. A thing was turned into a place, and the places nearby were near in meaning. That map has been running in your head your entire life. The wonderful news of the last few years is only this: we finally learned to draw it down on paper.
And if you have read all the way to the bottom of an entire essay about the geometry of meaning, then you already feel the pull of it. You are, like me, the kind of person who finds it a little marvelous that “the king of rock and roll” and “Elvis” should live next door to each other, and that we have, at last, built a machine that can walk the short distance between them.
If you liked this post, you may also be interested in my next AI article: Why Does an AI Never Repeat Itself?
Petter, this deserves many, many more reads.
I’ve been on my own self-study path of looking under the hood of LLMs (it seems we are both never content to just use these technologies for their practical purpose). This article is one of the best non (semi?) technical breakdowns I’ve read (I know LLMs =! Vector databases but they share the same concepts).
Such a brilliant combination of history and metaphor and plain-good story telling.
It must have taken you an age! Hats off to you!
Thanks Petter for such an amazingly thorough and careful breakdown. I’ve often felt that the word ‘intelligence’ in AI is actually a disservice.
It’s great for marketing but (as your analysis clearly shows) the underlying process in AI systems is very different from what we’d call intelligence - intelligence isn’t about being trained on trillions of bits of data to determine proximity. Rather, intelligence is the ability to generate new data on often very little data - an imaginative leap as it were. It is the ability to cope with situations where there is very little pre existing data.
I use the word ‘disservice’ for AI because by linking it to intelligence we set it up to fail. If we recognised that it is more akin to memory - I like to think of it as a logical search engine, where are searching abstract logic rather than for things - then we can really understand how best to use it (and how best not to use it).
I think also by using the word intelligence it’s making people assume we’ve solved all the problems, and that just scaling is all we need. That’s a shame, because although it’s an amazing technology and a great step forward, it’s just one of many more steps we need to take to truly create artificial intelligence.