Tag: embedding

Semantics Illustrated

This is a longer version of my recent Language Corner post. Language Corner is a feature I write regularly on LinkedIn. It is focused on linguistics, so I cut most of the quantitative material and moved it here. I motivated that post with a reference to Semantle, a popular online word game, and showed how the game uses cosine similarity.

Cosine Similarity

The target word that day was “inmate.” The nearest word to “inmate,” with a similarity score of 81, is “inmates,” plural. The nearest word after that is “prisoner,” a synonym, with a score of 72. These scores can be a little misleading because they’re not a straight scale. They are “cosine similarity” in the Word2Vec pretrained embedding model.

Most of my coursework in linguistics focused on syntax – why words and word-parts are where they are in a sentence. The state of the art in semantics was observing that words reside in “semantic fields.” Today, thanks to AI language models, we can study semantic distance with precision.

For instance, “gold” denotes both a metal and a currency. It occupies the intersection of at least two different semantic fields. One, it shares with iron and tungsten. The other, with bitcoin and chalcedony beads. Instead of sets intersecting, as in a Venn diagram, I always pictured them as axes crossing.

Gold sits at the corner of metals and currency. Gold is also a color, so we’re going to need at least three dimensions. Word2Vec uses 300 – which is small, as embeddings go.

Word Embeddings

The great success of Large Language Models (LLMs) owes to their training data being self-labeled. Training a neural network generally requires tons of pre-labeled data. You feed in images from ImageNet, your neural net tries to guess the subject, and the labels provide feedback. By the way, I wrote an explainer on deep neural networks back in 2021, before the LLM mania took over.

The task assigned to an LLM is simply to predict the next word in a text. So, each word is a test case. The context leading up to the word is the input, and the word itself is the label. Word embeddings arise as a byproduct of this activity.

Inputs to a neural network need to be encoded in some way. Quantitative variables are straightforward, but others are not. An embedding is a layer in the neural network that allows it to learn the most suitable representation for a given input. Here is an example from a project where I needed to accept car models as input:

carmodel_inputs = keras.layers.Input(shape=[], dtype="string")
carmodel_indices = keras.layers.Lambda(
    lambda carmodel: table.lookup(carmodel)
)(carmodel_inputs)
models_embed = keras.layers.Embedding(input_dim=table_len, output_dim=10)(
    carmodel_indices
)
all_inputs = keras.layers.concatenate([num_scaled, cats_encoded, models_embed])

 

This is an efficient way to handle the encoding problem, and it has the advantage that you can save the embedding for later use. The developers of Word2Vec ran their text-completion training including an embedding with output_dim = 300, and then saved the embedding.

It’s important to note that word embeddings are built on syntactic substitutability. The word nearest to “north” is “south.”

The Word Globe

Semantle calculates similarity as the cosine (times 100) of the angle between the two vectors that represent the words. Imagine if all the words in the language were arranged on the globe, in three dimensions instead of 300, with word-vectors radiating from the center of the Earth.

The cosine of 44 degrees is 0.72, the similarity between “inmate” and “prisoner.” An arc of 44 degrees, on planet Earth, would be roughly 3,000 miles – the distance from Boston to Dublin. If that seems like a long flight to reach the next synonym, think of how many words are totally unrelated (cos = 0) and how much room they have to spread out in 300 dimensions.

A single vector in 300-D space has an entire 299-D subspace (its orthogonal complement) of words totally unrelated to it. Below is the distribution of similarity scores relative to “inmate,” calculated on the Word2Vec embedding. Most words are near zero. Those above 0.60 are too few to see on the chart.

Nonetheless, this word-globe is key to understanding how to beat Semantle. The first guess gives the absolute distance from “giant.” That’s a circle. The second guess gives the absolute distance from “enemy.” That’s another circle (actually, they’re 298-dimensional hyperspheres, but you get the idea).

You can now load the Word2Vec embedding into a short Python script and calculate where the two circles cross.

Principal Component Analysis

Now that Semantle is licked, we come back to the semantic fields containing “gold.” Gold is the textbook example because it participates in a few distinct fields. I chose a handful of words, all within striking distance, and projected them down to three dimensions – from fifteen hundred. For this exercise, I have switched to OpenAi’s text-embedding-3-small, which I have written about previously.

Readers of this blog are probably familiar with Principal Component Analysis (PCA). This algorithm finds the one dimension that captures the most variance across the sample, and then the second most, etc., finally projecting the points onto this subspace.

I feel good about this projection. The three principal components explain a respectable 35% of the total variance:

  • PC1: 16.2%
  • PC2: 11.0%
  • PC3: 7.4%

PC1 is the color-wealth axis, with ordinary colors on the right, “money” and “wealth” on the left, and “gold” in the middle. It’s not surprising that PC1 captures most of the variance. Apart from “gold,” colors have nothing to do with wealth.

PC2 captures the “metal” concept omitted by PC1. Industrial metals like “iron” and “palladium” are on the right. Lastly, PC3 is the wealth-metal axis, with “bullion” on the left and “iron” on the right. Colors sit this one out, with values near zero.

Literary Analysis with RAG

The situation was dire. Napoleon’s army was far from its supply lines, with the harsh Russian winter closing in. Their only hope of shelter lay in the capital city of Moscow, but, arriving there in September 1812, they found the city in flames. Of the half-million men who had set out with Napoleon, only ten thousand would survive the retreat.

If you ask ChatGPT about this, it will give the conventional answer, plus some convincing – and incorrect – ideas about the book. 

The conventional version of this history says that the Russians purposely torched their own city. Writing in War and Peace, however, Count Leo Tolstoy won’t admit such a desperate tactic. He presents a strong case that Moscow, “a town built of wood, where scarcely a day passes without conflagration” would naturally burn once its people – and the fire department – evacuated.

If you ask ChatGPT about all this, it will give the conventional answer. If you ask, “according to Tolstoy,” it will still give the conventional answer, plus some convincing – and incorrect – ideas about the book. That’s because ChatGPT has never read the book!

It’s adorable, because it bluffs about the reading, exactly like a slacking college student. What was the professor thinking, assigning a thousand-page novel?

Now, thanks to Retrieval Augmented Generation (RAG), you can help ChatGPT answer such questions by priming it with selected passages from the novel. So, I did. I wanted to demonstrate that RAG would support more-advanced text analyses: 

  1. Answer questions using passages from a single novel
  2. Answer questions using passages from two novels and compare them
  3. Compare passages from two novels based on an unseen question
  4. Compare passages from two novels based on a similarity search

This week, we’ll cover the basics using War and Peace, and then I’ll share the two-novel results next week.

War and Peace and RAG

The basic idea behind RAG is simply to query a text database for the priming material, before handing the problem over to a Large Language Model (LLM) like ChatGPT. To be precise, I am using the OpenAI API to work with the GPT 3.5 model.

The only AI involved in the retrieval step is that we use an “embedding model” to convert the text and the query string into vectors. Apart from that, it’s a text search. You could, conceivably, use old-school text search techniques to do the job. I haven’t tried that, yet. What I tried were these three embedding models:

  • text-embedding-3-large
  • text-embedding-ada-002
  • text-embedding-3-small

The embedding models convert chunks of text into vectors of varying length, hence the “large” and “small” size designations. If you’re an AI person reading this, you already know about mapping words to vectors. Mapping paragraphs to vectors follows the same principle. Both are due to Mikolov, et al. See Distributed Representations of Sentences and Documents.

If you’re a language person, well, you won’t be surprised to learn that the English lexicon can be arranged into a spatial array so that “cat” and “dog” end up together. And, if a three-D word space is good, a 300-D word space is better!

I downloaded some novels from the Gutenberg project, did some basic text parsing on them, and then converted each into its own vector database. I used the Chroma database natively and with the LangChain library. Other popular vector databases include Milvus and Pinecone.

I parsed the novels into chapters first, so that Chroma would pick up the chapter headings as metadata. If you use Project Gutenberg, be sure to stop parsing at THE END because there are about 600 lines of legal stuff after that. 

Based on the context provided, it is implied that the Russians did not intentionally set fire to Moscow. The fire in Moscow was attributed to a combination of factors such as the presence of foreign troops, abandoned wooden buildings, soldiers’ carelessness with pipes and campfires, and the general chaos and looting that ensued. The burning of Moscow was seen as a result of the circumstances rather than a deliberate act of arson by the Russians.

The main thing to tune, here, is the size of text chunks in the database. You’ll be passing text chunks to the LLM via an API, and there’s a size limit. GPT 3.5 has a window of roughly 3,000 words.

The typical RAG use case is simply to find one good chunk with the answer. For the Moscow fire question, that is chunk #1440, which happens to be only 0.12 “cosine distance” away from the query text:

{‘source’: ‘data\\tolstoy\\Chapter_255.txt’, ‘start_index’: 10994}

But what if you’re asking “who is the best Russian general?” or “how many siblings does Natasha have?” Clues will be scattered throughout the text, and you will need more, smaller chunks. 

Prince Andrew doesn’t marry Natasha immediately because he decides to conform to his father’s wish to propose and postpone the wedding for a year. This decision is made in the hopes that either his own or Natasha’s feelings may change within that time period. Despite Natasha’s distress and desire to marry sooner, Prince Andrew feels bound by his father’s wish and the decision to delay the wedding.

I would say that smaller is better because the retriever can always fetch multiple chunks from the same neighborhood – as long as they’re at least big enough to be picked up by the vector search. After some experimentation, I settled on a chunk size of 1,200 characters, with the ada-002 embedding, which has 1,536 dimensions.

All operations on the API are sized (and priced) in “tokens,” so it’s a good idea to employ the tiktoken counter, and keep an eye on your token limits. My 1,200-character chunks run around 220 tokens.

LangChain’s recursive text splitter does its best to honor sentences and paragraphs, but not semantics. It still feels like taking a favorite novel and chopping it up in a Cuisinart. Greg Kamradt has invented a semantic text splitter, which can detect and split based on topic changes, but its implementation in LangChain isn’t great.

Chroma’s query method takes a string, vectorizes it, and then does a similarity search against the embeddings in the database. Normally, you look to maximize “cosine similarity” but, with Chroma, you must minimize “cosine distance.” You can also call the OpenAI embedding function on your own, and then search by vector directly. Just be sure to use the same embedding model in all cases.

That covers the basics and the single-novel case. Next week, we’ll use RAG to compare two novels.

Sidebar: While writing this, I felt the need to review some points from War and Peace, so I pulled the book off the shelf … and then remembered I had just built a searchable database. Old habits die hard.