El Mecanismo de Atención y Razonamiento en Modelos de IA

Did you know that your brain and models like ChatGPT use similar principles to anticipate responses? This similarity comes from something called priming and the attention mechanism, central concepts in understanding how language models like ChatGPT work.

What is priming and how does it influence your decisions?

Priming is a psychological phenomenon that causes the brain to respond automatically after receiving certain stimuli, even if you are not aware of it.

For example, when you hear simple mathematical questions about adding small numbers such as 1+1 or 2+2, the immediate answers you give later in other types of questions may be subtly influenced by those first interactions.

• Our brains prioritize certain information automatically.
• These influences can produce predictable answers in different people, leading to overlaps between them.

In similar classes or experiments, many people tend to think of the same vegetable after answering simple questions about sums, a clear effect of priming.

How do LLMs like ChatGPT use attention?

Language models (LLMs) like ChatGPT use a similar mechanism called attention. This mechanism allows assigning specific weights to words according to their importance within the given context.

For example, in the sentence "the black cat is sleeping":

• The word "cat" has a significant weight.
• Words like "sleeping" or "playing" become probabilistic by context.
• If we remove the word "cat", other options open up more widely, showing how each word conditions the result.

This differentiates ChatGPT from simpler methods like your cell phone's predictive keyboard, which usually only consider the last word typed, ignoring the broader context.

What is the context window and how does it affect ChatGPT?

The context window refers to the total volume of information that a model can hold during your interaction with it. In ChatGPT 4.0, this window goes up to 128,000 tokens.

To give you an idea:

• In English, this is roughly equivalent to 40,000 words (about 160 pages).
• In Spanish, it is usually fewer words because of the average word length.

This concept is vital, as ChatGPT uses the entire previous conversation, not just the last interaction, to generate accurate responses. This explains why it sometimes seems to "forget" previous information; in reality, it simply changes the priority of the context used.

What impact does priming have on interaction with ChatGPT?

When you perform an exercise similar to the numerical addition exercise followed by a request to name a vegetable, priming also affects the responses generated by the model.

• The model responds influenced by previous stimuli.
• It is common for the model to name a vegetable as a predicted response:
• For example, "broccoli" is a very common response after similar exercises.

This clearly reflects how attention and context act on artificial intelligence and make its responses resemble human reactions.

What was your experience, did you agree with other people? Share it in the comments, it will be interesting to see how many thought of the same vegetable.