Tokenización

Understanding how language models (LLMs) define and handle words is essential to correctly choose and apply such tools in real-world circumstances. Embeddings, vector representations with specific dimensions, allow you to find semantic relationships, but first you need to determine what exactly constitutes a "word".

What is tokenization in LLM and how does it work?

Tokenization in language models such as GPT, Cloth or Lama involves breaking text into units called tokens. Unlike simple space divisions performed automatically by humans, these models employ algorithmic methods and neural networks to identify relevant textual patterns.

Imagine a cook who slices all his ingredients in exactly the same way, without considering the particularities of each food:

• This rigidity could hinder the preparation of certain dishes.
• Similarly, a tokenizer using too rigid a method would lose efficiency in correctly interpreting the context of each word.

That's why LLM tokenizers examine millions of texts to intuitively decide how best to split words according to their context and semantic relevance.

Why do tokens matter on a practical level?

Whenever you interact with a tool like OpenAI, you will notice terms like "128k context window" or pricing associated with the number of tokens:

• The cost is generally billed per tokens used.
• A "context window" limits the volume of parseable or generable text; understanding this limitation helps to better manage prompts.

Because of this, knowing how tokens are determined allows:

• Optimize prompts sent to models to reduce costs.
• Properly fragment problems into several parts if they exceed the model's context capabilities.

How do language peculiarities affect the effectiveness of the prompt?

The effectiveness of the prompt also depends on a thorough knowledge of the language being worked on. Key factors such as:

• Correct use of question or exclamation marks in Spanish.
• Accurate use of contractions (e.g., "de él" into "del").
• Considerations such as the proper use of capital letters.

These details significantly impact the effectiveness of the response generated by the model. A carefully designed prompt, considering all these idiomatic peculiarities, improves the quality of the results.

Why are LLMs especially good at writing code?

Another known strength of LLMs lies in code generation. This is because:

• There are huge volumes of code available for machine learning.
• The programmatic syntax has very well-defined and predictable rules, making it easy to process algorithmically.

In addition, solving mathematical problems is more challenging for LLMs because:

• Mathematics is mostly not based on natural language semantics.
• Often, it is preferred to translate these problems to automatic code generation to address them effectively.

This knowledge paves the way for learning advanced prompting techniques and using tools such as OpenAI's Playground.