Estructura y definición de tokens en Python

How to define the first tokens in a programming language?

Defining tokens is a crucial step in developing any programming language. In the case of the Platzi programming language (LPP), we will start by defining a subset that includes numbers, variables, identifiers, keywords and various special characters. These elements form the basis on which we will create our first tokens. We will learn how to generate a token.py file, where we will define two classes: token type and token itself. We will use enums to differentiate a list of finite values, and we will take advantage of typing 's namedtuple class to improve the definition of tokens.

What is the process to start defining tokens?

1. Create a token.py file: This file will be the core of our definitions. We will start by importing the necessary modules.

   from enum import Enum, auto, uniquefrom typing import NamedTuple

2. Define the TokenType class: We create a class containing the different types of tokens we will use.

   @uniqueclass TokenType(Enum): ASSIGN = auto() COMMA = auto() EOF = auto() # End of file FUNCTION = auto() IDENT = auto() # Identifier ILLEGAL = auto() INT = auto() LEFT_BRACE = auto() LET = auto() LEFT_PAREN = auto() PLUS = auto() RIGHT_BRACE = auto() RIGHT_PAREN = auto() SEMICOLON = auto()

3. Create the Token class: We use NamedTuple to define the structure of our tokens.

   class Token(NamedTuple): token_type: TokenType literal: str

4. Add additional methods: We integrate the special method __str__ to control how the token is printed.

   def __str__(self)-> str: return f "Type: {self.token_type}, Literal: {self.literal}"

How to prepare the development environment?

Before we start programming, we need to prepare our environment with Python 3.8. To do this, we will follow these steps:

• Create a virtual environment: we make sure that all development work is isolated.

  python3.8 -m venv bnbsource bnb/bin/activate

• Install the necessary dependencies: We will do it through the requirements.txt file that contains nose for tests and mypy for typing.

  pip3 install -r requirements.txt

How to manage version control with git?

It is essential to organize our project using git. We set up dedicated branches to divide the work and ensure the stability of our code base.

1. Check the current branch: Use the main branch as a base.

   git branch

2. Change or create a new branch:

   git checkout -b define-tokens.

Where to find additional resources and code summaries?

For ease of learning, full source code and other resources will be available on GitHub. Look for related information in the course comments or check the repository directly.

Remember, defining your own tokens and understanding their purpose is vital in the journey of building a programming language. Simplicity and clarity are allies throughout this process, so don't hesitate to review and practice. As we move forward, the concept of tokens and their implementation will become clearer. Keep going!