Preprocesamiento Inicial

Text processing is a fundamental skill in data analysis, especially when working with user reviews. Proper text cleaning not only improves the quality of our analysis, but also optimizes computational resources by focusing only on relevant information. Regular expressions (Regex) are powerful tools that allow us to manipulate text efficiently, identifying specific patterns that we want to preserve or eliminate.

How to prepare a review dataset for analysis?

Before performing any content analysis on reviews, it is crucial to perform a proper preprocessing. This initial step helps us to highlight relevant content and reduce computational costs by eliminating unnecessary information.

To begin, we must select the columns that we really need. In this case, we will focus on:

• The "review body" column (review content).
• The "start" column (score)

It is recommended to work with a copy of the original dataframe to keep the original data intact:

filter_data = df.copy()

Then, we can check the first rows to understand the structure of our data:

filter_data[[['review_body', 'start']].head(3)

Checking for null values

A fundamental step is to check for null values in our dataset:

filter_data.isnull().sum().

If we do not find null values, we can proceed directly to text cleaning. Otherwise, we should implement strategies to handle these missing values.

How to use Regex to clean up review texts?

Regular expressions (Regex) are sequences of characters that define search patterns. Thanks to them, we can describe, identify and manipulate text strings efficiently. They are especially useful for finding patterns such as ats, URLs or HTML tags.

To implement cleaning, we need to import the appropriate libraries:

import reimport string

Then, we create a function that applies all the necessary transformations:

def clean(text): # Convert to lowercase text = text.lower()    
 # Remove text in square brackets text = re.sub(r'\[.*?\]', '', text)    
 # Remove URLs text = re.sub(r'https?://www+|www.\S+', '', text)    
 # Remove HTML tags text = re.sub(r'<.*?>', '', text)    
 # Remove punctuation marks text = re.sub(r'[{}]'.format(string.punctuation), '', text)    
 # Remove line breaks text = re.sub(r'\n', '', text)    
 # Remove words containing numbers text = re.sub(r'\w*, '', text)    
 # Remove emojis or special characters text = re.sub(r'[^\x00-\x7F]+', '', text)    
 # Remove leading and trailing whitespace text = text.strip()    
 return text

Applying the cleanup function

Once our function is defined, we apply it to the column containing the reviews:

filter_data['clean_review'] = filter_data['review_body'].apply(clean).

This process may take a few seconds depending on the size of the dataset. In the example, it took approximately 8 seconds.

What specific transformations do we perform on the text?

Our clean function performs several important transformations:

1. Lowercase conversion: Standardizes all text for easier analysis.
2. Removal of bracketed text: Removes content that is usually metadata or clarifications.
3. Removal of URLs: Web addresses rarely add to the sentiment or theme of the review.
4. HTML tag removal: Cleans up markup text that may be present in reviews copied from websites.
5. Punctuation mark removal: Simplifies text for further analysis.
6. Line break removal: Normalizes text formatting.
7. Removal of words with numbers: Removes references to users or codes that do not contribute to the semantic content.
8. Removal of emojis and special characters: We focus on pure textual content.
9. Removing white spaces: Clean up the formatting for a more consistent text.

When comparing the original review with the cleaned one, we can observe significant differences such as the conversion to lowercase and the elimination of punctuation marks, which highlights the most relevant part of each comment.

Text preprocessing is a fundamental step that should not be underestimated in any textual data analysis project. With these techniques, you will be prepared to extract valuable information from your review datasets and get more accurate and relevant insights.

Have you used regular expressions in your data analysis projects? Share your experiences and doubts in the comments section.