Tareas periódicas con Celery

How to work with periodic tasks in Celery?

Learning how to schedule periodic tasks is an essential skill for automating processes that require constant execution, such as data collection or server maintenance. In this content, you will discover how to handle periodic tasks using Celery, a powerful Python task manager that makes it easy to schedule and run jobs in the background.

What is Celery and how is it used?

Celery is a Python library designed to manage tasks in an asynchronous and scheduled manner. It allows you to define specific tasks and schedule their execution at regular intervals, which is useful when you need repetitive processes without human intervention. Some key aspects of Celery include:

• Asynchronous task management: Allows long processes to run in the background, improving application performance.
• Compatibility with different message brokers: It can be used with RabbitMQ, Redis, among others.
• Scheduling of periodic tasks: Provides a module called celery.beat, which allows you to configure tasks to run automatically on a set schedule.

How to configure periodic tasks in Celery?

First, you must import the necessary tools to work with Celery. Here's how to structure your script to create a recurring task:

from celery import Celeryfrom celery.schedules import crontab
app = Celery('app_name',  broker='redis://localhost:6379/0')
 # Set up periodic taskapp.conf.beat_schedule = { ' task_name': { ' task': 'module_name.specific_task', ' schedule': crontab(minute=0,  hour='*/1'), # Run every hour },}

How to extract data regularly with Python?

Suppose you need to collect data from a web page at regular intervals. You can use libraries like beautifulsoup4 to parse the HTML and extract the information you need. This is complemented with Celery to schedule periodic data download.

import requestsfrom bs4 import BeautifulSoup
def download_data(): url = " https://ejemplo.com/pagina-deseada" response = requests.get(url) soup = BeautifulSoup(response.text, 'html.parser')    
 # Extract page-specific data extract_data = soup.find_all('div',  class_='class-specific')    
 with open('data.json', 'w') as file: file.write(str(extract_data))

How to run a Celery periodic task?

To start the tasks, you must run Celery with a worker and, if you are scheduling periodic tasks, you also need to start the beat scheduler.

1. Start the worker: Process that executes the tasks scheduled by Celery.

   celery -A application_name worker --loglevel=info

2. Start the beat scheduler: It is in charge of sending the periodic tasks to the worker.

   celery -A application_name beat --loglevel=info

Tips for working with periodic tasks in Celery?

• Start simple: Set up a basic task to make sure Celery components are properly installed and working together.
• Logging and monitoring: Use logs to understand what is going wrong in case of errors, and monitor tasks to optimize their execution times.
• Scalability: Design tasks in a modular fashion for easy scalability and customization.
• Atomic tasks: Make your tasks small and perform a single function to reduce complexity and facilitate debugging.

Celery is a powerful tool that can handle multiple tasks and schedule them efficiently, offering an ideal solution for projects that require automation and background processing. With practice and a good approach, you can master the use of periodic tasks and improve the efficiency of your applications.