Super importante entender el tema de layers: ademas que util: cuando se usan varios lambdas se pueden repetir las librerias he ahi donde se pueden agregar al lambda como capas/layers organizandolas.
Bienvenida e introducci贸n al curso
Iniciando con Big Data
Cloud Computing en proyectos de BigData
Introducci贸n al manejo de datos en Cloud
Datos en Cloud
驴Qu茅 nube deber铆a utilizar en mi proyecto de Big Data?
Arquitecturas
Arquitecturas Lambda
Arquitectura Kappa
Arquitectura Batch
Extracci贸n de informaci贸n
Llevar tu informaci贸n al cloud
Demo - Creando nuestro IDE en la nube con Python - Boto3
驴C贸mo usar Boto3?
API Gateway
Storage Gateway
Kinesis Data Streams
Configuraci贸n de Kinesis Data Streams
Demo - Despegando Kinesis con Cloudformation
Kinesis Firehose
Demo - Configuraci贸n de Kinesis Firehose
Reto - Configurando Kinesis Firehose
AWS - MSK
Demo - Despliegue de un cl煤ster con MSK
Transformaci贸n de Informaci贸n
AWS - Glue
Demo - Instalando Apache Zeppelin
Creaci贸n del Developer Endpoint
Demo - Conectando nuestro developer Endpoint a nuestro Zeppelin Edpoint
Demo - Creando nuestro primer ETL - Crawling
Demo - Creando nuestro primer ETL - Ejecuci贸n
Demo - Creando nuestro primer ETL - Carga
AWS - EMR
Demo - Desplegando nuestro primer cl煤ster con EMR
Demo - Conect谩ndonos a Apache Zeppelin en EMR
Demo- Despliegue autom谩tico de EMR con cloudformation
AWS - Lambda
Ejemplos AWS- Lambda
Demo - Creando una lambda para BigData
Carga de Informaci贸n
AWS - Athena
Demo - Consultando data con Athena
AWS - RedShift
Demo - Creando nuestro primer cl煤ster de RedShift
AWS - Lake Formation
Consumo de informaci贸n
AWS - ElasticSearch
Demo - Creando nuestro primer cl煤ster de ElasticSearch
AWS - Kibana
AWS - QuickSight
Demo - Visualizando nuestra data con QuickSight
Seguridad, Orquestaci贸n y Automatizaci贸n
Seguridad en los Datos
AWS Macie
Demo - Configurando AWS Macie
Apache Airflow
Demo - Creando nuestro primer cl煤ster en Cloud Composer
Arquitectura de referencia
Clase p煤blica
驴Qu茅 es Big Data?
You don't have access to this class
Keep learning! Join and start boosting your career
Lambda functions in AWS are a key tool in Big Data management and processing thanks to their ability to execute code in response to events and their integration with other AWS services. Here's how to create a Lambda function from scratch, step by step, and the critical aspects to consider.
Triggers are events that initiate the execution of a Lambda function. For Big Data projects, it is common to use SNS or SQS. If you opt for SQS, you will connect to standard queues. These services allow you to orchestrate complex workflows by notifying your Lambda function when certain events occur.
"Layers is a feature that simplifies the management of shared libraries between multiple Lambda functions. By using layers, you can centralize and replicate libraries efficiently, reducing administration time.
In Big Data, environment variables play a crucial role in the secure handling of connections, such as those to databases. It is always recommended to encrypt this information, using services such as KMS to ensure confidentiality.
The role of the Lambda function defines which services it can interact with. For example, a role can grant access to CloudWatch Logs and CloudFormation. It is essential to apply the principle of least privilege, granting only the necessary permissions.
Proper optimization and configuration of Lambda functions are essential to maximize efficiency in Big Data projects. Let's explore several important configurations.
It is possible to deploy Lambda functions within a VPC, defining the subnet and the security group. This allows you to precisely control the network environment in which your function runs.
Dead letter queues are crucial tools that ensure that critical events are not lost in Big Data management. In the case of regular failures or errors in the execution of functions, problematic messages can be redirected to a secondary queue for further review and processing.
Enabling X-Ray allows detailed tracking of the execution of Lambda functions. This is vital for identifying bottlenecks and lag times, providing an in-depth analysis of the performance of your cloud applications.
Finally, some more advanced aspects enable greater efficiency and tracking in Lambda function execution.
Lambda allows a default concurrency of 1000 simultaneous instances. It is possible to increase this number up to 20,000 by making a request to AWS. Reserve concurrency to ensure that your critical functions always have resources available when they need them.
It is essential to log all executions using CloudWatch Logs to ensure you capture metrics and events that can be critical to troubleshooting and understanding the behavior of your application.
In Big Data projects, different services play complementary roles:
I encourage you to continue exploring these capabilities in your projects, taking full advantage of the tools and configurations available in AWS to transform Big Data efficiently and securely.
Contributions 5
Questions 0
Super importante entender el tema de layers: ademas que util: cuando se usan varios lambdas se pueden repetir las librerias he ahi donde se pueden agregar al lambda como capas/layers organizandolas.
Buenas no entendi la funcionalidad de concurrencias en Lambda. Alguien me pudiera explicar brevemente? Muchas gracias.
excelente!!
Want to see more contributions, questions and answers from the community?