Manejo de Big Data
Bienvenida
Introducción a big data
Google Cloud Platform para big data
Explorando Google Cloud Platform para big data e IA
Cloud Dataflow
BigQuery
Integración de datos con BigQuery y Data Studio
Cloud Pub/Sub
Casos de uso en tiempo real con Cloud Pub/Sub y Dataflow
Herramientas de inteligencia artificial
Introducción a machine learning
Cómo funciona Vertex AI
Flujo de trabajo con Vertex AI
Visión artificial con Vertex AI
Machine Learning API
Uso de AutoML
Conclusiones
Proyecto: Clon de Google Photos (Parte 4 de 4)
Terminemos el proyecto
Utilización de Vision API
Creación de la base de datos y prueba del sistema
In the world of Big Data, the ability to capture and analyze data in real time is crucial for fast and efficient decision making. Using PubSub and Dataflow on Google Cloud, we can implement a solution that not only receives real-time data but also processes and stores it for further analysis. In this content, you will learn how to create a workflow from streaming cab data in New York to storing it in BigQuery, all by integrating different Google Cloud tools.
The Dataflow configuration starts in the Google Cloud console. Here, we must identify the service in the Big Data section. Upon login, we are presented with two options to create a job: use a template or create a custom SQL. For this case study, we use a SQL query that defines how we want to process and store the data:
SELECT TIMESTAMP_TRUNC(event_timestamp, MINUTE) AS start_period, COUNT(pickup) AS pickup_countFROM `project.dataset.service_pubsub`WHERE status = 'pick up'GROUP BY start_period.This code allows us to:
SELECT and GROUP BY to group the data in one-minute intervals and count the passengers picked up.Once the job is configured, Dataflow starts receiving and processing events in real time. Subsequently, all this data, transformed and grouped, is stored in BigQuery. There we can perform more detailed and persistent analysis. During job creation, we specify:
taxi_data has been chosen to store data such as start period and passenger count.Once the data is stored in BigQuery, we can perform more complex queries to obtain detailed analysis. For example, visualizing how many collection events occurred at a certain point in time. In addition, this data can be explored and graphically represented in Data Studio, providing a clear and understandable visualization of the time series of our data.
When using Data Studio, the following can be performed:
This comprehensive approach helps to understand how real-time data can be effectively integrated and visualized for better informed and timely decision making.
Contributions 16
Questions 14
Consulta:
SELECT
TUMBLE_START('INTEVARL 1 MINUTE') as period_start,
SUM(passenger_count) AS pickup_count
FROM pubsub.topic.`pubsub-public-data`.`taxirides-realtime`
WHERE
ride_status = "pickup"
GROUP BY
TUMBLE(event_timestamp, 'INTERVAL 1 MINUTE')
es increible apicla algo que no explica como lo obtuvo, yo no estoy pudiendo crear el job de ninguna forma
antes de ver la parte practica de esta clase mira esto:
https://www.youtube.com/watch?v=f5DOsB7Nlw0&list=PLIivdWyY5sqKwVLe4BLJ-vlh9r9zCdOse&index=3
Para ejemplos practivos utilicé datos que muestran en la busqueda, pero se debió mostrar desde donde se buscó el servicio de pubsub porque parece que actualmente ya no está almenos no bajo la misma ruta
falta la sentencia para poder seguir la clase mejor
SELECT
TUMBLE_START('INTERVAL 1 MINUTE') as period_start,
SUM(passenger_count) AS pickup_count
FROM pubsub.topic.`pubsub-public-data`.`taxirides-realtime`
WHERE
ride_status = "pickup"
GROUP BY
TUMBLE(event_timestamp,'INTERVAL 1 MINUTE')Falta incluir el query
Hola, muchas gracias por la explicación.
Quisiera comentarte que en la versión actual no veo la opción de “Crear desde SQL”
Podrías indicar cómo encontrar está opción:
Want to see more contributions, questions and answers from the community?