Apache Spark: transformaciones

How to create RDDs in Databricks?

We will explore how to generate RDDs (Resilient Distributed Datasets) using the Databricks platform to perform distributed data analysis efficiently. RDDs are the fundamental unit within Apache Spark, providing fault tolerance and efficiency in the execution of parallel operations.

How to import and configure a notebook in Databricks?

To start, make sure your cluster is active in Databricks by selecting the Platzi cluster or one you already have configured. Then, head to the Workspace section in Databricks, where you can import a notebook from your file system or a URL, such as from a GitHub repository.

1. Importing a file:
   • Head to File and use drag-and-drop to load compatible files such as .dbc, .scala, .py, .sql, R, .ipynb, etc.
2. Import from a URL:
   • If you have a notebook hosted in an online repository, you can import the notebook directly via its URL.

The example notebook, "What are Apache Spark RDDs Part 2?", will be available in the course resources. Once imported, attack your cluster to start running the code.

What are Spark Session and Spark Context?

Before proceeding to create RDDs, let's understand two essential concepts in Spark:

• Spark Session: is the main access point for interacting with Spark, and facilitates command execution.
• Spark Context: It allows direct communication with the Spark cluster, managing executions.

In Databricks, these components are already configured, eliminating the need to manually initialize Spark Session or Spark Context, optimizing the environment for the user.

How to create an RDD in Spark?

Creating an RDD can be done in different ways depending on your needs. In the following, we will explore how to create an empty RDD and a partitioned RDD:

# Create an empty RDDrdd_empty = sc.emptyRDD()
 # Create an RDD with three partitionsrdd_with_partitions = sc.parallelize([1, 2, 3, 4, 5], numSlices=3)
 # Collect and view partitionsprint(rdd_with_partitions.collect()) # Output: [1, 2, 3, 4, 5]print(rdd_with_partitions.getNumPartitions()) # Output: 3

In the example, parallelize is the function that distributes the data across partitions, allowing their parallel execution.

What are transformations and how to apply them to RDDs?

Transformations in Spark are operations that are applied to RDDs to generate new RDDs. These are lazy, meaning that they are not executed immediately until an action is performed that triggers them.

1. Map and Filter:

   • Map: applies a function to each RDD element.

     rdd_mapped = rdd_with_partitions.map(lambda x: x * 2)print(rdd_mapped.collect()) # Output: [2, 4, 6, 8, 10].

   • Filter: filter the elements that meet a specific condition.

     rdd_filtering = rdd_with_partitions.filter(lambda x: x % 2 == 0)print(rdd_filtering.collect()) # Output: [2, 4]

2. Distinct and ReduceByKey:

   • Distinct: Returns the unique elements of the RDD.

     rdd_unique = rdd_with_partitions.distinct()print(rdd_unique.collect()) # Output: [1, 2, 3, 4, 5].

   • ReduceByKey: Applies a reduce function to key-value pairs.

     rdd_pairs = sc.parallelize([('A', 3), ('B', 2), ('A', 2), ('B', 4)])rdd_reduced = rdd_pairs.reduceByKey(lambda a, b: a + b)print(rdd_reduced.collect()) # Output: [('A', 5), ('B', 6)]

3. SortByKey:

   • Sorts the elements of an RDD based on their keys.

   rdd_ordered = rdd_pairs.sortByKey(ascending=False)print(rdd_ordered.collect()) # Output: [('B', 6), ('A', 5)]

What should we keep in mind when working with transformations?

Transformations such as map, filter, distinct, among others, are fundamental to modify and structure the data in an RDD. It is important to remember that all these operations generate new RDDs, since the original RDDs are immutable.

In summary, learning how to handle Databricks and Apache Spark efficiently allows us to take full advantage of their capabilities for large and complex data analysis in a distributed and fault-tolerant manner. Keep experimenting and expanding your skills with practice!