Difference between RDD and Dataframes in Pyspark

by | Jun 6, 2024 | Pyspark | 0 comments

In Apache Spark, the core data structures for handling and processing data are RDDs (Resilient Distributed Datasets) and DataFrames/Datasets, which are part of the higher-level Spark SQL API.

PySpark DataFrames are lazily evaluated. They are implemented on top of RDDs. When Spark transforms data, it does not immediately compute the transformation but plans how to compute later. When actions such as collect() are explicitly called, the computation starts. 

Resilient Distributed Datasets (RDDs)

RDDs (Resilient Distributed Datasets) are the low-level API in Apache Spark. They are immutable distributed collections of objects that can be processed in parallel.


  1. Immutability: Once created, RDDs cannot be modified. Operations on RDDs always produce new RDDs.
  2. Distributed: Data is distributed across multiple nodes in a cluster.
  3. Lazy Evaluation: Transformations on RDDs are lazy, meaning they are not executed until an action is performed.
  4. Fault Tolerance: RDDs are fault-tolerant and can recompute lost partitions using lineage information.
  5. Typed: RDDs are strongly typed and can handle any type of object.
  6. Fine-Grained Operations: Allows fine-grained transformations like map, filter, flatMap, etc.

Example Usage:

from pyspark import SparkContext

sc = SparkContext("local", "RDD example")

# Create an RDD from a list
data = [1, 2, 3, 4, 5]
rdd = sc.parallelize(data)

# Apply transformations and actions
rdd2 = rdd.map(lambda x: x * 2)
result = rdd2.collect()


DataFrames are the higher-level API in Spark SQL. They are similar to RDDs but provide a richer set of operations. A DataFrame is a distributed collection of data organized into named columns, similar to a table in a relational database or a data frame in R or Python’s pandas.


  1. Schema: DataFrames have schemas, meaning the data is organized into named columns.
  2. Optimized Execution: They use Catalyst optimizer for query optimization and Tungsten for efficient execution.
  3. API: Provide a rich API for SQL queries, aggregations, and transformations.
  4. Integration: Can easily integrate with various data sources like JSON, CSV, Parquet, and databases.
  5. Ease of Use: Higher-level abstraction makes it easier to write more concise and readable code.
  6. Interoperability: Can be converted to/from RDDs and used with Spark SQL.

Example Usage:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("DataFrame example").getOrCreate()

# Create a DataFrame from a list
data = [(1, "Alice"), (2, "Bob"), (3, "Cathy")]
columns = ["id", "name"]
df = spark.createDataFrame(data, columns)

# Apply transformations and actions
df2 = df.withColumn("id_double", df["id"] * 2)

Comparison of RDDs and DataFrames

API LevelLow-level APIHigh-level API
SchemaNo schema, untypedSchema, typed
OptimizationsLimited optimizationCatalyst optimizer, Tungsten execution engine
Ease of UseRequires more code and understanding of Spark internalsConcise, SQL-like queries
TransformationsFine-grained transformations (map, filter, etc.)SQL-like transformations (select, groupBy, etc.)
InteroperabilityLess interoperability with external data sourcesHigh interoperability with various data sources
PerformanceGenerally slower due to lack of optimizationsFaster due to advanced optimizations

When to Use RDDs vs DataFrames

  • Use RDDs when:
    • You need low-level transformation and actions.
    • You are working with unstructured data.
    • You need strong typing and complex transformations.
  • Use DataFrames when:
    • You need higher-level abstractions and ease of use.
    • You want to leverage advanced optimizations for performance.
    • You are dealing with structured or semi-structured data.
    • You need integration with various data sources and SQL capabilities.


While RDDs provide more control over the data processing and are the building blocks of Spark, DataFrames offer higher-level abstractions, better optimizations, and easier integrations with data sources and SQL operations. For most use cases, DataFrames are recommended due to their simplicity and performance benefits. However, understanding both APIs can help you choose the right tool for specific tasks.

Written by HintsToday Team

Related Posts

Are Dataframes in PySpark Lazy evaluated?

Yes, DataFrames in PySpark are lazily evaluated, similar to RDDs. Lazy evaluation is a key feature of Spark's processing model, which helps optimize the execution of transformations and actions on large datasets. What is Lazy Evaluation? Lazy evaluation means that...

read more

BDL Ecosystem-HDFS and Hive Tables

Big Data Lake: Data Storage HDFS is a scalable storage solution designed to handle massive datasets across clusters of machines. Hive tables provide a structured approach for querying and analyzing data stored in HDFS. Understanding how these components work together...

read more

Get the latest news

Subscribe to our Newsletter


Submit a Comment

Your email address will not be published. Required fields are marked *