In Apache Spark, the core data structures for handling and processing data are RDDs (Resilient Distributed Datasets) and DataFrames/Datasets, which are part of the higher-level Spark SQL API.

PySpark DataFrames are lazily evaluated. They are implemented on top of RDDs. When Spark transforms data, it does not immediately compute the transformation but plans how to compute later. When actions such as collect() are explicitly called, the computation starts. 

Resilient Distributed Datasets (RDDs)

RDDs (Resilient Distributed Datasets) are the low-level API in Apache Spark. They are immutable distributed collections of objects that can be processed in parallel.

Characteristics:

1. Immutability: Once created, RDDs cannot be modified. Operations on RDDs always produce new RDDs.
2. Distributed: Data is distributed across multiple nodes in a cluster.
3. Lazy Evaluation: Transformations on RDDs are lazy, meaning they are not executed until an action is performed.
4. Fault Tolerance: RDDs are fault-tolerant and can recompute lost partitions using lineage information.
5. Typed: RDDs are strongly typed and can handle any type of object.
6. Fine-Grained Operations: Allows fine-grained transformations like map, filter, flatMap, etc.

Example Usage:

from pyspark import SparkContext

sc = SparkContext("local", "RDD example")

# Create an RDD from a list
data = [1, 2, 3, 4, 5]
rdd = sc.parallelize(data)

# Apply transformations and actions
rdd2 = rdd.map(lambda x: x * 2)
result = rdd2.collect()
print(result)

DataFrames

DataFrames are the higher-level API in Spark SQL. They are similar to RDDs but provide a richer set of operations. A DataFrame is a distributed collection of data organized into named columns, similar to a table in a relational database or a data frame in R or Python’s pandas.

Characteristics:

1. Schema: DataFrames have schemas, meaning the data is organized into named columns.
2. Optimized Execution: They use Catalyst optimizer for query optimization and Tungsten for efficient execution.
3. API: Provide a rich API for SQL queries, aggregations, and transformations.
4. Integration: Can easily integrate with various data sources like JSON, CSV, Parquet, and databases.
5. Ease of Use: Higher-level abstraction makes it easier to write more concise and readable code.
6. Interoperability: Can be converted to/from RDDs and used with Spark SQL.

Example Usage:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("DataFrame example").getOrCreate()

# Create a DataFrame from a list
data = [(1, "Alice"), (2, "Bob"), (3, "Cathy")]
columns = ["id", "name"]
df = spark.createDataFrame(data, columns)

# Apply transformations and actions
df2 = df.withColumn("id_double", df["id"] * 2)
df2.show()

Comparison of RDDs and DataFrames

Feature	RDDs	DataFrames
API Level	Low-level API	High-level API
Schema	No schema, untyped	Schema, typed
Optimizations	Limited optimization	Catalyst optimizer, Tungsten execution engine
Ease of Use	Requires more code and understanding of Spark internals	Concise, SQL-like queries
Transformations	Fine-grained transformations (map, filter, etc.)	SQL-like transformations (select, groupBy, etc.)
Interoperability	Less interoperability with external data sources	High interoperability with various data sources
Performance	Generally slower due to lack of optimizations	Faster due to advanced optimizations

When to Use RDDs vs DataFrames

• Use RDDs when:
  • You need low-level transformation and actions.
  • You are working with unstructured data.
  • You need strong typing and complex transformations.
• Use DataFrames when:
  • You need higher-level abstractions and ease of use.
  • You want to leverage advanced optimizations for performance.
  • You are dealing with structured or semi-structured data.
  • You need integration with various data sources and SQL capabilities.

Conclusion

While RDDs provide more control over the data processing and are the building blocks of Spark, DataFrames offer higher-level abstractions, better optimizations, and easier integrations with data sources and SQL operations. For most use cases, DataFrames are recommended due to their simplicity and performance benefits. However, understanding both APIs can help you choose the right tool for specific tasks.