HintsToday
Hints and Answers for Everything
recent posts
- Apache Spark RDDs: Comprehensive Tutorial
- Complete crisp PySpark Interview Q&A Cheat Sheet
- Python Lists- how it is created, stored in memory, and how inbuilt methods work — including internal implementation details
- Data Engineer Interview Questions Set1
- PySpark SQL API Programming- How To, Approaches, Optimization
about
Category: Pyspark
Apache Spark RDDs: Comprehensive Tutorial Table of Contents Introduction to RDDs Resilient Distributed Datasets (RDDs) are the fundamental data structure of Spark. They are: Key Characteristics: RDD Lineage RDD lineage is a graph of all the parent RDDs of an RDD. It’s built as a result of applying transformations to the RDD. Output would show…
A quick reference for date manipulation in PySpark:– Function Description Works On Example (Spark SQL) Example (DataFrame API) to_date Converts string to date. String TO_DATE(‘2024-01-15’, ‘yyyy-MM-dd’) to_date(col(“date_str”), “yyyy-MM-dd”) to_timestamp Converts string to timestamp. String TO_TIMESTAMP(‘2024-01-15 12:34:56’, ‘yyyy-MM-dd HH:mm:ss’) to_timestamp(col(“timestamp_str”), “yyyy-MM-dd HH:mm:ss”) date_format Formats date or timestamp as a string. Date, Timestamp DATE_FORMAT(CURRENT_DATE, ‘dd-MM-yyyy’) date_format(col(“date_col”), “dd-MM-yyyy”)…
Window functions in PySpark allow you to perform operations on a subset of your data using a “window” that defines a range of rows. These functions are similar to SQL window functions and are useful for tasks like ranking, cumulative sums, and moving averages. Let’s go through various PySpark DataFrame window functions, compare them with…
For Better understanding on Spark SQL windows Function and Best Usecases do refer our post Window functions in Oracle Pl/Sql and Hive explained and compared with examples. Window functions in Spark SQL are powerful tools that allow you to perform calculations across a set of table rows that are somehow related to the current row.…
Here’s an enhanced Spark SQL cheatsheet with additional details, covering join types, union types, and set operations like EXCEPT and INTERSECT, along with options for table management (DDL operations like UPDATE, INSERT, DELETE, etc.). This comprehensive sheet is designed to help with quick Spark SQL reference. Category Concept Syntax / Example Description Basic Statements SELECT SELECT col1, col2 FROM table WHERE…