by lochan2014 | Oct 11, 2024 | Pyspark
Q1.–We are working with large datasets in PySpark, such as joining a 30GB table with a 1TB table or Various Transformation on 30 GB Data, we have 100 cores limit to use per user , what can be best configuration and Optimization strategy to use in pyspark ? will... by lochan2014 | Oct 11, 2024 | Pyspark
To determine the optimal number of CPU cores, executors, and executor memory for a PySpark job, several factors need to be considered, including the size and complexity of the job, the resources available in the cluster, and the nature of the data being processed....