Author: lochan2014
Optimization in PySpark is crucial for improving the performance and efficiency of data processing jobs, especially when dealing with large-scale datasets. Spark provides several techniques and best practices to optimize the execution of PySpark applications. Before going into Optimization stuff why don’t we go through from start-when you starts executing a pyspark script via spark…
Apache Hive Overview Hive is a data warehouse infrastructure built on top of Hadoop and SQL-like language called HiveQL for querying data stored in various databases and file systems that integrate with Hadoop. Hive allows users to read, write, and manage large datasets residing in distributed storage using SQL. It simplifies the process of data…
What is Hadoop? Hadoop is an open-source, distributed computing framework that allows for the processing and storage of large datasets across a cluster of computers. It was created by Doug Cutting and Mike Cafarella and is now maintained by the Apache Software Foundation. History of Hadoop Hadoop was inspired by Google’s MapReduce and Google File…