• Spark is not a Hadoop replacement.
• Spark is a map/reduce replacement.
• Spark uses Stand Alone Machine Cluster manager.

What’s the difference between Spark and map/reduce?

• in map/reduce, let’s say we have 100 iterations for each iteration, we will read and write the intermediate data of that specific iteration from/to the disk. which massively slows the overall process

• in Spark, No read or write operations will occur until the completion of all the iterations.

• How is that done? By making all the processes within the 100 iterations be done through the memory, not the disk. which makes it much faster than having to deal with the disk after each iteration

• Spark works great with steaming data and generative data it’s called Unified Analytics

How does spark work?

1. Spark works with any source file system, Unlike Hadoop which only works with HDFS.
   • Spark reads the data from whichever source file system and converts it into a Resilient Distributed Dataset (RDD)
2. works with any resource manager(YARN, Mesos, Kubernetes)
3. Spark Engine/core

PySpark Libraries:

• SQL+DF
• MLIP
• Streaming Lib
• Graph Lib