• RDD is immutable

RDD Lineage:

• اول حاجة بتتعمل خالص هي ان RDD0 بتتكريت عن طريق انها بتقرأ الداتا من ال source file system بتاعي اللي انا محتاج اعمل عليه ال operations بتاعتي
• عشان اوصل ل النتيجة\الداتا\RDD النهائية لازم أمر بشوية transformations
• RDDs are logical objects that point to data location but don’t store data themselves.
• transformations→ Lazy operations
• امتى ال transactions بتتنفذ فعلا وتطلعلي o/p؟ لما أجي انفذ Action
• Actions are Eager operations
• RDD3.collect() → An action that is once triggered the whole lineage transformations are performed and the data is retrieved
• .take(5) → applies the operation lineage only on 5 records which massively reduces the cost

• The lineage is stored in a DAG, each new transformation is added to the lineage will be automatically updated to the DAG.

  

RDD Distribution:

• ال RDD بتتقسم ل Partitions و بعدين ال partitions بتتوزع على ال nodes و كل node بتبدأ ت execute بقى

• مثال مثلا لو انا عندي RDD من 4 partitions وعندي 4 nodes كل نود هتعمل ال transformation على ال partitions الخاص بيها دا اسمه

  • Narrow Operation → 4 partitions as input, 4 partitions gets produced.

    

• Wide Operationn → 3 input partitions, 2 output partitions due to Shuffle operations