hello everyone welcome to Big Data thoughts today we would be talking about spark architecture in depth so I had done a previous video which gave a quick introduction to spark architecture but today we are going to talk in depth about many terms or concepts that anybody needs to know before starting to work in spark so let's get started first of all let's talk about what was the history of why spark was introduced so Apache spark began began in 2009 as a research project at us Berkeley and at that point of time Hadoop and mapreduce were
very very prevalent and dominant in the Big Data World it and it was a very good solution to processing large-scale data in a distributed fashion but like everything has its pros and cons my produce also had its cons where it was not meant for iterative processing and there were other drawbacks that were there so there was a need to look at a different kind of a solution the amp lab worked with multiple people who were using mapreduce to understand what are the drawbacks what exactly is wanted and then a number of use cases were discussed
and then the um the whole thing about evolution of spark came in and that was an answer to the problems that mapreduce was facing so this is how spark started in 2009 but what is spark if we have to Simply Define spark it is nothing but an open source so it initially it started with apache's Park and open source version of spark so that was a open source unified analytics engine for large-scale data processing and it was an answer to the problems that mapreduce was facing now let's talk about spark architecture and each of the
things that you need to know to understand Spark so first of all let's start with a very very basic when we talk about any spark application what would it consist of so it would have a driver so all of these are processes they are nothing but processors running on some VM or some machine now what it would have a spark application is a driver and a bunch of executors so you can think of driver as the brain or the heart which is driving the whole execution and there are a bunch of executors a cluster of
machines or VMS which are nothing but they are running executors they are actually doing the work so it's a combination of a driver and multiple executors and then we would have a cluster manager which will be managing this entire execution now let's look into each of these now what is a driver a driver is the main brain or heart of this whole uh spark because it it is the one which runs the main function the starting point the driver is sitting on one of the nodes in the cluster so since we are talking about a
distributed system there is definitely a cluster a set of machines which are connected to each other and functioning together so that huge data can be processed the data can be distributed across these nodes and processed so one of the nodes in the cluster would host the driver driver is nothing but a process what is the main job of a driver a driver's job is primarily uh what it does is it maintains the entire information about the spark application because it's like I said driver is the brain and heart of this running this whole spark application
so it maintains the entire information about the spark application it also responds to user programmer input and it is helping to analyze distribute schedule work across executors so the main work of distribution of scheduling is done by the driver so driver process is absolutely essential it is the heart of the spark application and it stores the entire information during the whole lifetime of the application so that's the driver then comes the executors what are the executors they are work horses they are the ones who are actually executing as the name suggests they are actually executing
the work given by the driver to them so they are responsible for executing the work they are responsible for reporting the state of the competition back to the driver executor is doing what the driver is telling them but at the same time they need to report back about what competition they are doing so that is the executor the third thing that we saw in that simple uh high level diagram was a cluster manager now what does a cluster manager do a cluster manager controls all the physical machines and allocates resources to spark applications so you
would also hear the term resource manager which the job of the resource manager is actually to allocate resources but if you talk about cluster manager think of it as a governing body who is looking at and controlling all the machines and allocating resources cluster manager can come like there can be different cluster managers it can be yeah Me shows us Sparks own Standalone cluster manager because in a cluster at one point of time there may be multiple spark applications that are running so there needs to be someone who manages those all the machines that are
there in the cluster in terms of allocation of resources managing so that job is done by a cluster manager so three fundamental things when we talk about spark application driver executor and cluster manager okay uh now let's talk about few more things uh like spark session what is the entry point for any spark application the first entry point or the fundamental beginning is Park session so we need to create an object of spark session to end uh start executing our spark code so that's the entry point and this is a snippet we can actually use
languages like Java python R to write a spark program so this is an example from java where we are importing a package and then we are creating a spark session object how it is done so that's the entry point now the other thing that is very very fundamental to spark or the basic building block is a data frame a data frame is the most common structured API that spark provides think of a data frame simplistically as a table which has rows and columns so list of columns and the type in those columns is the schema
what columns are there and their type consists of the schema but a data frame can be thought of as the basic building block of spark and it can be visualized as a table which has rows and columns then what is a partition so when we talk about data frame we are talking about huge data because it's a it's we are talking about Big Data terabytes of data so a data frame when we talk about a data frame it is the actual data you can visualize as row and column but in order for every executor to
actually do the execution there has to be a chunking of the data the whole of the data cannot be processed by a single executor so what is done is the data is partitioned so that every executor works on one partition so spark is breaking up the data into chunks called partitions so that it can be assigned to executors and each executor can work in parallel so partition how it should be visualized is it is just a chunk of rows which are sitting on a machine in our cluster because every executor will be allocated one partition
of the data so data is physically chunked into partitions a data frames partitions represent how the data is physically distributed so a data frame is nothing but a row and column but that row and column is chunked further so let's say we have 100 rows it can be passed partitioned into 10 partitions of 10 rows each this is just an example it is not 10 or some number but it is partitioned to create smaller chunks of data so that each executor can work on their own partition and Achieve parallelism so if there is only one
partition spark will only have a parallelism of one no matter how many executors we have in the cluster even if we have thousands of executors running in the cluster if the partition is only one the parallelism is only one if we have many partitions let's say and there is only one executor then also the parallelism is one because at one point of time the executor can execute only on one partition so ideally in a distributed setup when we are running spark there are multiple executors the cluster manager resource manager has decided which executors will participate
in executing uh this whole program and working on the data and accordingly the data partitions will go to those executors now another concept that is there is transformation what is transformation so in spark we talk about data frames and we also talk about rdds so rdd is nothing but it is a resilient distributed data set so what is happening is transformation means uh spark operations when they're they're always executed on rdds whatever program we write we will have Transformations and actions so transformation is where the operations are performed on an rdd and it will result
in a single creation of singular multiple rdd so rdd is a immutable a thing it can never be altered so if there is one rdd we are apply a transformation it will create another rdd again if a transformation is applied it will create another so rad is immutable it keeps on creating new rdds with each transformation and it can it actually creates a lineage of rdd starting from the first to the next so transformation is nothing but an operation which is executed on the rdds and it results in a new uh rdd and when so
another thing to understand is in spark we say that Transformations are lazy why do we say they are lazy because none of the transformation will get executed unless an action is called so there are two things transformation and action action is where actual action is happening meaning we can have multiple Transformations written but nothing is going to actually get executed until action is encountered and this is on purpose because when we write code we may write multiple transformations but when the when spark creates a logical and a physical plan it needs to optimize it so
so it waits till it encounters an action and then only it starts the execution so that's why we say spark has lazy transformation okay in Transformations also there are two types narrow and wide what does it mean as the name suggests narrow Transformations example if we use a map function or a filter function they will actually result in narrow Transformations why because it will not cause shuffling of data across the network there will not be any data movement between partitions and the data actually lives on a single partition so these are narrow as the name
suggests data is not getting shuffled across the network when there is data shuffling across the network it is going to take more time so and there is a huge uh bandwidth required time required com and it it basically consumes resources so we try to avoid as much as possible the shuffling of data so but it is not always possible that there will be no Shuffle of data that's why we say there are narrow Transformations and wider Transformations so what does a wider transformation mean a y transformation means that operations like group by Reduce by so
when we say Group by of course data from different partitions need to be brought somewhere to group it up so that means there is a huge shuffling of data happening across the nodes now this since there is a data movement and there are multiple partitions that are there and we are moving them across these are known as y Transformations or Shuffle Transformations because the data is shuffling around and the aim while writing or optimizing the program is to reduce white Transformations they cannot be eliminated but we can reduce them as far as possible now these
are very very fundamental terms that we need to know when we start learning about spark so this was the first part in the series in the next I will go into deeper um level where we will look at how actually your programs gets executed and what happens internally I hope this has helped you to this will help you to get started on your spark Journey please like share and subscribe to the channel to get more interesting videos thank you