What is MapReduce?

in this video we are going to discuss what is MapReduce MapReduce is one of the main components in the Hadoop ecosystem in case of MapReduce what is happening we're having a huge set of data or data set that very data set will be divided into multiple smaller parts and they'll be assigned to the working nodes so that the processing can be done in parallel in a faster way on the working nodes the mapper methods will be working and this particular method mapper method will be working on the chunk of the data set or the part

of the data set assigned to that this input will be in the form of key value pairs key means the file references and values means that data sets the output of this mapper methods will be obtained as the input to the reducer method the reduce it will execute some functions these functions are the custom functions depending upon the business need some functions will be operating on this particular output of mapper methods and those functions will do some aggregation some processing on data then the reducer will produce a final output and that will be also in

the form of key value pairs and that is a basic concept of this MapReduce so let us go for further more discussions with some diagrams and examples to discuss what is MapReduce so here you are having this input and here we are having these map tasks and then we're having the reduced tasks will be there whether reduced method will be working here it was working as map methods here it will be working the reduced methods and the final output will be obtained in the aggregated form that is the basic theme behind this MapReduce so the

MapReduce is one of the main components of the Hadoop ecosystem in our Hadoop ecosystem video we have discussed there are so many components are there under this Hadoop ecosystem that also we had that MapReduce ecosystem component MapReduce is designed to produce a large amount of data in parallel by dividing the work into some smaller pieces and independent tasks so large amount of data will not be processed at a time it will be divided into smaller pieces and those pieces will be assigned to the working nodes and those tasks will be executed in parallel for the

faster processing the whole job is taken from the user and divided into smaller tasks and assign them to the working nodes MapReduce programs take input as a list and convert to the output also as a list so it will take the input as a list and it also converts output in the form of a list so let us go for some further criterias the map or the mapper tasks a set of keys and values we can say that it is it is as a key value pair as input so this particular data will be in

the form of key value pairs now questions might be coming in mind that what is key and what is the value so key is actually nothing but our reference to our data set and values are nothing but the data sets so key can be treated as a reference to a data set or reference to a file and the value is nothing but our data set the data may be in a structured or unstructured form and the framework can make it into keys and values so the data set may be in the structured that means in

the form of say database and databases tables where the data can be divided or can be represented in the form of rows and columns and unstructured mid where will be going for takes files PDFs we're having the images were having the videos and they'll be known as the unstructured data the framework can make it into keys and values the key are the difference of input files and values are the data sets that user can create a custom business logic based on their need for the data processing so what kind of processing will be done that

can be customized depending upon the business need the respective operations will be carried out the respective processing will be carried out on the data set the task is applied on every input value now we are going for the tasks the reducer takes the key value pair which it which is created by the mapper as input so mapper is taking the input and mappers output will be the input to the reducer and reducer will produce the respective output accordingly the key value pairs are shorted by the key elements in case of reducers and in the reducer

we perform the shorting aggregation or summarization type of jobs that means here we are going for some aggregation type of job we can go for say summation we can go for say counting we can go for say maximum minimum calculations and so on how MapReduce task works so now let us go for the the macro view of the system the given inputs are processed by the user different methods all different business logics are working on the mapper section so business logic will be working at the mapper section and the mapper generates intermediate data and reducers

takes them as input as I told you earlier that the output of the mapper will be the input to the respective reducer that data are processed by the user different function in the reducer section so the data processing will be done at the reducer section depending upon the business logic depending upon the user different function or operation and the final output is stored in HDFS that is Hadoop distributed file system where the final result will be stored there now let us go through one proper diagram here so you are having here is the FS split

so multiple splits are there so Hadoop distributed file system split will be there so input will be key value pairs in this way it will be the input will be obtained to the respective mappers so now here the mapping is taking place and it will be now it will be dealing with multiple keys and the respective value so q1 value one two key k2 value K in this way the mapper will be working now their outputs will be coming to this saffle and short so here the aggregate values by the key so depending upon the

same key so they will get aggregated then the result will be obtained that means the output of this shuffle and short opera we'll be obtained to the reducer so these are the reduce methods are working here so we are having the q1 intermediate values and then key K intermediate values will be coming to that respective reducers and then final q1 and final value and final key K and final value will be obtained in this way so initially we are having this key K q1 value 1 he came value K in this way we're having then

here we are having this suffle and shot then the reduced method will be working on it then we are fine finally getting final key one value final key K then final value in this way that things will be obtained as output the pictorial representation on how the MapReduce task works we have shown that one in this diagram so let us go for another elaborate example for the better understanding so here we are having one example here so deer beer River car car River dear God beer so these are the the value sets were having so

set of values were having so now they'll be splitted in this way the splitting has been done now the mapper is there so here the mapping is taking place so I'm finding dear for one count beer for counter one and River for count one card for count to one car for count one and river for counter one in this way the mapping is taking place here you are having this suppling and shorting so here you can find we're having this beer only the beer keys are there or the car keys are there only the deer

keys are there only the river keys are there then here we are going for the reducing so this beer has occurred for two so here the aggregation method is count actually so here car has offered for three times deer has occurred for two times and even has occurred for two times so here this is my reducer which is doing the reducing and here you are having this final result that is beer two car three deer two and liver two so here you can find that how the overall map produced what count process process is getting

executed into multiple different phases so this example is a very interesting one and also it is clear our doubts and it is cleared of a conception also the pictorial representation on how the map produced a squawks we have shown that one in this way so in this video we have got the idea that is what is nap reduce thanks for watching this video