Configuring the amount of storage shared by Slave to Master in Hadoop Cluster
In this blog, you will get to know how to contribute limited/specific amount of storage from Data node to the Master Node in a Hadoop Cluster. Here, I am setting up the cluster on AWS. I already have created two EC2 instance with RedHat Linux AMI and installed hadoop and jdk software.
I have already configured the hdfs-site.xml and core-site.xml file in the data node and the master node, and have the cluster running.
Note: The data node is contributing it’s whole size i.e 10Gb.
Now, we are going to customize the size of storage provided by Data node. Let’s make a EBS volume and attach to the Data Node. Here, i have done using AWSCLI.
We can cross check by going to AWS Portal
Now that we have a extra storage attached to device. We need to partition, format and mount to a folder so that we can use it.
Partition the newly attached volume and provide the size of partition as 1Gb
Now, format the partition created and make a new folder and mount the formatted partition to it.
Final step is to update the hdfs-site.xml file. Just change the name of folder to the newly created folder where the EBS storage of 1GB is mounted. Here the folder name i gave is updatedDataNode.
Restart the service and we can see that the size of storage provided by Data Node is changed to 1Gb.
