dotlinux blog

Install Hadoop Multinode Cluster using CDH4 in RHEL/CentOS 6.5

In the era of big data, Hadoop has become a cornerstone for distributed data storage and processing. Cloudera Distribution including Apache Hadoop (CDH) provides a reliable, tested, and optimized version of Hadoop. In this blog post, we will walk you through the step - by - step process of installing a Hadoop multinode cluster using CDH4 on RHEL/CentOS 6.5 systems. This guide assumes a basic understanding of Linux systems and networking concepts.

2026-06

Table of Contents#

  1. Prerequisites
  2. Cluster Nodes Setup
  3. Installing CDH4 Repositories
  4. Configuring SSH Key - Based Authentication
  5. Installing and Configuring Hadoop Components
  6. Starting the Hadoop Cluster
  7. Verifying the Cluster Installation
  8. Troubleshooting Common Issues
  9. References

1. Prerequisites#

  • Hardware: A minimum of three nodes (one master and at least two slaves) with sufficient RAM (recommended 4GB or more per node), CPU, and storage.
  • Operating System: RHEL/CentOS 6.5 installed on all nodes.
  • Network: Ensure that all nodes are connected to the same network and can communicate with each other using their hostnames.
  • Root Access: You should have root access on all nodes for the installation process.

2. Cluster Nodes Setup#

2.1 Network Configuration#

  • On each node, set a static IP address. Edit the /etc/sysconfig/network - scripts/ifcfg - eth0 file (adjust the interface name according to your system) to configure the static IP, subnet mask, gateway, and DNS.
DEVICE="eth0"
BOOTPROTO="static"
IPADDR=192.168.1.100 # Replace with your desired IP
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
DNS1=8.8.8.8
ONBOOT="yes"
  • Update the /etc/hosts file on all nodes to contain the mapping of IP addresses to hostnames. For example:
192.168.1.100 master
192.168.1.101 slave1
192.168.1.102 slave2

2.2 Disable SELinux#

SELinux can interfere with the Hadoop installation. Edit the /etc/selinux/config file and set SELINUX=disabled. Then, reboot the system or run setenforce 0 to disable it temporarily.

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
setenforce 0

2.3 Disable Firewall#

The firewall can block the communication between Hadoop nodes. Disable the iptables service on all nodes.

service iptables stop
chkconfig iptables off

3. Installing CDH4 Repositories#

3.1 Download the CDH4 Repository File#

On all nodes, download the CDH4 repository file from Cloudera's official repository.

wget http://archive.cloudera.com/cdh4/redhat/6/x86_64/cdh/cloudera-cdh4.repo
mv cloudera-cdh4.repo /etc/yum.repos.d/

3.2 Import the GPG Key#

Import the GPG key to verify the packages from the CDH4 repository.

rpm --import http://archive.cloudera.com/cdh4/redhat/6/x86_64/cdh/RPM-GPG-KEY-cloudera

3.3 Update the Package List#

Run the following command to update the local package list.

yum update

4. Configuring SSH Key - Based Authentication#

4.1 Generate SSH Keys on the Master Node#

ssh-keygen -t rsa

Press Enter for all prompts to use the default settings.

4.2 Copy the Public Key to Slaves#

Copy the public key from the master node to all slave nodes. Replace slave1 with the actual hostname of the slave node.

ssh-copy-id root@slave1

You will be prompted to enter the root password of the slave node. Repeat this step for all slave nodes.

4.3 Test SSH Connection#

Test the SSH connection from the master node to the slave nodes without a password.

ssh slave1

If you can log in without entering a password, the SSH key - based authentication is set up correctly.

5. Installing and Configuring Hadoop Components#

5.1 Install Hadoop Components on the Master Node#

yum install hadoop-0.20-mapreduce-jobtracker hadoop-0.20-mapreduce-tasktracker hadoop-hdfs-namenode hadoop-hdfs-datanode hadoop-hdfs-secondarynamenode

5.2 Install Hadoop Components on Slave Nodes#

On each slave node, run the following command:

yum install hadoop-0.20-mapreduce-tasktracker hadoop-hdfs-datanode

5.3 Configure Hadoop#

  • core-site.xml: Edit the /etc/hadoop/conf/core-site.xml file on all nodes to set the Hadoop filesystem URI.
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://master:8020</value>
    </property>
</configuration>
  • hdfs - site.xml: Edit the /etc/hadoop/conf/hdfs - site.xml file on all nodes to set the replication factor and other HDFS - related properties.
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
</configuration>
  • mapred - site.xml: Edit the /etc/hadoop/conf/mapred - site.xml file on all nodes to set the jobtracker address.
<configuration>
    <property>
        <name>mapred.job.tracker</name>
        <value>master:8021</value>
    </property>
</configuration>

5.4 Create HDFS Directories#

On the master node, create the necessary HDFS directories.

mkdir -p /data/hdfs/namenode
mkdir -p /data/hdfs/datanode
chown -R hdfs:hdfs /data/hdfs

6. Starting the Hadoop Cluster#

6.1 Format the Namenode#

On the master node, format the HDFS namenode.

sudo -u hdfs hdfs namenode -format

6.2 Start HDFS Services#

On the master node, start the HDFS services.

service hadoop-hdfs-namenode start
service hadoop-hdfs-secondarynamenode start

On all nodes (both master and slaves), start the datanode service.

service hadoop-hdfs-datanode start

6.3 Start MapReduce Services#

On the master node, start the jobtracker and tasktracker services.

service hadoop-0.20-mapreduce-jobtracker start
service hadoop-0.20-mapreduce-tasktracker start

7. Verifying the Cluster Installation#

7.1 Check HDFS Status#

On the master node, check the HDFS status.

sudo -u hdfs hdfs dfsadmin -report

This command should show information about the datanodes, such as the number of live datanodes.

7.2 Run a Sample MapReduce Job#

Create a sample text file on the master node.

echo "Hello, Hadoop!" > test.txt

Upload the file to HDFS.

sudo -u hdfs hdfs dfs -put test.txt /user/root/test.txt

Run a simple word count MapReduce job.

sudo -u hdfs hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-0.20-mapreduce-examples.jar wordcount /user/root/test.txt /user/root/output

Check the output of the job.

sudo -u hdfs hdfs dfs -ls /user/root/output

8. Troubleshooting Common Issues#

  • Network Connectivity: If nodes cannot communicate, check the IP addresses, hostnames, and firewall settings. Use ping and telnet commands to test network connectivity between nodes.
  • Permission Issues: Make sure that the HDFS directories have the correct ownership and permissions. You may need to run chown and chmod commands as the hdfs user.
  • Service Startup Failures: Check the service log files in /var/log/hadoop - * to find the root cause of the service startup failures.

References#