CDH 5.3.1
Cloudera’s 100% Open Source Hadoop Platform
CDH is Cloudera's open source software distribution and consists of Apache Hadoop and additional key open source projects to ensure you get the most out of Hadoop and your data.
It is the only Hadoop solution to offer unified querying options (including batch processing, interactive SQL, text search, and machine learning) and necessary enterprise security features (such as role-based access controls).
Version:
Please note: CDH requires manual installation from the command line.For a faster, automated installation download 
Installing the Latest CDH 5 Release
Important:
If you use Cloudera Manager, do not use these command-line instructions.
This information applies specifically to CDH 5.3.x. If you use an earlier version of CDH, see the documentation for that version located at Cloudera Documentation.
Ways To Install CDH 5
You can install CDH 5 in any of the following ways:
Install Cloudera Manager, CDH, and managed services in a .
Note: Cloudera recommends that you use this automated method if possible.Or use one of the manual methods described below:
If you use one of these manual methods rather than Cloudera Manager, the first (downloading and installing the "1-click Install" package) is recommended in most cases because it is simpler than building or adding a repository.
Download and install the CDH 5 1-click Install" package; OR
Add the CDH 5 repository; OR
Build your own CDH 5 repository
Install from a CDH 5 tarball — see, the next topic, "How Packaging Affects CDH 5 Deployment".
How Packaging Affects CDH 5 Deployment
Installing from Packages
To install and deploy YARN, follow the directions on this page and proceed with .
To install and deploy MRv1, follow the directions on this page and then proceed with .
Installing from a Tarball
Note: The instructions in this Installation Guide are tailored for a package installation, as described in the sections that follow, and do not cover installation or deployment from tarballs.
If you install CDH 5 from a tarball, you will install YARN.
In CDH 5, there is no separate tarball for MRv1. Instead, the MRv1 binaries, examples, etc., are delivered in the Hadoop tarball itself. The scripts for running MRv1 are in the bin-mapreduce1 directory in the tarball, and the MRv1 examples are in the examples-mapreduce1 directory.
Before You Begin Installing CDH 5 Manually
The instructions on this page are for new installations. If you need to upgrade from an earlier release, see .
For a list of supported operating systems, see .
These instructions assume that the sudo command is configured on the hosts where you will be doing the installation. If this is not the case, you will need the root user (superuser) to configure it.
Note:
If you are migrating from MapReduce v1 (MRv1) to MapReduce v2 (MRv2, YARN), see for important information and instructions.
Important: Running Services
When starting, stopping and restarting CDH components, always use the service (8) command rather than running scripts in /etc/init.d directly. This is important because service sets the current working directory to / and removes most environment variables (passing only LANG and TERM), to create a predictable environment for the service. If you run the scripts in /etc/init.d, locally-set environment variables could produce unpredictable results. If you install CDH from RPMs, service will be installed as part of the Linux Standard Base (LSB).
Important: Java Development Kit:
if you have not already done so, install the Oracle Java Development Kit (JDK); see .
High Availability
In CDH 5 you can configure high availability both for the NameNode and the JobTracker or Resource Manager.
For more information and instructions on setting up a new HA configuration, see .
Important:If you decide to configure , do not install hadoop-hdfs-secondarynamenode. After completing the , follow the installation instructions under .
To upgrade an existing configuration, follow the instructions under .
Steps to Install CDH 5 Manually
Step 1: Add or Build the CDH 5 Repository or Download the "1-click Install" package.
If you are installing CDH 5 on a system, you can download Cloudera packages using yum or your web browser.
If you are installing CDH 5 on a system, you can download the Cloudera packages using zypper or YaST or your web browser.
If you are installing CDH 5 on an system, you can download the Cloudera packages using apt or your web browser.
On Red Hat-compatible Systems
Use one of the following methods to add or build the CDH 5 repository or download the package on Red Hat-compatible systems. Note:
Use only one of the three methods.
OR
OR
Do this on all the systems in the cluster.
To download and install the CDH 5 "1-click Install" package:
Click the entry in the table below that matches your Red Hat or CentOS system, choose Save File, and save the file to a directory to which you have write access (it can be your home directory).
OS Version Click this Link Red Hat/CentOS/Oracle 5 Red Hat/CentOS/Oracle 6 Install the RPM. For Red Hat/CentOS/Oracle 5:
$ sudo yum --nogpgcheck localinstall cloudera-cdh-5-0.x86_64.rpm
For Red Hat/CentOS/Oracle 6 (64-bit):
$ sudo yum --nogpgcheck localinstall cloudera-cdh-5-0.x86_64.rpm
Now continue with , and then choose , or ; or do both steps if you want to install both implementations.
Note: Make sure your repositories are up to dateBefore proceeding, make sure the repositories on each system are up to date:
sudo yum clean all
This ensures that the system repositories contain the latest software (it does not actually install anything).
OR: To add the CDH 5 repository:
Click the entry in the table below that matches your RHEL or CentOS system, navigate to the repo file for your system and save it in the /etc/yum.repos.d/directory.
For OS Version | Click this Link |
|---|---|
RHEL/CentOS/Oracle 5 | |
RHEL/CentOS/Oracle 6 (64-bit) |
Now continue with , and then choose , or ; or do both steps if you want to install both implementations.
Note: Make sure your repositories are up to dateBefore proceeding, make sure the repositories on each system are up to date:
sudo yum clean all
This ensures that the system repositories contain the latest software (it does not actually install anything).
OR: To build a Yum repository:
If you want to create your own yum repository, download the appropriate repo file, create the repo, distribute the repo file and set up a web server, as described under .
Now continue with , and then choose , or ; or do both steps if you want to install both implementations.
Note: Make sure your repositories are up to dateBefore proceeding, make sure the repositories on each system are up to date:
sudo yum clean all
This ensures that the system repositories contain the latest software (it does not actually install anything).
On SLES Systems
Use one of the following methods to download the CDH 5 repository or package on SLES systems. Note:
Use only one of the three methods.
OR
OR
To download and install the CDH 5 "1-click Install" package:
Download the CDH 5 "1-click Install" package.
Click , choose Save File, and save it to a directory to which you have write access (for example, your home directory).
Install the RPM:
$ sudo rpm -i cloudera-cdh-5-0.x86_64.rpm
Update your system package index by running:
$ sudo zypper refresh
Now continue with , and then choose , or ; or do both steps if you want to install both implementations.
OR: To add the CDH 5 repository:
Run the following command:
$ sudo zypper addrepo -f http://archive.cloudera.com/cdh5/sles/11/x86_64/cdh/cloudera-cdh5.repo
Update your system package index by running:
$ sudo zypper refresh
Now continue with , and then choose , or ; or do both steps if you want to install both implementations.
Note: Make sure your repositories are up to dateBefore proceeding to the next step, make sure the repositories on each system are up to date:
sudo zypper clean --all
This ensures that the system repositories contain the latest software (it does not actually install anything).
OR: To build a SLES repository:
If you want to create your own SLES repository, create a mirror of by following that explain how to create a SLESrepository from the mirror.
Now continue with , and then choose , or ; or do both steps if you want to install both implementations.
Note: Make sure your repositories are up to dateBefore proceeding to the next step, make sure the repositories on each system are up to date:
sudo zypper clean --all
This ensures that the system repositories contain the latest software (it does not actually install anything).
On Ubuntu or Debian Systems
Use one of the following methods to download the CDH 5 repository or package.
OR
OR
Note:
Use only one of the three methods.
There is an extra step if you are adding a repository on Ubuntu Trusty, as described below.
Unless you are adding a repository on Ubuntu Trusty, don't forget to run apt-get update after downloading, adding, or building the repository.
To download and install the CDH 5 "1-click Install" package:
Download the CDH 5 "1-click Install" package:
OS Version Click this Link Wheezy Precise Trusty Install the package by doing one of the following:
Choose Open with in the download window to use the package manager.
Choose Save File, save the package to a directory to which you have write access (for example, your home directory), and install it from the command line. For example:
sudo dpkg -i cdh5-repository_1.0_all.deb
Note: Make sure your repositories are up to dateBefore proceeding to the next step, make sure the repositories on each system are up to date:
sudo apt-get update
This ensures that the system repositories contain the latest software (it does not actually install anything).
Now continue with , and then choose , or ; or do both steps if you want to install both implementations.
OR: To add the CDH 5 repository:
Download the appropriate cloudera.list file by issuing one of the following commands. You can use another HTTP client if wget is not available, but the syntax may be different.
Important: Ubuntu 14.04 (Trusty)If you are running Ubuntu Trusty, you need to perform an additional step after adding the repository. See "Additional Step for Trusty" below.
OS Version Command Debian Wheezy $ sudo wget 'http://archive.cloudera.com/cdh5/ubuntu/wheezy/amd64/cdh/cloudera.list' \ -O /etc/apt/sources.list.d/cloudera.list
Ubuntu Precise $ sudo wget 'http://archive.cloudera.com/cdh5/ubuntu/wheezy/amd64/cdh/cloudera.list' \ -O /etc/apt/sources.list.d/cloudera.list
Ubuntu Lucid $ sudo wget 'http://archive.cloudera.com/cdh5/ubuntu/lucid/amd64/cdh/cloudera.list' \ -O /etc/apt/sources.list.d/cloudera.list
Ubuntu Trusty $ sudo wget 'http://archive.cloudera.com/cdh5/ubuntu/trusty/amd64/cdh/cloudera.list' \ -O /etc/apt/sources.list.d/cloudera.list
Note: Make sure your repositories are up to dateUnless you are adding a repository on Ubuntu Trusty, make sure the repositories on each system are up to date before proceeding to the next step:
sudo apt-get update
This ensures that the system repositories contain the latest software (it does not actually install anything).
Additional step for Trusty
This step ensures that you get the right ZooKeeper package for the current CDH release. You need to prioritize the Cloudera repository you have just added, such that you install the CDH version of ZooKeeper rather than the version that is bundled with Ubuntu Trusty.
To do this, create a file at /etc/apt/preferences.d/cloudera.pref with the following contents:
Package: * Pin: release o=Cloudera, l=Cloudera Pin-Priority: 501
Note:
You do not need to run apt-get update after creating this file.
Now continue with , and then choose , or ; or do both steps if you want to install both implementations.
OR: To build a Debian repository:
If you want to create your own apt repository, create a mirror of and then .
Note: Make sure your repositories are up to dateBefore proceeding to the next step, make sure the repositories on each system are up to date:
sudo apt-get update
This ensures that the system repositories contain the latest software (it does not actually install anything).
Now continue with , and then choose , or ; or do both steps if you want to install both implementations.
Step 2: Optionally Add a Repository Key
Before installing YARN or MRv1: (Optionally) add a repository key on each system in the cluster. Add the Cloudera Public GPG Key to your repository by executing one of the following commands:
For Red Hat/CentOS/Oracle 5 systems:
$ sudo rpm --import http://archive.cloudera.com/cdh5/redhat/5/x86_64/cdh/RPM-GPG-KEY-cloudera
For Red Hat/CentOS/Oracle 6 systems:
$ sudo rpm --import http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/RPM-GPG-KEY-cloudera
For all SLES systems:
$ sudo rpm --import http://archive.cloudera.com/cdh5/sles/11/x86_64/cdh/RPM-GPG-KEY-cloudera
For Ubuntu or Debian systems:
OS Version Command Debian Wheezy $ wget http://archive.cloudera.com/cdh5/debian/wheezy/amd64/cdh/archive.key -O archive.key $ sudo apt-key add archive.key
Ubuntu Precise $ wget http://archive.cloudera.com/cdh5/ubuntu/precise/amd64/cdh/archive.key -O archive.key $ sudo apt-key add archive.key
Ubuntu Lucid $ wget http://archive.cloudera.com/cdh5/ubuntu/lucid/amd64/cdh/archive.key -O archive.key $ sudo apt-key add archive.key
Ubuntu Trusty $ wget http://archive.cloudera.com/cdh5/ubuntu/trusty/amd64/cdh/archive.key -O archive.key $ sudo apt-key add archive.key
This key enables you to verify that you are downloading genuine packages.
Step 3: Install CDH 5 with YARN
Note:
Skip this step if you intend to use only MRv1. Directions for installing MRv1 are in .
To install CDH 5 with YARN:
Note:
If you decide to configure, do not install hadoop-hdfs-secondarynamenode. After completing the , follow the installation instructions under .
Install and deploy ZooKeeper.
Important:Cloudera recommends that you install (or update) and start a ZooKeeper cluster before proceeding. This is a requirement if you are deploying high availability (HA) for the NameNode.
Follow instructions under .
Install each type of daemon package on the appropriate systems(s), as follows.
Where to install
Install commands
Resource Manager host (analogous to MRv1 JobTracker) running:
Red Hat/CentOS compatible
sudo yum clean all; sudo yum install hadoop-yarn-resourcemanager
SLES
sudo zypper clean --all; sudo zypper install hadoop-yarn-resourcemanager
Ubuntu or Debian
sudo apt-get update; sudo apt-get install hadoop-yarn-resourcemanager
NameNode host running:
Red Hat/CentOS compatible
sudo yum clean all; sudo yum install hadoop-hdfs-namenode
SLES
sudo zypper clean --all; sudo zypper install hadoop-hdfs-namenode
Ubuntu or Debian
sudo apt-get install hadoop-hdfs-namenode
Secondary NameNode host (if used) running:
Red Hat/CentOS compatible
sudo yum clean all; sudo yum install hadoop-hdfs-secondarynamenode
SLES
sudo zypper clean --all; sudo zypper install hadoop-hdfs-secondarynamenode
Ubuntu or Debian
sudo apt-get install hadoop-hdfs-secondarynamenode
All cluster hosts except the Resource Manager running:
Red Hat/CentOS compatible
sudo yum clean all; sudo yum install hadoop-yarn-nodemanager hadoop-hdfs-datanode hadoop-mapreduce
SLES
sudo zypper clean --all; sudo zypper install hadoop-yarn-nodemanager hadoop-hdfs-datanode hadoop-mapreduce
Ubuntu or Debian
sudo apt-get install hadoop-yarn-nodemanager hadoop-hdfs-datanode hadoop-mapreduce
One host in the cluster running:
Red Hat/CentOS compatible
sudo yum clean all; sudo yum install hadoop-mapreduce-historyserver hadoop-yarn-proxyserver
SLES
sudo zypper clean --all; sudo zypper install hadoop-mapreduce-historyserver hadoop-yarn-proxyserver
Ubuntu or Debian
sudo apt-get install hadoop-mapreduce-historyserver hadoop-yarn-proxyserver
All client hosts running:
Red Hat/CentOS compatible
sudo yum clean all; sudo yum install hadoop-client
SLES
sudo zypper clean --all; sudo zypper install hadoop-client
Ubuntu or Debian
sudo apt-get install hadoop-client
Note:
The hadoop-yarn and hadoop-hdfs packages are installed on each system automatically as dependencies of the other packages.
Step 4: Install CDH 5 with MRv1
Note:
If you are also installing YARN, you can skip any packages you have already installed in .
Skip this step and go to if you intend to use only YARN.
Important: Before proceeding, you need to decide:
Whether to configure High Availability (HA) for the NameNode and/or JobTracker; see the for more information and instructions.
Where to deploy the NameNode, Secondary NameNode, and JobTracker daemons. As a general rule:
The NameNode and JobTracker run on the same "master" host unless the cluster is large (more than a few tens of nodes), and the master host (or hosts) should not run the Secondary NameNode (if used), DataNode or TaskTracker services.
In a large cluster, it is especially important that the Secondary NameNode (if used) runs on a separate machine from the NameNode.
Each node in the cluster except the master host(s) should run the DataNode and TaskTracker services.
If you decide to configure , do not install hadoop-hdfs-secondarynamenode. After completing the , follow the installation instructions under .
First, install and deploy ZooKeeper. Important:
Cloudera recommends that you install (or update) and start a ZooKeeper cluster before proceeding. This is a requirement if you are deploying high availability (HA) for the NameNode or JobTracker.
Follow instructions under . Make sure you create the myid file in the data directory, as instructed, if you are starting a ZooKeeper ensemble after a fresh install.
Next, install packages.
Install each type of daemon package on the appropriate systems(s), as follows. Note:
On Ubuntu systems, Ubuntu may try to start the service immediately after you install it. This should fail harmlessly, but if you want to prevent it, there is advice .
Where to install | Install commands |
|---|---|
JobTracker host running: | |
Red Hat/CentOS compatible | sudo yum clean all; sudo yum install hadoop-0.20-mapreduce-jobtracker |
SLES | sudo zypper clean --all; sudo zypper install hadoop-0.20-mapreduce-jobtracker |
Ubuntu or Debian | sudo apt-get update; sudo apt-get install hadoop-0.20-mapreduce-jobtracker |
NameNode host running: | |
Red Hat/CentOS compatible | sudo yum clean all; sudo yum install hadoop-hdfs-namenode |
SLES | sudo zypper clean --all; sudo zypper install hadoop-hdfs-namenode |
Ubuntu or Debian | sudo apt-get install hadoop-hdfs-namenode |
Secondary NameNode host (if used) running: | |
Red Hat/CentOS compatible | sudo yum clean all; sudo yum install hadoop-hdfs-secondarynamenode |
SLES | sudo zypper clean --all; sudo zypper install hadoop-hdfs-secondarynamenode |
Ubuntu or Debian | sudo apt-get install hadoop-hdfs-secondarynamenode |
All cluster hosts except the JobTracker, NameNode, and Secondary (or Standby) NameNode hosts running: | |
Red Hat/CentOS compatible | sudo yum clean all; sudo yum install hadoop-0.20-mapreduce-tasktracker hadoop-hdfs-datanode |
SLES | sudo zypper clean --all; sudo zypper install hadoop-0.20-mapreduce-tasktracker hadoop-hdfs-datanode |
Ubuntu or Debian | sudo apt-get install hadoop-0.20-mapreduce-tasktracker hadoop-hdfs-datanode |
All client hosts running: | |
Red Hat/CentOS compatible | sudo yum clean all; sudo yum install hadoop-client |
SLES | sudo zypper clean --all; sudo zypper install hadoop-client |
Ubuntu or Debian | sudo apt-get install hadoop-client |
Step 5: (Optional) Install LZO
If you decide to install LZO ( Lempel–Ziv–Oberhumer compression), proceed as follows. For information about choosing a compression format, see Note:If you are upgrading to a new version of LZO, rather than installing it for the first time, you must first remove the old version; for example, on a RHEL system:
yum remove hadoop-lzo
Add the repository on each host in the cluster. Follow the instructions for your OS version:
For OS Version Do this Red Hat/CentOS/Oracle 5 Navigate to and save the file in the /etc/yum.repos.d/ directory. Red Hat/CentOS 6 Navigate to and save the file in the /etc/yum.repos.d/ directory. SLES Ubuntu or Debian Navigate to and save the file as /etc/apt/sources.list.d/gplextras.list. Important: Make sure you do not let the file name default to cloudera.list, as that will overwrite your existing cloudera.list.Run the following command:
$ sudo zypper addrepo -f http://archive.cloudera.com/gplextras5/sles/11/x86_64/gplextras/ cloudera-gplextras5.repo
Update your system package index by running:
$ sudo zypper refresh
Install the package on each host as follows:
For OS version Install commands Red Hat/CentOS compatible sudo yum install hadoop-lzo
SLES sudo zypper install hadoop-lzo
Ubuntu or Debian sudo apt-get install hadoop-lzo
Continue with installing and deploying CDH. As part of the deployment, you will need to do some additional configuration for LZO, as shown under.
Important: Make sure you do this configuration after you have to a custom location and set alternatives to point to it.
Step 6: Deploy CDH and Install Components
Now proceed with:
.