Hadoop Installation steps

-

Environment required for Hadoop: The production environment of Hadoop is UNIX, but it can also be used in Windows using Cygwin. Java 1.6 or above is needed to run Map Reduce Programs. For Hadoop installation from tar ball on the UNIX environment you need

  1. Java Installation
  2. SSH installation
  3. Hadoop Installation and File Configuration

1) Java Installation

Step 1. Type "java -version" in prompt to find if the java is installed or not. If not then download java from http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html . The tar filejdk-7u71-linux-x64.tar.gz will be downloaded to your system.

 Step 2. Extract the file using the below command
#tar zxf jdk-7u71-linux-x64.tar.gz 
Step 3. To make java available for all the users of UNIX move the file to /usr/local and set the path. In the prompt switch to root user and then type the command below to move the jdk to /usr/lib.
# mv jdk1.7.0_71 /usr/lib/
  
Now in ~/.bashrc file add the following commands to set up the path.
# export JAVA_HOME=/usr/lib/jdk1.7.0_71
# export PATH=PATH:$JAVA_HOME/bin

 Now, you can check the installation by typing "java -version" in the prompt.

2) SSH Installation

SSH is used to interact with the master and slaves computer without any prompt for password. First of all create a Hadoop user on the master and slave systems

 # useradd hadoop  
# passwd Hadoop  
To map the nodes open the hosts file present in /etc/ folder on all the machines and put the ip address along with their host name.
# vi /etc/hosts
  
Enter the lines below
190.12.1.114    hadoop-master  
190.12.1.121    hadoop-salve-one  
190.12.1.143   hadoop-slave-two 

Set up SSH key in every node so that they can communicate among themselves without password. Commands for the same are:

# su hadoop   
$ ssh-keygen -t rsa   
$ ssh-copy-id -i ~/.ssh/id_rsa.pub tutorialspoint@hadoop-master   
$ ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop_tp1@hadoop-slave-1   
$ ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop_tp2@hadoop-slave-2   
$ chmod 0600 ~/.ssh/authorized_keys   
$ exit  

3) Hadoop Installation

Hadoop can be downloaded from http://developer.yahoo.com/hadoop/tutorial/module3.html
Now extract the Hadoop and copy it to a location.

$ mkdir /usr/hadoop  
$ sudo tar vxzf  hadoop-2.2.0.tar.gz ?c /usr/hadoop  

Change the ownership of Hadoop folder
$sudo chown -R hadoop  usr/hadoop 
Change the Hadoop configuration files:
All the files are present in /usr/local/Hadoop/etc/hadoop
1) In hadoop-env.sh file add
export JAVA_HOME=/usr/lib/jvm/jdk/jdk1.7.0_71  
2) In core-site.xml add following between configuration tabs,
<configuration>  
<property>  
<name>fs.default.name</name>  
<value>hdfs://hadoop-master:9000</value>  
</property>  
<property>  
<name>dfs.permissions</name>  
<value>false</value>  
</property>  
</configuration>  

3) In hdfs-site.xmladd following between configuration tabs,
<configuration>  
<property>  
<name>dfs.data.dir</name>  
<value>usr/hadoop/dfs/name/data</value>  
<final>true</final>  
</property>  
<property>  
<name>dfs.name.dir</name>  
<value>usr/hadoop/dfs/name</value>  
<final>true</final>  
</property>  
<property>  
<name>dfs.replication</name>  
<value>1</value>  
</property>  
</configuration>  

4) Open the Mapred-site.xml and make the change as shown below
<configuration>  
<property>  
<name>mapred.job.tracker</name>  
<value>hadoop-master:9001</value>  
</property>  
</configuration>
  
5) Finally, update your $HOME/.bahsrc
cd $HOME  
vi .bashrc  
Append following lines in the end and save and exit  
#Hadoop variables   
export JAVA_HOME=/usr/lib/jvm/jdk/jdk1.7.0_71  
export HADOOP_INSTALL=/usr/hadoop  
export PATH=$PATH:$HADOOP_INSTALL/bin   
export PATH=$PATH:$HADOOP_INSTALL/sbin  
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL   
export HADOOP_COMMON_HOME=$HADOOP_INSTALL  
export HADOOP_HDFS_HOME=$HADOOP_INSTALL   
export YARN_HOME=$HADOOP_INSTALL
  
On the slave machine install Hadoop using the command below
# su hadoop   
$ cd /opt/hadoop   
$ scp -r hadoop hadoop-slave-one:/usr/hadoop   
$ scp -r hadoop hadoop-slave-two:/usr/Hadoop 
Configure master node and slave node
$ vi etc/hadoop/masters  
hadoop-master  
$ vi etc/hadoop/slaves  
hadoop-slave-one   
hadoop-slave-two 
After this format the name node and start all the deamons
# su hadoop   
$ cd /usr/hadoop   
$ bin/hadoop namenode -format  
  
$ cd $HADOOP_HOME/sbin  
$ start-all.sh  

Comments

Post a Comment

Popular posts from this blog

How to download a file using command prompt (cmd) Windows?

-
Image
If you’re like most Internet users, you download various types of files now and then. You either download these files with your web browser or some download manager software. Have you ever considered downloading files using the command prompt i.e. CMD? If not, we recommend you to try it. It’s quite interesting apart from being very useful as well. In this article, we’ll show you how to download a file using command prompt aka CMD. How to download a file using command prompt?   There are several useful commands for the Command Prompt and one of them is  bitsadmin . Though bitsadmin has several uses, here we shall limit this article to its role in downloading files only. If you want to read the details about bitsadmin syntax, continue reading this section. Otherwise, you can scroll down to the next section –  Downloading a file .   The syntax of bitsadmin is : BITSADMIN [/RAWRETURN] [/WRAP | /NOWRAP] command   As you know, the parts of the syntax inside the square bra
Read more

The future of Artificial Intelligence: 6 ways it will impact everyday life

-
Image
Technology moves at breakneck speed, and we now have more power in our pockets than we had in our homes in the 1990s. Artificial intelligence (AI) has been a fascinating concept of science fiction for decades, but many researchers think we’re finally getting close to making AI a reality.  NPR notes that in the last few years, scientists have made breakthroughs in “machine learning,” using neural networks, which mimic the processes of real neurons. This is a type of “deep learning” that allows machines to process information for themselves on a very sophisticated level, allowing them to perform complex functions like facial recognition. Big data is speeding up the AI development process, and we may be seeing more integration of AI technology in our everyday lives relatively soon. While much of this technology is still fairly rudimentary at the moment, we can expect sophisticated AI to one day significantly impact our everyday lives. Here are 6 ways AI might affect us in the futu
Read more

How to Include ThreeJs in Your Projects

-
Image
HOW TO INCLUDE three.js IN YOUR PROJECTS There are quite a few different ways to include three.js in your JavaScript application, some of them simple, some of them a little more complex, however, they all boil down to one simple idea: you need to include one (and  only one) of these three files in your project: three.js three.min.js three.module.js You can view them all  on the three.js GitHub repo , which we’ll explore in the next chapter. Each of these files a version of the three.js core, meaning that once you’ve included one of them in your application, you are all ready to get started in creating beautiful 3D scenes. There are two main ways to include these files in your project: include them in a  script tag  in your HTML import them directly into your JavaScript files. We’ll explore how to do in Section Two What’s the Difference Between these Files? The  three.js  file is intended to be included in a  <script>  tag in your  index.html  file, like s
Read more