准备工作：
1、笔记本4G内存 ，操作系统WIN7 （屌丝的配置）
2、工具VMware Workstation
3、虚拟机：CentOS6.4共四台
虚拟机设置：
每台机器：内存512M，硬盘40G，网络适配器：NAT模式
选择高级，新生成虚机Mac地址（克隆虚拟机，Mac地址不会改变，每次最后手动重新生成）
编辑虚拟机网络：
点击NAT设置，查看虚机网关IP，并记住它，该IP在虚机的网络设置中非常重要。
NAT设置默认IP会自动生成，但是我们的集群中IP需要手动设置。
本机Win7 ：VMnet8 网络设置
实验环境：
| Ip | hostname | role |
| 192.168.249.130 | SY-0130 | ActiveNameNode |
| 192.168.249.131 | SY-0131 | StandByNameNode |
| 192.168.249.132 | SY-0132 | DataNode1 |
| 192.168.249.133 | SY-0133 | DataNode2 |
Linux网络设置：
1、新建用户如：hadoop。不建议使用root用户搭建集群（root权限过大）
2、使得hadoop用户获得sudo权限。
[root@SY-0130 ~]# vi /etc/sudoers## Allow root to run any commands anywhereroot ALL=(ALL) ALLhadoop ALL=(ALL) ALL
3、查看当前虚机当前网络使用网卡设备
[root@SY-0130 hadoop]# ifconfigeth2 Link encap:Ethernet HWaddr 00:50:56:35:8E:E8 inet addr:192.168.249.130 Bcast:192.168.249.255 Mask:255.255.255.0 inet6 addr: fe80::250:56ff:fe35:8ee8/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:877059 errors:0 dropped:0 overruns:0 frame:0 TX packets:597769 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:865720294 (825.6 MiB) TX bytes:324530557 (309.4 MiB) Interrupt:19 Base address:0x2024 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:1354 errors:0 dropped:0 overruns:0 frame:0 TX packets:1354 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:196675 (192.0 KiB) TX bytes:196675 (192.0 KiB)
[root@SY-0130 ~]# cat /proc/net/dev，当前网卡，我的为eth2
[root@SY-0130 ~]# cat /proc/net/devInter-| Receive | Transmit face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed lo: 196675 1354 0 0 0 0 0 0 196675 1354 0 0 0 0 0 0 eth2:865576893 875205 0 0 0 0 0 0 324425517 596433 0 0 0 0 0 0
4、查看当前网卡对应的Mac地址
[root@SY-0130 ~]# cat /etc/udev/rules.d/70-persistent-net.rules # This file was automatically generated by the /lib/udev/write_net_rules# program, run by the persistent-net-generator.rules rules file.## You can modify it, as long as you keep each rule on a single# line, and change only the value of the NAME= key.# PCI device 0x1022:0x2000 (vmxnet)SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:b5:fd:bb", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"# PCI device 0x1022:0x2000 (vmxnet)SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:35:8e:e8", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"
5、Network Configuration
[root@SY-0130 ~]# setup
选择eth2，进行设置，更改为当前网卡设备eth2，并且进行IP、网管、DNS设置。
DNS Server 与Win7的网络中的DNS Server一致，这样虚拟机也可以连接Internet网了，方便下载安装软件。
另外还有将防火墙关闭。
6、修改hostname
[root@SY-0130 ~]# vi /etc/sysconfig/networkNETWORKING=yesHOSTNAME=SY-0130
7、修改hosts
[hadoop@SY-0130 ~]$ sudo vi /etc/hosts
#添加如下内容
192.168.249.130 SY-0130192.168.249.131 SY-0131192.168.249.132 SY-0132192.168.249.133 SY-0133
8、重启虚机. reboot
软件安装：
(注：用户hadoop登录SY-130)
1、SY-130用户目录创建toolkit 文件夹，用来保存所有软件安装包，建立labc文件作为本次实验环境目录。
[hadoop@SY-0130 ~]$ mkdir labc
[hadoop@SY-0130 ~]$ mkdir toolkit
[hadoop@SY-0130 ~]$ ls
labc toolkit
#我将下载的软件包存放在toolkit中如下
[hadoop@SY-0130 toolkit]$ lshadoop-2.5.2.tar.gz hadoop-2.6.0.tar.gz jdk-7u71-linux-i586.gz scala-2.10.3.tgz spark-1.2.0-bin-hadoop2.3.tgz zookeeper-3.4.6.tar.gz
1、JDK安装及环境变量设置
[hadoop@SY-0130 ~]$ mkdir lab
#我将jdk7安装在lab目录
[hadoop@SY-0130 jdk1.7.0_71]$ pwd
/home/hadoop/lab/jdk1.7.0_71
#环境变量设置：
[hadoop@SY-0130 ~]$ vi .bash_profile
# User specific environment and startup programsexport JAVA_HOME=/home/hadoop/lab/jdk1.7.0_71PATH=$JAVA_HOME/bin:$PATH:$HOME/binexport PATHexport CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
#设置生效
[hadoop@SY-0130 ~]$ source .bash_profile
2、Hadoop2.6安装及设置
#解压toolkit 文件夹中的hadoop-2.6.0.tar.gz到. /home/hadoop/labc目录
[hadoop@SY-0130 hadoop-2.6.0]$ pwd
/home/hadoop/labc/hadoop-2.6.0
JDK，Hadoop基本安装完成，除了一些配置，现在可以将此虚拟机进行克隆。
该虚拟机命名:
130.ActiveNameNode
我将它克隆了3次，分别命名为：
131.StandbyNameNode
132.DataNode1
133..DataNode2
并且将克隆之后的3个虚拟机，重新生成了Mac地址，查看了他们分别使用的网卡，更改了对应的IP，dns，hostname , hosts，关闭防火墙。 具体操作按照上述说明操作即可。在网络配置这块我花了不少的时间。
软件配置：
至此，我拥有了四台安装好了JDK、Hadoop、及配置好对应的IP，能够访问Internet的Linux虚机。
在具体Hadoop HA 配置前，为了让节点之间通信方便，将4节点之间设置SSH免密码登录。
1、SSH免密码登录
[hadoop@SY-0130 ~]$ ssh-keygen -t rsa #一直回车即可.
#查看生成公钥
[hadoop@SY-0130 .ssh]$ ls
id_rsa id_rsa.pub known_hosts
#远程复制id_rsa.pub到SY-0131, SY-0132, SY-0133 节点。
[hadoop@SY-0130 .ssh]$ scp id_rsa.pub
[hadoop@SY-0130 .ssh]$ scp id_rsa.pub
[hadoop@SY-0130 .ssh]$ scp id_rsa.pub
#注意：SY-130为ActiveName , 在此我只配置了SY-0130到其他节点的免密码登录，即只是单向，没有设置双向。
#完成上述配置后，测试SY-130免密码登录
#连接sy-0131[hadoop@SY-0130 ~]$ ssh sy-0131Last login: Tue Jan 6 07:32:46 2015 from 192.168.249.1[hadoop@SY-0131 ~]$ #ctrl+d 可退出连接#连接sy-0132[hadoop@SY-0130 ~]$ ssh sy-0132Last login: Tue Jan 6 21:25:16 2015 from 192.168.249.1[hadoop@SY-0132 ~]$ #连接sy-0132[hadoop@SY-0130 ~]$ ssh sy-0133Last login: Tue Jan 6 21:25:18 2015 from 192.168.249.1[hadoop@SY-0133 ~]$ #测试成功
2、Hadoop设置
#进入hadoop安装目录
[hadoop@SY-0130 hadoop-2.6.0]$ pwd
/home/hadoop/labc/hadoop-2.6.0
#修改 hadoop-env.sh ，添加Java环境变量
[hadoop@SY-0130 hadoop-2.6.0]$ vi etc/hadoop/hadoop-env.sh
# The java implementation to use.export JAVA_HOME=/home/hadoop/lab/jdk1.7.0_71
#修改core-site.xml
fs.defaultFS hdfs://SY-0130:8020 The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. The uri's scheme determines the config property (fs.SCHEME.impl) naming the FileSystem implementation class. The uri's authority is used to determine the host, port, etc. for a filesystem.
#修改hdfs-site.xml
dfs.nameservices hadoop-test Comma-separated list of nameservices. dfs.ha.namenodes.hadoop-test nn1,nn2 The prefix for a given nameservice, contains a comma-separated list of namenodes for a given nameservice (eg EXAMPLENAMESERVICE). dfs.namenode.rpc-address.hadoop-test.nn1 SY-0130:8020 RPC address for nomenode1 of hadoop-test dfs.namenode.rpc-address.hadoop-test.nn2 SY-0131:8020 RPC address for nomenode2 of hadoop-test dfs.namenode.http-address.hadoop-test.nn1 SY-0130:50070 The address and the base port where the dfs namenode1 web ui will listen on. dfs.namenode.http-address.hadoop-test.nn2 SY-0131:50070 The address and the base port where the dfs namenode2 web ui will listen on. dfs.namenode.name.dir file:///home/hadoop/labc/hdfs/name Determines where on the local filesystem the DFS name node should store the name table(fsimage). If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy. dfs.namenode.shared.edits.dir qjournal://SY-0131:8485;SY-0132:8485;SY-0133:8485/hadoop-test A directory on shared storage between the multiple namenodes in an HA cluster. This directory will be written by the active and read by the standby in order to keep the namespaces synchronized. This directory does not need to be listed in dfs.namenode.edits.dir above. It should be left empty in a non-HA cluster. dfs.datanode.data.dir file:///home/hadoop/labc/hdfs/data Determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. Directories that do not exist are ignored. dfs.ha.automatic-failover.enabled false Whether automatic failover is enabled. See the HDFS High Availability documentation for details on automatic HA configuration. dfs.journalnode.edits.dir /home/hadoop/labc/hdfs/journal/
#修改mapred-site.xml
mapreduce.framework.name yarn The runtime framework for executing MapReduce jobs. Can be one of local, classic or yarn. mapreduce.jobhistory.address SY-0131:10020 MapReduce JobHistory Server IPC host:port mapreduce.jobhistory.webapp.address SY-0131:19888 MapReduce JobHistory Server Web UI host:port
#修改：yarn-site.xml
The hostname of the RM. yarn.resourcemanager.hostname SY-0130 The address of the applications manager interface in the RM. yarn.resourcemanager.address ${yarn.resourcemanager.hostname}:8032 The address of the scheduler interface. yarn.resourcemanager.scheduler.address ${yarn.resourcemanager.hostname}:8030 The http address of the RM web application. yarn.resourcemanager.webapp.address ${yarn.resourcemanager.hostname}:8088 The https adddress of the RM web application. yarn.resourcemanager.webapp.https.address ${yarn.resourcemanager.hostname}:8090 yarn.resourcemanager.resource-tracker.address ${yarn.resourcemanager.hostname}:8031 The address of the RM admin interface. yarn.resourcemanager.admin.address ${yarn.resourcemanager.hostname}:8033 The class to use as the resource scheduler. yarn.resourcemanager.scheduler.class org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler fair-scheduler conf location yarn.scheduler.fair.allocation.file ${yarn.home.dir}/etc/hadoop/fairscheduler.xml List of directories to store localized files in. An application's localized file directory will be found in: ${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid}. Individual containers' work directories, called container_${contid}, will be subdirectories of this. yarn.nodemanager.local-dirs /home/hadoop/labc/yarn/local Whether to enable log aggregation yarn.log-aggregation-enable true Where to aggregate logs to. yarn.nodemanager.remote-app-log-dir /tmp/logs Amount of physical memory, in MB, that can be allocated for containers. yarn.nodemanager.resource.memory-mb 30720 Number of CPU cores that can be allocated for containers. yarn.nodemanager.resource.cpu-vcores 12 the valid service name should only contain a-zA-Z0-9_ and can not start with numbers yarn.nodemanager.aux-services mapreduce_shuffle
#修改slaves
SY-0131SY-0132SY-0133
#在/home/hadoop/labc/hadoop-2.6.0/etc/hadoop下，增加fairscheduler.xml
102400 mb, 50 vcores 153600 mb, 100 vcores 200 300 1.0 root,yarn,search,hdfs 102400 mb, 30 vcores 153600 mb, 50 vcores 102400 mb, 30 vcores 153600 mb, 50 vcores
将etc/hadoop/目录中的这几个配置文件通过scp 命令远程拷贝到SY-0131，SY-0132，SY-0133节点对应目录。
3、Hadoop 启动(HDFS , YARN启动)
注意：所有操作均在Hadoop部署目录下进行。启动Hadoop集群：Step1 :在各个JournalNode节点上，输入以下命令启动journalnode服务：sbin/hadoop-daemon.sh start journalnodeStep2:在[nn1]上，对其进行格式化，并启动：bin/hdfs namenode -formatsbin/hadoop-daemon.sh start namenodeStep3:在[nn2]上，同步nn1的元数据信息：bin/hdfs namenode -bootstrapStandbyStep4:启动[nn2]：sbin/hadoop-daemon.sh start namenode经过以上四步操作，nn1和nn2均处理standby状态Step5:将[nn1]切换为Activebin/hdfs haadmin -transitionToActive nn1Step6:在[nn1]上，启动所有datanodesbin/hadoop-daemons.sh start datanode关闭Hadoop集群：在[nn1]上，输入以下命令sbin/stop-dfs.sh
web地址访问:
activenamenode
standbynamenode
datanodes info
4G内存，任性一次，开始Hadoop之旅！
PS:
一、Hdfs命令的区别：
1、如果Apache hadoop版本是0.x 或者1.x,
bin/hadoop hdfs fs -mkdir -p /in
bin/hadoop hdfs fs -put /home/du/input /in
2、如果Apache hadoop版本是2.x.
bin/hdfs dfs -mkdir -p /in
bin/hdfs dfs -put /home/du/input /in
二、 有时候DataNode启动不了有如下原因：
1、因重复格式化namenode时候，集群ID会更改，原先已有数据的datanode中记录的集群ID与NameNode不一致，该问题会导致datanode启动不了。
在第一次格式化dfs后，启动并使用了hadoop，后来又重新执行了格式化命令（hdfs namenode -format)，这时namenode的clusterID会重新生成，而datanode的clusterID 保持不变。
#对比clusterID :
namenode
[hadoop@SY-0131 current]$ pwd/home/hadoop/labc/hdfs/name/current[hadoop@SY-0131 current]$ cat VERSION#Tue Jan 06 23:39:38 PST 2015namespaceID=313333531clusterID=CID-c402aa07-4128-4cad-9d65-75afc5241fe1cTime=0storageType=NAME_NODEblockpoolID=BP-1463638609-192.168.249.130-1420523102441layoutVersion=-60
datanode
[hadoop@SY-0132 current]$ pwd/home/hadoop/labc/hdfs/data/current [hadoop@SY-0132 current]$ cat VERSION#Tue Jan 06 23:41:36 PST 2015storageID=DS-9475efc9-f890-4890-99e2-fdedaf1540c5clusterID=CID-c402aa07-4128-4cad-9d65-75afc5241fe1cTime=0datanodeUuid=d3f6a297-9b79-4e17-9e67-631732f94698storageType=DATA_NODElayoutVersion=-56
2、data目录的权限不够
本博客文章除特别声明，全部都是原创！
可以转载, 但必须以超链接形式标明文章原始出处和作者信息及版权声明。
尊重原创，转载请注明： 转载自JackyKen
本文链接地址: