hadoop安装---linux5（RHEL5）

Now,let us begin to install the hadoop on your linux 5 (RHEL5)

firstly

we should start ssh service inside your Linux

[[email protected] conf]# service sshd start

Starting sshd: [ OK ]

create a folder in usr

[[email protected] usr]# mkdir hadoop

then put the essential sources into the folder hadoop:

hadoop-1.0.0.tar.gz,you can download from http://hadoop.apache.org/

then change temportatry to usr/hadoop

[[email protected] hadoop]# tar –zxvf hadoop-1.0.0.tar.gz we use this command to tar it

then you will see another folder,

[[email protected] hadoop]# cd hadoop-1.0.0

[[email protected] hadoop-1.0.0]# cd conf

[[email protected] conf]# ls

capacity-scheduler.xml hadoop-policy.xml slaves

configuration.xsl hdfs-site.xml ssl-client.xml.example

core-site.xml log4j.properties ssl-server.xml.example

fair-scheduler.xml mapred-queue-acls.xml taskcontroller.cfg

hadoop-env.sh mapred-site.xml

hadoop-metrics2.properties masters

[[email protected] conf]# vi hadoop-env.sh

we will see and modify the file,pay attention to the bold sentences!!!

#Set Hadoop-specific environment variables here.

# The only required environment variable is JAVA_HOME. All others are

# optional. When running a distributed configuration it is best to

# set JAVA_HOME in this file, so that it is correctly defined on

# remote nodes.

# The java implementation to use. Required.

export JAVA_HOME=/usr/java/jdk1.6.0_20

# Extra Java CLASSPATH elements. Optional.

export HADOOP_CLASSPATH=/usr/hadoop/hadoop-1.0.0

export PATH=$PATH:/usr/hadoop/hadoop-1.0.0/

# The maximum amount of heap to use, in MB. Default is 1000.

# export HADOOP_HEAPSIZE=2000

# Extra Java runtime options. Empty by default.

# export HADOOP_OPTS=-server

# Command specific options appended to HADOOP_OPTS when specified

export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS"

[[email protected] conf]# source hadoop-env.sh

[[email protected] conf]#

[[email protected] conf]# cd ..

[[email protected] hadoop-1.0.0]# bin/hadoop

we can use this to check whether the file is modified correctly…

Usage: hadoop [--config confdir] COMMAND

where COMMAND is one of:

namenode -format format the DFS filesystem

secondarynamenode run the DFS secondary namenode

namenode run the DFS namenode

datanode run a DFS datanode

dfsadmin run a DFS admin client

mradmin run a Map-Reduce admin client

fsck run a DFS filesystem checking utility

fs run a generic filesystem user client

balancer run a cluster balancing utility

fetchdt fetch a delegation token from the NameNode

jobtracker run the MapReduce job Tracker node

pipes run a Pipes job

tasktracker run a MapReduce task Tracker node

historyserver run job history servers as a standalone daemon

job manipulate MapReduce jobs

queue get information regarding JobQueues

version print the version

jar <jar> run a jar file

distcp <srcurl> <desturl> copy file or directories recursively

archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive

classpath prints the class path needed to get the

Hadoop jar and the required libraries

daemonlog get/set the log level for each daemon

CLASSNAME run the class named CLASSNAME

Most commands print help when invoked w/o parameters.

[[email protected] hadoop-1.0.0]#

then

Setup passphraseless ssh

Now check that you can ssh to the localhost without a passphrase:

$ ssh localhost

If you cannot ssh to localhost without a passphrase, execute the following commands:

$ ssh-****** -t dsa -P '' -f ~/.ssh/id_dsa

$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

then modify the configuration

[[email protected] conf]# vi core-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<name>fs.default.name</name>

<value>hdfs://localhost:9000</value>

</property>

</configuration>

[[email protected] conf]# vi hdfs-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<name>dfs.replication</name>

</property>

</configuration>

[[email protected] conf]# vi mapred-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<name>mapred.job.tracker</name>

<value>localhost:9001</value>

</property>

</configuration>

create namenode

[[email protected] hadoop-1.0.0]# bin/hadoop namenode -format

12/04/08 12:54:48 INFO namenode.NameNode: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG: host = localhost.localdomain/127.0.0.1

STARTUP_MSG: args = [-format]

STARTUP_MSG: version = 1.0.0

STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1214675; compiled by 'hortonfo' on Thu Dec 15 16:36:35 UTC 2011

************************************************************/

12/04/08 12:54:49 INFO util.GSet: VM type = 32-bit

12/04/08 12:54:49 INFO util.GSet: 2% max memory = 19.33375 MB

12/04/08 12:54:49 INFO util.GSet: capacity = 2^22 = 4194304 entries

12/04/08 12:54:49 INFO util.GSet: recommended=4194304, actual=4194304

12/04/08 12:54:52 INFO namenode.FSNamesystem: fsOwner=root

12/04/08 12:54:53 INFO namenode.FSNamesystem: supergroup=supergroup

12/04/08 12:54:53 INFO namenode.FSNamesystem: isPermissionEnabled=true

12/04/08 12:54:53 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100

12/04/08 12:54:53 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)

12/04/08 12:54:53 INFO namenode.NameNode: Caching file names occuring more than 10 times

12/04/08 12:54:54 INFO common.Storage: Image file of size 110 saved in 0 seconds.

12/04/08 12:54:54 INFO common.Storage: Storage directory /tmp/hadoop-root/dfs/name has been successfully formatted.

12/04/08 12:54:54 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at localhost.localdomain/127.0.0.1

************************************************************/

[[email protected] hadoop-1.0.0]#

go to explorer to check the log

http://localhost:50070/

hadoop安装---linux5（RHEL5）

http://localhost:50030/

hadoop安装---linux5（RHEL5）

Copy the input files into the distributed filesystem:

[[email protected] hadoop-1.0.0]# bin/hadoop fs -put conf input

Run some of the examples provided:

[[email protected] hadoop-1.0.0]# bin/hadoop jar hadoop-examples-*.jar grep input output 'dfs[a-z.]+'

12/04/08 13:53:16 INFO mapred.FileInputFormat: Total input paths to process : 16

12/04/08 13:53:17 INFO mapred.JobClient: Running job: job_201204081256_0001

12/04/08 13:53:18 INFO mapred.JobClient: map 0% reduce 0%

12/04/08 13:53:58 INFO mapred.JobClient: map 6% reduce 0%

12/04/08 13:54:01 INFO mapred.JobClient: map 12% reduce 0%

12/04/08 13:54:17 INFO mapred.JobClient: map 25% reduce 0%

12/04/08 13:54:36 INFO mapred.JobClient: map 31% reduce 0%

12/04/08 13:54:41 INFO mapred.JobClient: map 37% reduce 8%

12/04/08 13:54:45 INFO mapred.JobClient: map 43% reduce 8%

12/04/08 13:54:49 INFO mapred.JobClient: map 50% reduce 12%

12/04/08 13:54:53 INFO mapred.JobClient: map 56% reduce 12%

12/04/08 13:54:56 INFO mapred.JobClient: map 62% reduce 12%

12/04/08 13:54:59 INFO mapred.JobClient: map 68% reduce 16%

12/04/08 13:55:03 INFO mapred.JobClient: map 75% reduce 16%

12/04/08 13:55:06 INFO mapred.JobClient: map 81% reduce 20%

12/04/08 13:55:09 INFO mapred.JobClient: map 87% reduce 20%

12/04/08 13:55:13 INFO mapred.JobClient: map 93% reduce 27%

12/04/08 13:55:16 INFO mapred.JobClient: map 100% reduce 27%

12/04/08 13:55:22 INFO mapred.JobClient: map 100% reduce 31%

12/04/08 13:55:28 INFO mapred.JobClient: map 100% reduce 100%

12/04/08 13:55:35 INFO mapred.JobClient: Job complete: job_201204081256_0001

12/04/08 13:55:35 INFO mapred.JobClient: Counters: 30

12/04/08 13:55:35 INFO mapred.JobClient: Job Counters

12/04/08 13:55:35 INFO mapred.JobClient: Launched reduce tasks=1

12/04/08 13:55:35 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=175469

12/04/08 13:55:35 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0

12/04/08 13:55:35 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0

12/04/08 13:55:35 INFO mapred.JobClient: Launched map tasks=16

12/04/08 13:55:35 INFO mapred.JobClient: Data-local map tasks=16

12/04/08 13:55:35 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=87301

12/04/08 13:55:35 INFO mapred.JobClient: File Input Format Counters

12/04/08 13:55:35 INFO mapred.JobClient: Bytes Read=26846

12/04/08 13:55:35 INFO mapred.JobClient: File Output Format Counters

12/04/08 13:55:35 INFO mapred.JobClient: Bytes Written=180

12/04/08 13:55:35 INFO mapred.JobClient: FileSystemCounters

12/04/08 13:55:35 INFO mapred.JobClient: FILE_BYTES_READ=82

12/04/08 13:55:35 INFO mapred.JobClient: HDFS_BYTES_READ=28568

12/04/08 13:55:35 INFO mapred.JobClient: FILE_BYTES_WRITTEN=367514

12/04/08 13:55:35 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=180

12/04/08 13:55:35 INFO mapred.JobClient: Map-Reduce Framework

12/04/08 13:55:35 INFO mapred.JobClient: Map output materialized bytes=172

12/04/08 13:55:35 INFO mapred.JobClient: Map input records=760

12/04/08 13:55:35 INFO mapred.JobClient: Reduce shuffle bytes=172

12/04/08 13:55:35 INFO mapred.JobClient: Spilled Records=6

12/04/08 13:55:35 INFO mapred.JobClient: Map output bytes=70

12/04/08 13:55:35 INFO mapred.JobClient: Total committed heap usage (bytes)=3252289536

12/04/08 13:55:35 INFO mapred.JobClient: CPU time spent (ms)=22930

12/04/08 13:55:35 INFO mapred.JobClient: Map input bytes=26846

12/04/08 13:55:35 INFO mapred.JobClient: SPLIT_RAW_BYTES=1722

12/04/08 13:55:35 INFO mapred.JobClient: Combine input records=3

12/04/08 13:55:35 INFO mapred.JobClient: Reduce input records=3

12/04/08 13:55:35 INFO mapred.JobClient: Reduce input groups=3

12/04/08 13:55:35 INFO mapred.JobClient: Combine output records=3

12/04/08 13:55:35 INFO mapred.JobClient: Physical memory (bytes) snapshot=2292494336

12/04/08 13:55:35 INFO mapred.JobClient: Reduce output records=3

12/04/08 13:55:35 INFO mapred.JobClient: Virtual memory (bytes) snapshot=6338478080

12/04/08 13:55:35 INFO mapred.JobClient: Map output records=3

12/04/08 13:55:36 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

12/04/08 13:55:37 INFO mapred.FileInputFormat: Total input paths to process : 1

12/04/08 13:55:38 INFO mapred.JobClient: Running job: job_201204081256_0002

12/04/08 13:55:39 INFO mapred.JobClient: map 0% reduce 0%

12/04/08 13:55:55 INFO mapred.JobClient: map 100% reduce 0%

12/04/08 13:56:11 INFO mapred.JobClient: map 100% reduce 100%

12/04/08 13:56:16 INFO mapred.JobClient: Job complete: job_201204081256_0002

12/04/08 13:56:16 INFO mapred.JobClient: Counters: 30

12/04/08 13:56:16 INFO mapred.JobClient: Job Counters

12/04/08 13:56:16 INFO mapred.JobClient: Launched reduce tasks=1

12/04/08 13:56:16 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=13535

12/04/08 13:56:16 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0

12/04/08 13:56:16 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0

12/04/08 13:56:16 INFO mapred.JobClient: Launched map tasks=1

12/04/08 13:56:16 INFO mapred.JobClient: Data-local map tasks=1

12/04/08 13:56:16 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=14047

12/04/08 13:56:16 INFO mapred.JobClient: File Input Format Counters

12/04/08 13:56:16 INFO mapred.JobClient: Bytes Read=180

12/04/08 13:56:16 INFO mapred.JobClient: File Output Format Counters

12/04/08 13:56:16 INFO mapred.JobClient: Bytes Written=52

12/04/08 13:56:16 INFO mapred.JobClient: FileSystemCounters

12/04/08 13:56:16 INFO mapred.JobClient: FILE_BYTES_READ=82

12/04/08 13:56:16 INFO mapred.JobClient: HDFS_BYTES_READ=295

12/04/08 13:56:16 INFO mapred.JobClient: FILE_BYTES_WRITTEN=42409

12/04/08 13:56:16 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=52

12/04/08 13:56:16 INFO mapred.JobClient: Map-Reduce Framework

12/04/08 13:56:16 INFO mapred.JobClient: Map output materialized bytes=82

12/04/08 13:56:16 INFO mapred.JobClient: Map input records=3

12/04/08 13:56:16 INFO mapred.JobClient: Reduce shuffle bytes=82

12/04/08 13:56:16 INFO mapred.JobClient: Spilled Records=6

12/04/08 13:56:16 INFO mapred.JobClient: Map output bytes=70

12/04/08 13:56:16 INFO mapred.JobClient: Total committed heap usage (bytes)=210763776

12/04/08 13:56:16 INFO mapred.JobClient: CPU time spent (ms)=2270

12/04/08 13:56:16 INFO mapred.JobClient: Map input bytes=94

12/04/08 13:56:16 INFO mapred.JobClient: SPLIT_RAW_BYTES=115

12/04/08 13:56:16 INFO mapred.JobClient: Combine input records=0

12/04/08 13:56:16 INFO mapred.JobClient: Reduce input records=3

12/04/08 13:56:16 INFO mapred.JobClient: Reduce input groups=1

12/04/08 13:56:16 INFO mapred.JobClient: Combine output records=0

12/04/08 13:56:16 INFO mapred.JobClient: Physical memory (bytes) snapshot=179437568

12/04/08 13:56:16 INFO mapred.JobClient: Reduce output records=3

12/04/08 13:56:16 INFO mapred.JobClient: Virtual memory (bytes) snapshot=749031424

12/04/08 13:56:16 INFO mapred.JobClient: Map output records=3

[[email protected] hadoop-1.0.0]#

Examine the output files:

Copy the output files from the distributed filesystem to the local filesytem and examine them:

[[email protected] hadoop-1.0.0]# bin/hadoop fs -get output output

[[email protected] hadoop-1.0.0]# cat output/*

cat: output/_logs: Is a directory

1 dfs.replication

1 dfs.server.namenode.

1 dfsadmin

[[email protected] hadoop-1.0.0]#

The pseudo system is installed succefully

hadoop安装---linux5（RHEL5）

相关推荐