Hadoop

From Cncz
Jump to navigation Jump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

Apache Hadoop links

Setup terminal rooms

An Ubuntu package for hadoop (downloaded from ftp.nluug.nl) has been added to the science ubuntu repository.

hadoop_1.1.1-1_x86_64.deb

Local users

uid: 201 for hdfs
uid: 202 for mapred
gid:  49 for hadoop

In /etc/hadoop/hadoop-env.sh, the HADOOP_CLIENT_OPTS environment variable has been changed from -Xmx128m to -Xmx1024m.

Stand-alone test

With this setup, we could successfully run the example job:

$ cd /scratch/
$ mkdir input 
$ cp /usr/share/hadoop/templates/conf/*.xml input # heeft niks met configuratie te maken, dit is het genereren van input data
$ hadoop jar /usr/share/hadoop/hadoop-examples-*.jar grep input output 'dfs[a-z.]+' 
$ cat output/*