Apache Hadoop 2.5.0 on Docker Janos Matyas 18 August 2014

A few weeks ago we have released an Apache Hadoop 2.3 Docker image – in a very short time this become the most popular Hadoop image in the Docker registry.

Following on the success of our Hadoop 2.3.0, 2.4.0 and 2.4.1 Docker image, the feedbacks and requests we have received and aligning with the Hadoop release cycle, we have released an Apache Hadoop 2.5.0 Docker image – same as the previous version this is available as a trusted and automated build on the official Docker registry.

Please note that beside this Hadoop image, we have released and maintain a pseudo-distributed and fully distributed Hadoop Docker image provisioned with Apache Ambari. As they are provisioned with Ambari you have the option to change, add and remove Hadoop components using cluster blueprints.

Also we are happy to let you know that this release of Apache Hadoop contains a few of SequenceIQ’s open source contributions and fixes around YARN schedulers. We are working on an SLA enforcer for Hadoop – very soon to be open sourced – and part of that work we are contributing back to the community. Also there is a major contribution of ours coming in the next release of Hadoop – 2.6.0. Stay tuned and follow us on LinkedIn, Twitter or Facebook.

Build the image

In case you’d like to try directly from the Dockerfile you can build the image as:

1
docker build  -t sequenceiq/hadoop-docker .

Pull the image

As it is also released as an official Docker image from Docker’s automated build repository – you can always pull or refer the image when launching containers.

1
docker pull sequenceiq/hadoop-docker:2.5.0

Start a container

In order to use the Docker image you have just build or pulled use:

1
docker run -i -t sequenceiq/hadoop-docker:2.5.0 /etc/bootstrap.sh -bash

Testing

You can run one of the stock examples:

1
2
3
4
5
6
cd $HADOOP_PREFIX
# run the mapreduce
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar grep input output 'dfs[a-z.]+'

# check the output
bin/hdfs dfs -cat output/*

Hadoop native libraries, build, Bintray, etc

The Hadoop build process is no easy task – requires lots of libraries and their right version, protobuf, etc and takes some time – we have simplified all these, made the build and released a 64b version of Hadoop nativelibs on our Bintray repo. Enjoy.

Should you have any questions let us know through our social channels as LinkedIn, Twitter or Facebook.

Comments

Recent Posts