Running Hadoop 2.6.0 in Docker containers Janos Matyas 02 December 2014

Yesterday the Hadoop community has released the 2.6.0 version of Hadoop – the 4th major release of this year – and one which contains quite a few new and interesting features:

  • Rolling upgrades – the holly grail for enterprises to switch Hadoop versions
  • Long running services in YARN
  • Heterogeneous storage in HDFS

These were the most popular features, though beside these there were quite a few extremely important ones – at least for us and our our Periscope project. As you might be aware we are working on an SLA policy based autoscaling API for Apache YARN and we were closely following/been involved or contributed to these JIRA’s below:

  • YARN-2248 – CS changes for moving apps between queues
  • YARN-1051 – YARN Admission Control/Planner

These tasks/subtasks (beside a few others) are all coming with the new major release and opening up brand new opportunities to make Hadoop YARN a more dynamic environment. Considering these and the Apache Slider project it’s pretty clear to see that exciting times are coming.

We have combined all these above with Docker, our open source projects – Cloudbreak and Periscope – and we are leveraging these new innovations, so stay tuned and get the code or our Docker containers to start with.

In the meanwhile (as usuall) we have released our Hadoop 2.6.0 container to ease your quick start with Hadoop.

DIY – Build the image

In case you’d like to try directly from the Dockerfile you can build the image as:

1
docker build  -t sequenceiq/hadoop-docker:2.6.0 .

Pull the image

As it is also released as an official Docker image from Docker’s automated build repository – you can always pull or refer the image when launching containers.

1
docker pull sequenceiq/hadoop-docker:2.6.0

Start the container

In order to use the Docker image you have just built or pulled use:

1
docker run -i -t sequenceiq/hadoop-docker:2.6.0 /etc/bootstrap.sh -bash

Testing

You can run one of the stock examples:

1
2
3
4
5
6
cd $HADOOP_PREFIX
# run the mapreduce
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep input output 'dfs[a-z.]+'

# check the output
bin/hdfs dfs -cat output/*

Hadoop native libraries, build, Bintray, etc

The Hadoop build process is no easy task – requires lots of libraries and their right version, protobuf, etc and takes some time – we have simplified all these, made the build and released a 64b version of Hadoop nativelibs on our Bintray repo. Enjoy.

Should you have any questions let us know through our social channels as LinkedIn, Twitter or Facebook.

Comments

Recent Posts