Yesterday the Hadoop community has released the
2.6.0 version of Hadoop – the 4th major release of this year – and one which contains quite a few new and interesting features:
- Rolling upgrades – the holly grail for enterprises to switch Hadoop versions
- Long running services in YARN
- Heterogeneous storage in HDFS
These were the most popular features, though beside these there were quite a few extremely important ones – at least for us and our our Periscope project. As you might be aware we are working on an SLA policy based autoscaling API for Apache YARN and we were closely following/been involved or contributed to these JIRA’s below:
These tasks/subtasks (beside a few others) are all coming with the new major release and opening up brand new opportunities to make Hadoop YARN a more
dynamic environment. Considering these and the Apache Slider project it’s pretty clear to see that exciting times are coming.
We have combined all these above with Docker, our open source projects – Cloudbreak and Periscope – and we are leveraging these new innovations, so stay tuned and get the code or our Docker containers to start with.
In the meanwhile (as usuall) we have released our Hadoop 2.6.0 container to ease your quick start with Hadoop.
DIY – Build the image
In case you’d like to try directly from the Dockerfile you can build the image as:
Pull the image
As it is also released as an official Docker image from Docker’s automated build repository – you can always pull or refer the image when launching containers.
Start the container
In order to use the Docker image you have just built or pulled use:
You can run one of the stock examples:
1 2 3 4 5 6
Hadoop native libraries, build, Bintray, etc
The Hadoop build process is no easy task – requires lots of libraries and their right version, protobuf, etc and takes some time – we have simplified all these, made the build and released a 64b version of Hadoop
nativelibs on our Bintray repo. Enjoy.