Adventures in micro-cluster land

The past few months I had been busy tinkering with a micro-cluster during weekends to have a more in-depth peek at the Hadoop 2 eco-system. Six Odroid U3s stitched together by an el-cheapo DLink DGS-1008A gigabit switch. This is by far much more better than spinning Amazon EC2 instances or the two node Hadoop 1 cluster I tried a few years back.

One thing that keeps bugging me however is the intermittent connectivity am having with the cluster. A bit odd am getting a far more stable connection doing an ssh from the laptop into the cluster, than from one of the nodes in the cluster trying to ping or ssh another node.

This makes the master namenode unable to start properly and causes intermittent timeouts whenever it tries to start-cycle the host listings on the slaves config. And toying around ethtool with half/full duplex settings doesn't help either -- out of despair I tried even the 10 base half duplex but it still didn't work. The ODroid are using an smsc95xx USB 2.0 module and is purported to work in 100 base full duplex mode.

Am suspecting its either the cables or the network switch. Also looking at getting a powered USB charging hub to make setup a little bit more tidy. Alright, back to some other stuff for now.