Last week I released a new feature for the vanilla_improvements branch of StarCluster: multiple instance type support. It means that our cluster can now select the instance type to bid on depending on a configurable factor and the lowest spot market price for each type.
Want to see how it works? Head to the wiki. Want to know more about how I did it? Read further.
In the core version of StarCluster, when you add many nodes at once (command “addnode -n #”), StarCluster goes through three sequential checks[*] that all nodes need to fulfill in order to move forward and eventually start configuring the nodes within the cluster.
- Wait for the spot instance requests to propagate.
- Wait for all spot instance requests to become active.
- Wait for ssh on all those nodes to be active.
If you add a single node, that’s fine, but if you add 10, you lose some time as the first node might be ready a few minutes before the last node is. That is to say, you are wasting some computing time.
This was a big two weeks for StarCluster as the Datacratic’s version got 3 new features: config on master, multiple subnets support and the dns-suffix option.