StarCluster: Multiple node instance type support

Last week I released a new feature for the vanilla_improvements branch of StarCluster: multiple instance type support. It means that our cluster can now select the instance type to bid on depending on a configurable factor and the lowest spot market price for each type.

Want to see how it works? Head to the wiki. Want to know more about how I did it? Read further.

Continue reading

StarCluster: Streaming node addition

In the core version of StarCluster, when you add many nodes at once (command “addnode -n #”), StarCluster goes through three sequential checks[*] that all nodes need to fulfill in order to move forward and eventually start configuring the nodes within the cluster.

  1. Wait for the spot instance requests to propagate.
  2. Wait for all spot instance requests to become active.
  3. Wait for ssh on all those nodes to be active.

If you add a single node, that’s fine, but if you add 10, you lose some time as the first node might be ready a few minutes before the last node is. That is to say, you are wasting some computing time.

Continue reading