New StarCluster features

This was a big two weeks for StarCluster as the Datacratic’s version got 3 new features: config on master, multiple subnets support and the dns-suffix option.

Config On Master

This change adds a “–config-on-master” option flag to the start command. If enabled, the cluster configuration will be written to master:/etc/starcluster.

  • Pros
    • Allows to easily update the config by editing the file.
    • No more update config compressed/hashed data in metadata/tags and other “obscure” places.
  • Cons
    • No longer possible to start a stopped cluster via StarCluster. (This is technically fixable, but not planned at the moment.)

The core version of StarCluster doesn’t have any option to update the config, but our version does. Updating the config was challenging because part of it is stored in the security group tags and part of it is stored in the master instance user data which can only be updated when the instance is stopped. For us, stopping the instance to update the config was not an option so what I did is create a json object containing the difference between the primary configuration and the new configuration and stored it into… tags. It worked, but it was not very good because tags are limited to 255 chars and instances are limited to 10 tags. It means that if a new config was “too” different we had to split it into various tags and could potentially hit the 10 tags limit. There was a lot of details to care about. Having the configuration in a file is sooooo simpler. You want to update it? Update the file and restart your load balancer. No more need to maintain a “reload config” function and to apply a configuration diff read from tags whenever we start a cluster.

Overall, it’s easier to use and it removed a certain amount of code meaning less maintenance. Yay!

As for the con, we never shut down the cluster so it’s hardly affecting us.

Config On Master Commit

Multiple Subnets

It drops the “–subnet-id” option from the start command and replaces it with the “SUBNET_IDS” key in the cluster template from the config file.

The core version of StarCluster can only add instances in a single subnet. The issue with that arises when you use spot instances because it means that the price you have to pay is the one from the zone associated with that subnet. Having multiple subnets breaks that rule since now you can spawn spot instances in the cheapest zone associated to one of the configured subnets. Who doesn’t like cost reduction?

Multiple Subnets Commit

Dns Suffix

It adds the “–dns-suffix” option to the start command.

This is minor but was a must for our operation guys. It’s basically the same thing as the dns-prefix option… but suffix. What’s “better” though is that now if we name our cluster mycluster.datacratic.com we now have hostnames “master.mycluster.datacratic.com”, “node001.mycluster.datacratic.com” and so on which provides cleaner hostnames right out of StarCluster.

Dns Suffix Commit

Leave a Reply