Tag: StarCluster

bookmark_borderSGE/OGS: Clean your logs

Posted on by Frank-Mich L'Heureux

At Datacratic, part of our infrastructure runs in the cloud. Our elastic cluster is managed by StarCluster and job dispatch is managed by Open Grid Scheduler (OGS), a fork of Sun Grid Engine (SGE).

While I was looking at StarCluster’s output to help a new user, I realized there was a lot of timeouts. I dug a bit and found out that the command qacct was too slow. Following the lead, I understood that the said command parses a text file each time it is executed.

After a few years of operations, our main cluster has dispatched over 5 000 000 jobs. The log file parsed by qacct was about 2Gb in size. Tada! A couple of search queries taught me that OGS, in its install directory (<path to ogs>/util/logchecker.sh), has a script, ready to be configured, to rotate its logs. I configured and launched it. The timeouts are gone.

Lesson learned: StarCluster/OGS operators, it is important to configure and schedule that script to run every now and then if you want to keep your operations stable.

bookmark_borderStarCluster: Multiple node instance type support

Posted on by Frank-Mich L'Heureux

Last week I released a new feature for the vanilla_improvements branch of StarCluster: multiple instance type support. It means that our cluster can now select the instance type to bid on depending on a configurable factor and the lowest spot market price for each type.

Want to see how it works? Head to the wiki. Want to know more about how I did it? Read further.

Continue reading “StarCluster: Multiple node instance type support”

bookmark_borderStarCluster: Streaming node addition refactoring and dupe alias fixing

Posted on by Frank-Mich L'Heureux

About a month ago I created the streaming node addition functionality within StarCluster. As the time went by, I fixed some of its issues and found it a bit messy and hard to understand so I decided to move it to a separate file. The new version is ready and battle tested.

Another feature that I found to not be working as expected is the handler for nodes having the same alias. I fixed it and made a clean commit easy to pull/cherry-pick. It’s only a matter of calling _recover_duplicate_aliases.

bookmark_borderStarCluster: Streaming node addition

Posted on by Frank-Mich L'Heureux

In the core version of StarCluster, when you add many nodes at once (command “addnode -n #”), StarCluster goes through three sequential checks[*] that all nodes need to fulfill in order to move forward and eventually start configuring the nodes within the cluster.

  1. Wait for the spot instance requests to propagate.
  2. Wait for all spot instance requests to become active.
  3. Wait for ssh on all those nodes to be active.

If you add a single node, that’s fine, but if you add 10, you lose some time as the first node might be ready a few minutes before the last node is. That is to say, you are wasting some computing time.

Continue reading “StarCluster: Streaming node addition”

bookmark_borderStarCluster 0.95

Posted on by Frank-Mich L'Heureux

English version will follow.

La version 0.95 de StarCluster, à laquelle j’ai contribué à travers mon emploi chez Datacratic et mon implication sur le canal IRC, est sortie hier. Consultez la liste des changements pour plus d’informations.

StarCluster version 0.95, which I contributed to through my work at Datacratic and my implication via the IRC channel, was released yesterday. See the list of changes for more information.

bookmark_borderStarCluster 0.94

Posted on by Frank-Mich L'Heureux

La version 0.94 de StarCluster est sortie tout récemment. Le 22 juillet dernier pour être exact.

Pour ceux qui voudraient être un peu plus «bleeding edge», je viens de compléter le merge de la plus récente version de la branche develop à l’intérieur de notre branche vanilla_improvements. À la fin du readme, vous trouverez les ajouts que j’ai faits à la version originale.

StarCluster version 0.94 was recently released. July 22nd to be exact.

For those who would like to use the bleeding edge version, I just completed merging the latest commit on branch develop into our vanilla_improvements branch. At the end of the readme, you will find the additions I made to the original version.