Category: IT

Information Technology

bookmark_borderPython Poetry Index Error – list index out of range

Posted on by Frank-Mich L'Heureux

You run poetry and get this (undescriptive) error message.

[IndexError]
list index out of range

So far I’ve identified two things that you need to check.

  1. Credentials: If you are installing packages from a private repository, make sure poetry credentials are right.
  2. Presence of all required files in local packages: If you are installing a package from local files, make sure all files mentioned in the packages section of its pyproject.toml file are present. (I mostly got this error while building on Docker when forgetting to add them all in the Dockerfile.

bookmark_borderpython poetry 1.0.0 private repo issue fix

Posted on by Frank-Mich L'Heureux

On December 12th 2019, poetry v1.0.0 was released. With it, came a bad surprise for me: My CI/CD jobs as well as my Docker image builds started failing.

After investigating, I’ve found out that the password key/value was now missing from the  .config/pypoetry/auth.toml file. Digging some more, I’ve found out that poetry relies on a library called keyring to manage passwords.

Here is what I did to fix the problem.

First, I’ve noticed that poetry falls back to the previous method if keyring returns RuntimeError when it is called. Nice. It turns out that keyring comes with a backend aptly named “fail” which does that whatever the call is. So, it’s only a matter of configuring it.

As the keyring documentation states it, run python -c "import keyring.util.platform_; print(keyring.util.platform_.config_root())" to find where to put the configuration file. Then, in that directory, create keyringrc.cfg and put the following content in it:

[backend]                                    
default-keyring=keyring.backends.fail.Keyring

That’s it. Now you can call poetry config http-basic.... the same way you used to and the password will be stored in auth.tomllike before.

bookmark_borderSpeeding a CI/CD pipeline over CircleCI

Posted on by Frank-Mich L'Heureux

This post is annectodic, but I figured it could still be helpful in some way, so here I go.

Classic story. At work, the CircleCI CI/CD pipeline of the project I work on, as time went by, became slower and slower. Recently, it reached a bit over forty minutes. I worked on it and brought it back under ten minutes. Here is what I did.

Continue reading “Speeding a CI/CD pipeline over CircleCI”

bookmark_borderPython code formatters comparison: Black, autopep8 and YAPF

Posted on by Frank-Mich L'Heureux

Following some discussions at work and the will of the team to adopt a python code formatter, I set out to explore some of them. No need to say, the contenders had to aim towards pep8 compliance. Here are my findings on three of them.
Continue reading “Python code formatters comparison: Black, autopep8 and YAPF”

bookmark_borderAWS marketplace and no longer supported instance types

Posted on by Frank-Mich L'Heureux

I’ve been trying and trying to launch a Neo4j instance on the marketplace without success. It always gave me the nice “success” message, but when I went to the EC2 console: nothing!

A nice green “success” message even though it doesn’t work.

I finally decided to do it the hard way: manually. First step: select an instance type. I immediately try to select the same low cost instance I had picked in the marketplace (m3.medium) and to my surprise that type wasn’t there.

Instance type selection as per the Neo4j marketplace page.

Adding 1 + 1, I went back to the market place and tried with an instance type that still exists: m4.large… success!

So I don’t know who is to blame here, but here are my 2 questions to the internet:

  1. Why do we get the success message even though it doesn’t work?
  2. Why are unsupported instance types offered in the market place?

bookmark_borderPython logging to stackdriver

Posted on by Frank-Mich L'Heureux

I recently deployed a python application in google app engine / container engine. When I went to check the logs, everything was logged at the “ERROR” level even though my application uses python logging properly. As far as I know there are 2 ways to fix that:

  1. Use the stackdriver client, which requires an additional dependency and somewhat binds your program to google app engine.
  2. Format the logs in a way that stackdriver can parse them, which is easily configurable.

Continue reading “Python logging to stackdriver”

bookmark_borderRecipe: Testing multiple python versions with pyenv and tox

Posted on by Frank-Mich L'Heureux

If you develop a ton of python applications and you need to test under a lot of different versions, and by a lot I mean overlapping major/minor versions (like 3.5.3 and 3.5.4), then a good option is to use pyenv. Along with tox, you can easily test your application against various major/minor versions.

Here is how to do it.
Continue reading “Recipe: Testing multiple python versions with pyenv and tox”

bookmark_borderRecipe: Google App Engine, Cloud SQL and sqlalchemy

Posted on by Frank-Mich L'Heureux

Here is a recipe on how I made those thing work together both on a linux development environment and in production. The important thing to remember is: just like recipes for brownies, there are other recipes to achieve the same thing.

The following steps assume you have a Google cloud account with the proper permissions.

Continue reading “Recipe: Google App Engine, Cloud SQL and sqlalchemy”

bookmark_borderFixing Windows 10 sound issue/noise

Posted on by Frank-Mich L'Heureux

For a few months I have been struggling with small sound issues. It’s hard to put words to describe it, but it was like noise/static here and there, more often than not while playing games and rendering 3D.

I tried reinstalling audio drivers but that didn’t fix the issue. Finally, I found a post (to which I lost the link) with the solution.

In Windows 10 power options, I changed the configuration from “balanced” to “performance”.
Win10 customize power plan

bookmark_borderSGE/OGS: Clean your logs

Posted on by Frank-Mich L'Heureux

At Datacratic, part of our infrastructure runs in the cloud. Our elastic cluster is managed by StarCluster and job dispatch is managed by Open Grid Scheduler (OGS), a fork of Sun Grid Engine (SGE).

While I was looking at StarCluster’s output to help a new user, I realized there was a lot of timeouts. I dug a bit and found out that the command qacct was too slow. Following the lead, I understood that the said command parses a text file each time it is executed.

After a few years of operations, our main cluster has dispatched over 5 000 000 jobs. The log file parsed by qacct was about 2Gb in size. Tada! A couple of search queries taught me that OGS, in its install directory (<path to ogs>/util/logchecker.sh), has a script, ready to be configured, to rotate its logs. I configured and launched it. The timeouts are gone.

Lesson learned: StarCluster/OGS operators, it is important to configure and schedule that script to run every now and then if you want to keep your operations stable.