Scroll to the bottom if you want the recipe and skip story telling.

This is a story of software: You have a greenfield project. It’s beautiful and fast. Then you add features. For five years. Then you realize your cold start times went from ~1 second to ~5.5 seconds.

To be fair, a ton of things happened in between and I did realize that cold start time had gone up like crazy. Some mitigating steps were taken to “control” them. Although, my local tests had always shown that precompiling bytecode was the best way to improve cold start, I never managed to make it work into production. Despite the precompilation, it kept being slow.

Then comes the AI and spare time. And the conversation basically goes like

🧑‍🦱Me: Hey AI! I see improvements to cold start locally when I precompile my python bytecode. I don’t see it when it’s in production. I use the serverless framework. Can you tell what’s going on?

🤖AI Agent: Well duh! Serverless sets the date to 1970-01-01 on all files when packaging, which means the python interpreter rebuilds the bytecode every time it starts.

Found it. In lib/plugins/package/lib/zip-service.js:97-101:

zip.append(file.data, {
  name,
  mode,
  date: new Date(0), // necessary to get the same hash when zipping the same content
});

🧑‍🦱 –> 🤯 I WAS SO CLOSE! One detail close.

And as the code comment states it, the framework did that intentionally.

So started my learning of python bytecode modes and PEP 552. If you don’t know, there are 3 ways to generate bytecode which tie into their invalidation mode.

1. date based (TIMESTAMP): if .py timestamp >= .pyc timestamp, regenerate
2. hash based (CHECK_HASH): hash the .py file and put the hash in the .pyc file. If they don’t match, regenerate.
3. trust us, don’t look (UNCHECKED_HASH): if there is a .pyc file, blindly accept it.

Option 1 does not work for our context because serverless sets the date to 1970-01-01. Hash based works, but the interpreter will spend time rehashing all .py files to realize nothing needs to be regenerated at every cold start. “Trust us” is the fastest as it doesn’t do anything.

For context, I use the serverless-python-requirements plugin, which packages all the python dependencies into a zip file, along with the application itself. I determined that, using serverless-scriptable-plugin, I could generate the bytecode files. Then came a choice: Do that before it creates the zip (via before:package:createDeploymentArtifacts) or after (via after:package:createDeploymentArtifacts)

In both cases, all the 3rd-party packages can be precompiled with the UNCHECKED_HASH mode, since precompilation happens in the temporary .serverless directory. For our application code it’s a different story. If we use the “before” hook, it needs to compile “in place”, aka outside the .serverless directory, meaning that UNCHECKED_HASH is dangerous as updating our .py code files would not regenerate the .pyc files… a sure way to get confused. So CHECK_HASH seems better advised for this case as it achieves the objective with a small compromise. Small especially given that I suspect most applications to have way more bytecode files in their dependencies than in their applications directly.

The “after” hook is triggered after the final zip file is made. So one can unzip it, generate all bytecode with UNCHECKED_HASH, and rezip. So bytecode here is optimized at the expense of build time.

Ultimately, we chose “before” because the cold start gain of “after” was negligible, and given we use cheap build machines the extra one-minute build time was not worth it.

I did not keep track of all the numbers, but here are a few, all taken on the same function over enough cold starts to make it statistically significant.

Recipe

This recipe was tested with osls v3.76.1, v4.0.0 and v4.1.0. If you don’t know what osls is, it’s based on the very popular serverless framework‘s latest v3 release. I highly recommend it if you’re looking for a way to keep security updates while staying on the “v3” version. If you’re still on the mainline serverless, this recipe very likely works over that v3 as well, but I did not test it over v4 of the mainline.

1. Install serverless-scriptable-plugin and add it to your serverless.yml file under plugins.
2. In your serverless.yml file, under custom, add this (adjust to your paths):

  scriptHooks:
    before:package:createDeploymentArtifacts: bash precompile_bytecode.sh
      public.ecr.aws/sam/build-${self:provider.runtime}

3. The code for precompile_bytecode.sh (And yeah, my bash-foo is not that high, I used a coding agent.)

#!/usr/bin/env bash

# Robot Generated
# Precompiles Python bytecode in place, in the deps and app-source directories, before Serverless
# zips them up. Must run on `before:package:createDeploymentArtifacts`, after
# `serverless-python-requirements` has installed and slimmed its deps directory but before
# Serverless (and that same plugin's own after-hook, which injects the deps into the zip) touch
# either directory.
#
# Usage: precompile_bytecode.sh  <docker-image> <source-dir>
#
# <docker-image> must match the deployed Lambda runtime's Python build exactly, or the bytecode
# magic number won't match and Python would silently recompile from source at runtime instead of
# using the cache -- fails the build below rather than letting that go unnoticed.
set -u

docker_image="${1:?usage: $0 <docker-image> <source-dir>}"
source_dir="${2:?usage: $0 <docker-image> <source-dir>}"
deps_dir=.serverless/requirements

# Passed as real argv words to the container (mode/dir pairs), not interpolated into the `sh -c`
# script text below, so directory names can't be misparsed as shell syntax.
targets=()
present_targets=""
for spec in "unchecked-hash:$deps_dir" "checked-hash:$source_dir"; do
  mode="${spec%%:*}"
  dir="${spec#*:}"
  [ -d "$dir" ] || continue
  targets+=("$mode" "$dir")
  present_targets+="$dir:$mode "
done

if [ "${#targets[@]}" -eq 0 ]; then
  echo "precompile_bytecode: neither $deps_dir nor $source_dir found, nothing to do"
  exit 0
fi

start_time=$(date +%s)

if docker run --rm --platform linux/amd64 \
  --user "$(id -u):$(id -g)" \
  -v "$(pwd):/var/task" -w /var/task \
  "$docker_image" \
  sh -c '
    set -e
    while [ "$#" -ge 2 ]; do
      mode="$1"; dir="$2"; shift 2
      python3 -m compileall -q -j 0 --invalidation-mode "$mode" -- "$dir"
    done
  ' sh "${targets[@]}"
then
  echo "precompile_bytecode: added bytecode to $present_targets"
  failed=0
else
  echo "precompile_bytecode: ERROR - failed to precompile $present_targets" >&2
  echo "You may disable this script in serverless.yml to no longer precompile."
  failed=1
fi

end_time=$(date +%s)
echo "precompile_bytecode: completed in $((end_time - start_time))s"

exit "$failed"

Cover Picture: Gros plan sur des glaçons qui fondent sur une surface noire by Ray Suarez