AWS Lambda and Django With Sqlite
Running Django on AWS Lambda has become fairly mainstream and there are many tools to assist with this goal. There is however a pain point which is admittedly not a conventional use case but a possible use case none the less. Django supports a version of sqlite that is above the current version supported on the underlying operating systems of the AWS python lambda implementation. If you're interested lambda functions run on Amazon Linux 1 and 2 which are based on CentOS 6 and 7 respectively.
There are a few ways to get around this, all of them involving either building the source yourself or standing on the shoulders of other people who already have. Some people have created lambda layers to add the required libraries.
With lambda now supporting container images I thought an easy thing to do would be to precompile sqlite inside the image using the build process in the pysqlite3 repository. This repo formalises a build process for creating more up to date sqlite3
python modules. Using the magic of multi-stage builds we can build everything and only copy over the relevant output files. The final image won't even have the build tools which cuts down on image size. Here's an example:
ARG PYTHON_VERSION=3.9
FROM public.ecr.aws/lambda/python:${PYTHON_VERSION} as build
RUN yum groupinstall -y "Development Tools" && \
yum install -y tcl
WORKDIR /
RUN git clone --depth 1 --branch master https://github.com/coleifer/pysqlite3.git
RUN curl -sSL https://www.sqlite.org/src/tarball/sqlite.tar.gz?r=release \
--output sqlite.tar.gz
RUN tar xzf sqlite.tar.gz
WORKDIR /sqlite
RUN ./configure && make sqlite3.c
RUN cp /sqlite/sqlite3.[ch] /pysqlite3/
WORKDIR /pysqlite3
RUN python setup.py build_static build
FROM public.ecr.aws/lambda/python:${PYTHON_VERSION}
ARG PYTHON_VERSION
COPY --from=build /pysqlite3/build/lib.linux-*/pysqlite3/_sqlite3.*.so /var/lang/lib/python${PYTHON_VERSION}/lib-dynload/
You can see the source at github and pull the final product from dockerhub. I've set up monthly builds so hopefully things keep up to date for a while.