Autoreloader not starting in development Docker

Support
1

Hi everyone !

I am working on a project using Girder, for which multiple custom plugins were developed.
I was tasked to dockerize this, girder and plugins included.
I handcrafted a dockerfile based on the girder/girder image. The main difference is that I have additional steps to install our custom plugins (pip installs). This container works great !

However, now, I want to create a dev container. This container should adapt its responses if the code of the plugins is modified. I installed the plugins with the ‘-e’ options, used a volume to add the plugins’ code in the container and created a new config file with server mode set to ‘development’.

In the logs I can see that the mode is now ‘development’, but when I modify the code of my plugins, I don’t see those changes reflected in the responses (I tried with a test route returning True or False). I noticed that when running the girder/girder image I get a line telling me Started monitor thread 'AutoReloader', and that I don’t get it in my container’s logs.

I guess this is linked to my problem, but I don’t really see why I don’t get this log.

Here is some code :

dev.dockerfile

FROM node:14-bullseye as girder

WORKDIR /girder

# See http://click.pocoo.org/5/python3/#python-3-surrogate-handling for more detail on
# why this is necessary.
ENV LC_ALL=C.UTF-8
ENV LANG=C.UTF-8

COPY ./girder/scripts/setup.sh ./setup.sh
COPY ./girder/scripts/install_python.sh ./install_python.sh
RUN bash ./setup.sh
RUN bash ./install_python.sh
RUN rm ./setup.sh ./install_python.sh

RUN girder build --dev

EXPOSE 8080

CMD ["girder", "serve"]

FROM girder AS girder-plugins

COPY ./girder/scripts/install_plugins_dev.sh ./install_plugins_dev.sh
RUN chmod u+x ./install_plugins_dev.sh
RUN ./install_plugins_dev.sh
RUN girder build --dev

ARG GIRDER_CFG_FILE=./girder/girder.dist.cfg
COPY ${GIRDER_CFG_FILE} /girder/girder.cfg
ENV GIRDER_CONFIG /girder/girder.cfg
ENV WATCH_USEPOLLING true

docker-compose.yml

girder-dev:
    build:
      context: .
      dockerfile: ./girder/dev.dockerfile
      args:
        - GIRDER_CFG_FILE=./girder/girder.dev.cfg
    volumes:
      - ./girder/plugins:/girder/plugins
    networks:
      - mongodb_network
    profiles:
      - dev

For simplicity’s sake I didn’t include some bash files. You can consider that install_plugins_dev.sh runs python3.8 -m pip install -e plugins/<PLUGIN_NAME> for each of our custom plugins.

2

I’m guessing that because the plugins are installed in editable mode BEFORE they are mounted into the docker container, that python has references to the container files, not the mounted files, and therefore changing the mounted files doesn’t trigger the watcher to restart. I’d try doing the pip install -e plugin AFTER container start; I’d expect pip to create an egg directory as part of this outside the container (whereas it is currently inside the container and probably too late to watch what is outside).

This is just a guess, though.

– David

3

Thanks for your quick reply !
I tried something like you said. I created the serve_dev.sh script that installs the plugins and launches the server and I call it in the CMD of my container, so the pip install is executed after the volumes are mounted. However, it doesn’t appear to have an impact.

I also tried to call girder serve & before installing my plugin, and I never saw it being seen by girder (I checked /api/v1 to see if my endpoints were mounted). Is it normal or can it be related to my problem ?

Thanks in advance

The code :

serve_dev.sh

#!/bin/bash

set -e
set -o pipefail

pip install -e /girder/plugins/auth_ohif >> build-log.txt
girder build --dev >> build-log.txt 2>&1
echo "Installed plugins"
girder serve

dev.dockerfile

[...]

FROM girder AS girder-plugins

ARG GIRDER_CFG_FILE=./girder/girder.dist.cfg
COPY ${GIRDER_CFG_FILE} /girder/girder.cfg
ENV GIRDER_CONFIG /girder/girder.cfg
ENV WATCH_USEPOLLING true

COPY ./girder/scripts/serve_dev.sh ./serve_dev.sh
RUN chmod u+x ./serve_dev.sh
CMD [ "./serve_dev.sh" ]
4

I created a simple repro here. I found a workaround to the AutoReloader not starting (using the --dev option). However, I am not satisfied with this. Why isn’t the configuration file setting enough ?
Moreover, I now have a problem when the server restarts (when I modify the plugin). The server gets stuck and never finishes restarting. Any idea why ?

Here are the logs of my Docker container :

girder-1  | [26/Feb/2024:10:17:37] ENGINE Restarting because /app/girder/plugin/girder_plugin/test_rest.py changed.
girder-1  | [26/Feb/2024:10:17:37] ENGINE Stopped thread 'Autoreloader'.
girder-1  | [26/Feb/2024:10:17:37] ENGINE Bus STOPPING
girder-1  | [26/Feb/2024:10:17:38] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('0.0.0.0', 8080)) shut down
girder-1  | [26/Feb/2024:10:17:38] ENGINE Bus STOPPED
girder-1  | [26/Feb/2024:10:17:38] ENGINE Bus EXITING
girder-1  | Stopped asynchronous event manager thread.
girder-1  | [26/Feb/2024:10:17:38] ENGINE Bus EXITED
girder-1  | [26/Feb/2024:10:17:38] ENGINE Waiting for child threads to terminate...
girder-1  | [26/Feb/2024:10:17:38] ENGINE Re-spawning /usr/bin/python3 /usr/local/bin/girder serve --dev