Girder worker (docker) performance findings


#1

Here are my findings concerning performance of girder workers with docker. Below is a table with data collected while running this example that uses dcm files contained in an item to produce some outputs.

Number of .dcm files Memory used by worker (Mo) Time of execution (s)
2 70 16
10 80 18
100 300 36.7
Special case: 100+100 (2 workers) 599 50 (concurrent run)
Special case: Hello world <20 5

Note 1: The timings using two workers are not reliable. Indeed I used the same item for both workers so one’s results may have interfered with the other’s.
Note 2: Running the hello world image alone takes 2.5s on my computer.

Conclusions

Memory

  • the memory cost of a worker is negligible.
  • the memory cost of a worker + using the image of the example is about 70 Mo. So instantiating docker images costs some memory, depending on the image.
  • the memory costs of workers running the same image are independent.

Time

  • It is more costly to run a docker image from a worker than alone.
  • The “starting” time (before actually running the code) depends on the image.

Question

What are your data (any metric) concerning the overhead induced by using the docker_run task vs executing the same script in a non-docker task ?