2

I have access to a powerful computer at work so naturally I want to run a chess engine on it (stockfish for now and later leela). I am trying to use docker-compose to link a stockfish image and a jupyter notebook image; then use the python-chess library to interact with the uci version of the stockfish engine. I am having difficulty linking the images with docker-compose (I am new to docker). Here is what I have tried:

I set up my directory structure as below:

├── docker-compose.yml
├── jupyter
│   └── Dockerfile
└── stockfish
    └── Dockerfile

my Docker file for stockfish is from https://github.com/jessegersensonchess/stockfish-nnue-docker/blob/master/Dockerfile:

FROM ubuntu:20.04 as builder
RUN apt update && \
    apt upgrade -y && \
    apt-get install -y git make g++ wget && \
    git clone https://github.com/official-stockfish/Stockfish && \
    cd Stockfish/src && \
    CXXFLAGS='-march=native' make -j2 profile-build ARCH=x86-64-bmi2
WORKDIR /Stockfish/src
FROM ubuntu:20.04
RUN apt-get update && apt-get install -y libgomp1 wget && apt-get autoremove -y && apt-get autoclean
COPY --from=0 /Stockfish/src/stockfish /app/
COPY --from=0 /Stockfish/src/*.nnue /app/
WORKDIR /app
CMD ["./stockfish"]

My dockerfile for jupyter:

FROM jupyter/scipy-notebook
USER root
RUN ["bash","-c","source activate root && pip install python-chess"]
USER jovyan

from the stockfish directory I can build the stockfish image and confirm that the uci executable is in the WORKDIR:

sudo docker build -t stockfish .
sudo docker run -it stockfish bash
root@mycontainerid:/app# ls -1
nn-eba324f53044.nnue
stockfish    <-- the uci engine executable

from the jupyter directory I can build the jupyter image and confirm that I can import the python-chess library:

sudo docker build -t jupyter/scipy-notebook .
sudo docker run -d -P jupyter/scipy-notebook
in [1]: import chess.engine  <-- the library to connect to uci engines

Now I try to link the images using docker-compose. My yml file look like this:

version: '3'
services:
   this_jupyter:
      build: jupyter
      ports:
         - "8888:8888"
      volumes:
         - .:/home/joyvan
   this_stockfish:
      build: stockfish
      volumes:
         - .:/home/joyvan

Ive tried a few variations of the yml file without success:

sudo docker-compose ps
         Name                      Command              State            Ports
---------------------------------------------------------------------------------------
stockfish_this_jupyter_1     tini -g -- start-notebook.sh   Up       0.0.0.0:8888->8888/tcp
stockfish_this_stockfish_1   ./stockfish                    Exit 0

my question is (a) why is the stockfish container not running, and (b) where will the executable be located once I successfully link to the jupyter notebook container?

UPDATE when I start the containers with docker compose (sudo docker-compose up -d) I get the following log files,

log file for jupyter:

sudo docker-compose logs this_jupyter
Attaching to leela_this_jupyter_1
this_jupyter_1    | Executing the command: jupyter notebook
this_jupyter_1    | [I 23:33:10.836 NotebookApp] Writing notebook server cookie secret to /home/jovyan/.local/share/jupyter/runtime/notebook_cookie_secret
this_jupyter_1    | [I 23:33:11.782 NotebookApp] JupyterLab extension loaded from /opt/conda/lib/python3.8/site-packages/jupyterlab
this_jupyter_1    | [I 23:33:11.783 NotebookApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
this_jupyter_1    | [I 23:33:11.791 NotebookApp] Serving notebooks from local directory: /home/jovyan
this_jupyter_1    | [I 23:33:11.791 NotebookApp] Jupyter Notebook 6.1.4 is running at:
this_jupyter_1    | [I 23:33:11.791 NotebookApp] http://648f6d3c8eca:8888/?token=f2d986295231a8adf0a3b3263d1177512090e2309972ca2e
this_jupyter_1    | [I 23:33:11.791 NotebookApp]  or http://127.0.0.1:8888/?token=f2d986295231a8adf0a3b3263d1177512090e2309972ca2e
this_jupyter_1    | [I 23:33:11.791 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
this_jupyter_1    | [C 23:33:11.800 NotebookApp]
this_jupyter_1    |
this_jupyter_1    |     To access the notebook, open this file in a browser:
this_jupyter_1    |         file:///home/jovyan/.local/share/jupyter/runtime/nbserver-6-open.html
this_jupyter_1    |     Or copy and paste one of these URLs:
this_jupyter_1    |         http://648f6d3c8eca:8888/?token=f2d986295231a8adf0a3b3263d1177512090e2309972ca2e
this_jupyter_1    |      or http://127.0.0.1:8888/?token=f2d986295231a8adf0a3b3263d1177512090e2309972ca2e

and the log file for stockfish:

sudo docker-compose logs this_stockfish
Attaching to leela_this_stockfish_1
this_stockfish_1  | Stockfish 181020 by the Stockfish developers (see AUTHORS file)
4
  • Without a log it is difficult to answer your question, run the services with docker compose and then post the log you read running this command: docker-compose logs this_jupyter or docker-compose logs this_stockfish depending which service you want to read logs
    – g3k0
    Commented Oct 19, 2020 at 18:20
  • @Christian_g3k0 appreciate it. I added an update. The log files don't appear to be particularly informative. Docker takes some getting used to.
    – nak3c
    Commented Oct 19, 2020 at 23:42
  • I ran the docker compose locally on my machine and replicated your situation. The stockfish container exit with status code 0 it means there are no errors, I think there is something wrong with the stockfish dockerfile...it is not written to maintain the container running....
    – g3k0
    Commented Oct 20, 2020 at 16:07
  • @Christian_g3k0 much appreciated. I may play around with some other dockerfiles. Im still a bit puzzled that I can run the stockfish image by itselt (sudo docker run -it stockfish bash) and run the stockfish binary in the container's shell (still getting used to the jargon). Ill post updates if I figure anything out. Eventually want to get leela working as well; my work computer has 4x tesla v100's.
    – nak3c
    Commented Oct 20, 2020 at 23:26

1 Answer 1

2

not quite the solution I was looking for but everything is working with the following Dockerfile

FROM nvidia/cuda:10.0-cudnn7-devel

RUN apt-get update && apt-get dist-upgrade -y && apt clean all
RUN apt-get update && apt-get install -y curl wget supervisor git clang-6.0 ninja-build protobuf-compiler libprotobuf-dev python3-pip && apt-get clean all
RUN pip3 install meson
RUN pip3 install python-chess

RUN mkdir src
WORKDIR src/
COPY . .
RUN pip3 install jupyter
WORKDIR /src/notebooks
ENV TINI_VERSION v0.6.0
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /usr/bin/tini
RUN chmod +x /usr/bin/tini
ENTRYPOINT ["/usr/bin/tini", "--"]

RUN mkdir /lczero && \
    cd ~ && \
    git clone -b release/0.26 --recurse-submodules https://github.com/LeelaChessZero/lc0.git && \
    cd lc0 &&\
    ./build.sh

RUN cp /root/lc0/build/release/lc0 /lczero && \
    file /lczero/lc0 && \
    chmod +x /lczero/lc0

RUN cd /lczero && \
    wget https://github.com/LeelaChessZero/lczero-client/releases/download/v29/lc0-training-client-linux && \
    chmod +x lc0-training-client-linux && \
    wget https://storage.lczero.org/files/networks-contrib/J92-280

RUN git clone https://github.com/official-stockfish/Stockfish.git &&\
    cd Stockfish &&\
    cd src &&\
    make help &&\
    make net &&\
    make build ARCH=x86-64-modern

COPY supervisord.conf /etc/supervisor/conf.d/
COPY settings.json /lczero/settings.json
RUN service supervisor start

CMD ["jupyter", "notebook", "--port=8888", "--no-browser", "--ip=0.0.0.0", "--allow-root"]

# sudo docker exec -it $(sudo docker run --runtime=nvidia --name leela -it -d -p 8888:8888 leela:latest ) service supervisor start
6
  • What was your build command? Commented Apr 27, 2021 at 22:11
  • probably "sudo docker-compose -t leela ." I havnt worked on this project in a few months but managed to get leela and stockfish running in a jupyter notebook which was nice. Ill try to check and update this thread in the near future and please remind me if I forget.
    – nak3c
    Commented Apr 28, 2021 at 23:09
  • Upvoted already. I think this is an important idea, would be nice if you could supply all the surrounding documentation :) Commented Apr 30, 2021 at 18:28
  • 1
    agreed, I took a look at my scripts this afternoon. something broke with the python meson package but I should be able to fix it. Ill set up a public git repo and link here. may take a day or two.
    – nak3c
    Commented May 1, 2021 at 23:50
  • ok, so I created an initial commit here: github.com/nkinney06/LeelaAndStockfishInJupyter but a few things broke since I was working on this project so I had to simplify in a few spots. Some of the background jobs are not working like they used to. Nevertheless, this is a start and Ill try to update the project somewhat regularly. You may have trouble on your machine stemming from the supervisord.conf file. If so you need to regenerate with the createSuperuser.py script. Ill have to add some more documentation for portability in the near future.
    – nak3c
    Commented May 2, 2021 at 17:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.