Management of Python-related tools and container images

Table of Content (ToC)

• Management of Python-related tools and container images
• Table of Content (ToC)
• Overview
  • References
• Projects this repository helps to manage
  • Light images to support Machine Learning (ML) in Python
  • Docker images to support development on C++ and Python stacks
    • General C++/Python development
    • Python Machine Learning (ML)
    • Python cloud images
  • Python tools for OpenTravelData (OPTD)
  • Quality Assurance (QA) for OpenTravelData (OPTD)
  • Python bindings for OpenTravelData (OPTD)
  • Python bindings for OpenTREP
  • Python installation
• Procedures
  • New Python stable version
    • Add the new Python version to the C++/Python Docker images
    • Update the Python environment of the Python Jupyter Docker image
    • Update the Python environment of the Python light Docker image
    • Non Docker projects to update
      • Machine Learning (ML) induction
      • Python tools for OpenTravelData (OPTD)
      • Quality Assurance (QA) for OpenTravelData (OPTD)
      • Python bindings for OpenTravelData (OPTD)
      • Python bindings for OpenTREP
  • Installation or update of a Python virtual environment

Table of contents generated with markdown-toc

Overview

References

• This GitHub repository - Docker Image Management

Projects this repository helps to manage

Light images to support Machine Learning (ML) in Python

• ML Helpers GitHub repository
• Docker land page for infrahelpers/python-light
  • Docker image page for infrahelpers/python-light:py39-bullseye
• Quay land page for artificialintelligence/python-light
• Distributions:
  • Python 3.12:
    • py312-bookworm
    • py312-bullseye
    • py312-alp318
  • Python 3.11:
    • py311-bookworm
    • py311-bullseye
    • py311-alp318
  • Python 3.10:
    • py310-bookworm
    • py310-bullseye
  • Python 3.9:
    • py39-bookworm
    • py39-bullseye
  • Python 3.8:
    • py38-bookworm
    • py38-bullseye
  • Native Python on latest Alpine:
  • alp319-py311
• Badges:

Images to support Data Processing Pipelines (DPP)

• GitHub repsitory - Images for Data Processing Pipelines (DPP)
  • GitHub - Python version matrix in GitHub Actions
• Docker land page for infrahelpers/dpp
• Image tags: jdk17-python3.12, jdk11-python3.12, jdk8-python3.12, jdk17-python3.11, jdk11-python3.11, jdk8-python3.11

Images to support development on C++ and Python stacks

General C++/Python development

• GitHub repsitory - Docker for C++ development
• Docker land page for infrahelpers/cpppython
  • Docker image page for infrahelpers/cpppython:ubuntu2204
• Quay land page for cpppythondevelopment/base
• Distributions: ubuntu2404, ubuntu2204, ubuntu2004, debian12, debian11, centos9, centos8, centos7, fedora40, fedora39
• Badges:

Python Machine Learning (ML)

• GitHub repsitory - Docker for Python (Jupyter, Pandas)
• Docker land page for infrahelpers/python-jupyter
  • Docker image page for infrahelpers/python-jupyter:ubuntu2004
• Quay land page for artificialintelligence/python-jupyter
• Distributions: ubuntu2404, ubuntu2204, ubuntu2004, debian12, debian11, centos9, centos8
• Badges:

Python cloud images

• GitHub repsitory - Python cloud images
• Docker land page for infrahelpers/cloud-python
  • Docker image page for infrahelpers/cloud-python:py39-bullseye

• Image tags: pyspark-emr6, pyspark-emr6-light, pyspark-emr-dbs, pyspark-emr-jdk11, py312-bookworm, py311-bookworm, py310-bookworm, py39-bookworm, py38-bookworm

• Badges:

Python tools for OpenTravelData (OPTD)

• OpenTravelData (OPTD) tools

Quality Assurance (QA) for OpenTravelData (OPTD)

• OpenTravelData (OPTD) - Quality Assurance (QA)

Python bindings for OpenTravelData (OPTD)

• OpenTravelData (OPTD) Data Wrapper - Python Bindings
• OpenTravelData (OPTD) on PyPi

Python bindings for OpenTREP

• OpenTREP - Python wrapper
• OpenTrepWrapper on PyPi

Python installation

• How-to install Python virtual environment with pyenv and pipenv

Procedures

New Python stable version

• Python releases

Add the new Python version to the C++/Python Docker images

• Git repository: https://github.com/cpp-projects-showcase/docker-images

• Add the commands to install the new Python version in the Dockerfile files of all the distriibutions. For instance, adding Python 3.12.0 to Ubuntu 22.04

  $ cat ubuntu2204/Dockerfile
  ...
  # Python 3.12.3
  RUN pyenv install 3.12.3 && \
    pyenv global 3.12.3 && \
    python -mpip install -U pip pipenv
  RUN pyenv global system || echo "No default system version of Python. Sticking to 3.12.3"
  ...
  

• Keeping at least one of the older versions of Python (3.10.13 and 3.11.8 here) gives the teams managing downstream Docker images the time to catch up.

Update the Python environment of the Python Jupyter Docker image

• Git repository: https://github.com/machine-learning-helpers/docker-python-jupyter

The ML Python Jupyter Docker images are built on top of the generic C++/Python images, described in the section above. The Python versions are therefore controlled by those Docker images.

The Python virtual environment specifications need however to be upgraded. Files to update:

• .python-version
• Pipfile

Update the Python environment of the Python light Docker image

• Git repository: https://github.com/machine-learning-helpers/docker-python-light

The default Python versions for light images (e.g., Alpine, Debian and Debian Slim), as maintained by Docker itself (on GitHub and on Docker Hub), are usually the latest Python stable versions, only a few days after they have been released. There is hence no need to upgrade the Python version on the light Docker images.

Non Docker projects to update

Machine Learning (ML) induction

Reference: ML-related Python induction project

Note that this Git repository is also a sub-module of the Python Jupyter Docker image project

There are two folders in that project where the Python setup has to be updated:

• Root folder
  • .python-version
  • Pipfile
• ml_induction_python folder
  • Pipfile
  • requirements.txt
    • Note that the requirements.txt file may be upgraded with:

      $ pipenv install
      pipenv requirements > requirements.txt
      git add requirements.txt
      git commit -m "Upgreded the Python dependencies"
      

Then, the ML Docker project (see above) should be updated to point to the new Python ML induction head.

• Clone the ML Docker project and initialize sub-modules (if needed):

  $ mkdir -p ~/dev/ml && git clone https://github.com/machine-learning-helpers/docker-python-jupyter ~/dev/ml/docker-python-jupyter
  $ cd ~/dev/ml/docker-python-jupyter
  $ git submodule init
  $ git submodule update --recursive --force
  

• Pull (with Git) the head of the master branch on the ML induction module:

  $ cd ~/dev/ml/docker-python-jupyter/notebook/induction
  $ git checkout master
  $ git pull
  

• Add, commit and push (with Git) the newly updated module:

  $ cd ~/dev/ml/docker-python-jupyter
  $ git add notebook/induction
  $ git commit -m "[Modules] Updated the Python ML induction module"
  $ git push
  

Python tools for OpenTravelData (OPTD)

References:

• OpenTravelData (OPTD) tools

Files to update:

• Pipfile

Quality Assurance (QA) for OpenTravelData (OPTD)

References:

• OpenTravelData (OPTD) - Quality Assurance (QA)

Files to update:

• Pipfile
• requirements.txt
  • Copy that requirements.txt file as ci-scripts/requirements.txt in OpenTravelData (OPTD)

Python bindings for OpenTravelData (OPTD)

References:

• Python OpenTravelData (OPTD) bindings
• OpenTravelData on PyPi

Files to update:

• Pipfile
• Pipfile.lock
• (potentially) setup.py
• (potentially) pyproject.toml
• requirements.txt
• requirements-dev.txt
• (potentially) .travis.yml
• (potentially) tox.ini

Publish the component on PyPi:

user@laptop$ cd ~/dev/geo/python-opentraveldata
user@laptop$ rm -rf dist
user@laptop$ python -mbuild
user@laptop$ PYPIURL="https://test.pypi.org"
user@laptop$ twine upload -u __token__ --repository-url ${PYPIURL}/legacy/ dist/*
user@laptop$ PYPIURL="https://pypi.org"
user@laptop$ #keyring set ${PYPIURL}/ __token__
user@laptop$ twine upload -u __token__ --non-interactive dist/*

The result should available on PyPi, e.g., https://pypi.org/project/opentraveldata/0.0.9.post2/

Python bindings for OpenTREP

References:

• (Python) OpenTREP Wrapper
• OpenTrepWrapper on PyPi

Files to update:

• Pipfile

Publish the component on PyPi:

user@laptop$ rm -rf dist && mkdir dist
user@laptop$ python setup.py sdist bdist_wheel
user@laptop$ PYPIURL="https://test.pypi.org"
user@laptop$ twine upload -u __token__ --repository-url ${PYPIURL}/legacy/ dist/*
user@laptop$ PYPIURL="https://pypi.org"
user@laptop$ #keyring set ${PYPIURL}/ __token__
user@laptop$ twine upload -u __token__ --non-interactive dist/*

The resulting artifact should be available on PyPi, e.g.: https://pypi.org/project/OpenTrepWrapper/0.7.7.post3/

Installation or update of a Python virtual environment

• In a folder governed by Pipfile:
  • To install the dedicated Python virtual environment:

    $ pipenv install; pipenv install --dev
    

  • To update the dedicated Python virtual environment:

    $ pipenv update
    

• On MacOS, with the psycopg2 module, there may be some trouble at installation time related to an issue with SSL libraries. It can usually be solved with:

  $ LDFLAGS="-I/usr/local/opt/openssl/include" CPPFLAGS="-L/usr/local/opt/openssl/lib" pipenv install psycopg2
  Installing psycopg2
  Adding psycopg2 to Pipfile's [packages]
  ✔ Installation Succeeded 
  $ pipenv install; pipenv install --dev