Google Summer of Code -2019
Inzamam Iqbal - University of Moratuwa, Sri Lanka
Project : Create a user interface for Ganga that allows for the execution of tasks inside user specified virtual machines.
Organization: CERN-HSF
Mentors: Mark Smith, Alexander Richards, Ulrik Egede
Project Description
Ganga is used to execute a user defined computational task on a distributed back-end. Through this project we let the users define the environment in which their task need to be executed. So the worker node will pull the user defined container and execute the task on it.
Ganga repository: https://github.com/ganga-devs/ganga/
Tasks Completed
Phase 1: Research and experiment on possible technologies and tools to use.
- Looked into different options to run docker containers in user space.
- Created a document comparing singularity and udocker, comparing all the modes of udocker using a research paper and official documentations. (link: https://docs.google.com/document/d/1oXvovc3oXWSOfNrGw_1OBOdzmvorz9PkYgDeFd8lNyM/edit?usp=sharing)
- Tested Singularity and uDocker in CentOS 6.5 with kernel 2.6.32-431.29.2.el6.x86_64 and found out that singularity can’t run docker images in user-space in CentOS6 and uDocker is also not reliable in that version.
- Tested singularity and udocker in CentOS 7 with kernel 3.10.0-229.20.1.el7.x86_64 (in Azure VM).
- Found out that conversion of docker image is done better in udocker (always worked without a problem) than in singularity.
- Proposed user interface for specifying virtualization: https://docs.google.com/document/d/1adTWIBBF33W4_rYRDuQWu2HGiw_2_QshiLDLr2Zq0R4/edit?usp=sharing
Phase 2: Implementation
I have made the Pull request with virtualization support for LocalBackend.
The PR contains:
- Virtualization.py file in Utility. It handles the following tasks:
- Check the availability and permission of Docker
- Check the availability and version of Singularity
- Check the availability and permission of uDocker
- Download and install uDocker
- Docker.py and Singularity.py files which extends IVirtualization.py.
- These files are responisble for creating Docker and Singularity Objects which specify the details like container image.
- Modified Localhost.py file
- Injects relevant scripts to LocalHostExec.template.py.
- Copy the Virtualization.py file and image file (if user has given it as a simg file) to the worker node
- Modified LocalHostExec.template.py
- To run the tasks within user defined container and propagate the results and error back.
Phase 3: Finalizing
- Wrote relevant unit tests
- Documentation
Both are part of above mentioned pull request
Tasks Remaining
- Extend the Virtualization support to other Backends which will be similar to LocalBackend.
- Support for handling authentication for private docker hub images.
Comments
Post a Comment