Open-source containers move toward high-performance computing

Open-source containers move toward high-performance computing

Open-source containers are moving in a direction that many of us never anticipated.

Long recognized as providing an effective way to package applications with all of their required components, some are also tackling one of the most challenging areas in the compute world today — high-performance computing (HPC). And while containers can bring a new level of efficiency to the world of HPC, they’re also presenting new ways of working for enterprise IT organizations that are running HPC-like jobs.

How containers work

Containers offer many advantages to organizations seeking to distribute applications. By incorporating an application’s many dependencies (libraries, etc.) into self-sustainable images, they avoid a lot of installation problems. The differences in OS distributions have no impact, so separate versions of applications don’t have to be prepared and maintained, thus making developers’ work considerably easier.

The challenge of HPC

Until quite recently, the high-performance market with its emphasis on big data and supercomputing, paid little attention to containers. This was largely because the tightly coupled technology model of supercomputing didn’t fit well into the loosely coupled microservices world that containers generally serve. There were security concerns, as well, since. For example, Docker applications often bestow root privileges on those running them — an issue that doesn’t work very well in the supercomputing world where security is exceedingly important.

A significant change came about when Singularity — a container system with a focus on high-performance computing — became available. Now provided by Sylabs, Singularity began as an open-source project at Lawrence Berkeley National Laboratory in 2015.

Singularity was born because there was a lot of interest in containers for compute, but the commonly used containers (Docker) at the time did not support compute-focused, HPC-type use cases. Scientists used containers and shared their work on Docker Hub, but because Docker was not supportable on HPC, Singularity was created as a response to user demand for a compute-focused technology.