Open-source containers move toward high-performance computing

Open-source containers are moving in a direction that many of us never anticipated.

Long recognized as providing an effective way to package applications with all of their required components, some are also tackling one of the most challenging areas in the compute world today — high-performance computing (HPC). And while containers can bring a new level of efficiency to the world of HPC, they’re also presenting new ways of working for enterprise IT organizations that are running HPC-like jobs.

How containers work

Containers offer many advantages to organizations seeking to distribute applications. By incorporating an application’s many dependencies (libraries, etc.) into self-sustainable images, they avoid a lot of installation problems. The differences in OS distributions have no impact, so separate versions of applications don’t have to be prepared and maintained, thus making developers’ work considerably easier.

The challenge of HPC

Until quite recently, the high-performance market with its emphasis on big data and supercomputing, paid little attention to containers. This was largely because the tightly coupled technology model of supercomputing didn’t fit well into the loosely coupled microservices world that containers generally serve. There were security concerns, as well, since. For example, Docker applications often bestow root privileges on those running them — an issue that doesn’t work very well in the supercomputing world where security is exceedingly important.

A significant change came about when Singularity — a container system with a focus on high-performance computing — became available. Now provided by Sylabs, Singularity began as an open-source project at Lawrence Berkeley National Laboratory in 2015.

Singularity was born because there was a lot of interest in containers for compute, but the commonly used containers (Docker) at the time did not support compute-focused, HPC-type use cases. Scientists used containers and shared their work on Docker Hub, but because Docker was not supportable on HPC, Singularity was created as a response to user demand for a compute-focused technology.

Red Hat acquires CoreOS for $250 million in Kubernetes expansion

Red Hat, a company best known for its enterprise Linux products, has been making a big play for Kubernetes and containerization in recent years with its OpenShift Kubernetes product. Today the company decided to expand on that by acquiring CoreOS, a container management startup, for $250 million.

The company’s core products include CoreOS, a Linux distribution and Tectonic, a container management solution based on the open source Kubernetes container orchestration platform, originally developed by Google. (For more information on containers, see this article.)

CoreOs and Red Hat have been among the top contributors to Kubernetes, along with Google, FathomDB, ZTE Corporation, Huawei, IBM, Microsoft, Fujitsu and Mirantis.

Perhaps by working so closely on Kubernetes, CoreOS and Red Hat formed a bond, and it eventually made sense for them to come together and share customers and brain power. The companies also had competing Linux distros with CoreOS and Red Hat Atomic concentrating on containers, and perhaps the two can find some common developer ground by combining the two.

If the next generation of software is going to be in a hybrid cloud world where part lives on prem in the data center and part in the public cloud, having a cloud-native fabric to deliver applications in a single way is going to be critical. Red Hat’s president of products and technologies, Paul Cormier said that the combined companies are providing a powerful way to span environments.

“The next era of technology is being driven by container-based applications that span multi- and hybrid cloud environments, including physical, virtual, private cloud and public cloud platforms. Kubernetes, containers and Linux are at the heart of this transformation, and like Red Hat, CoreOS has been a leader in both the upstream open source communities that are fueling these innovations and its work to bring enterprise-grade Kubernetes to customers,” Cormier said in a statement.

As CoreOS CEO Alex Polvi told me in an interview last year, “As a company we helped create the whole container category alongside Google, Docker and Red Hat. We helped create a whole new category of infrastructure,” he said.

His company was early to the game by developing an enterprise Kubernetes product, and he was able to capitalize on that. “We called Kubernetes super-duper early and helped enterprises like Ticketmaster and Starbucks adopt Kubernetes,” he said.

He explained that Tectonic included four main categories, including governance, monitoring tools, chargeback accounting and one-click upgrades.

Red Hat CEO Jim Whitehurst told us in an interview last year that his company also came early to containers and Kubernetes. He said the company recognized containers included an operating system kernel, which was usually Linux. One thing they understood was Linux, so they started delving into Kubernetes and containerization and built OpenShift.

CoreOS has raised $50 million since its inception in 2013. Investors include GV (formerly Google Ventures) and Kleiner Perkins, which appear to have gotten nice returns. The most recent round was a $28 million Series B in May 2016 led by GV. One interesting aside is that Google, which has been a big contributor to Kubernetes itself and whose venture arm helped finance CoreOS, was scooped by Red Hat in this deal.

The deal is expected to close this month, and given we only have one day left, chances are it’s done.

Read more:

Sylabs launches Singularity Pro, a container platform for high-performance computing

Sylabs, the commercial company behind the open source Singularity container engine, announced its first commercial product today, Singularity Pro.

Sylabs was launched in 2015 to create a container platform specifically designed for scientific and high performance computing use cases, two areas that founder and CEO Gregory Kurtzer, says were left behind in the containerization movement over the last several years. (For an explanation of containers, see this article.)

Docker emerged as the container of engine of choice for developers, but Kurtzer says the container solutions developed early on focused on microservices. He says there’s nothing inherently wrong with that, but it left out some types of computing that relied on processing jobs instead of services, specifically high performance computing.

Kurtzer, who didn’t exactly just fall off the open source turnip truck, had more than 20 years of experience as a high performance computing architect working at the US Department of Energy Lab, where he founded CentOS, an open source enterprise Linux project and Warewulf, which he says has become the most utilized stateless HPC cluster provisioner.

He decided to shift his attention to containers when founded Sylabs and launched the first open source version of Singularity in April, 2016. Even then, he had a vision of creating a commercial version of the product. He saw Singularity as a Docker for HPC environments, and would run his company in a similar fashion to Docker, leading with the open source project, then building a commercial business on top of it — just as Docker had done.

Kurtzer now wants to bring Singularity to the enterprise with a focus not just on the HPC commercial market, but other high performance computing workloads such as artificial intelligence, machine learning, deep learning and advanced analytics.

“These applications carry data-intensive workloads that demand HPC-like resources, and as more companies leverage data to support their businesses, the need to properly containerize and support those workflows has grown substantially,” Kurtzer wrote in a blog post announcing the enterprise product.

Even though Singularity is designed to handle different kinds of workloads, it still works with container orchestration tools, specifically Kubernetes and Mesos, and it is also compatible with Microsoft’s Azure Batch tool and other cloud tools.

Kurtzer indicated Sylabs currently has 12 employees, and is operating on an undisclosed amount of seed money. It was funded by RStor, a startup itself currently operating in stealth mode.

Read more: