Parallel Workers Running in Linux Containers

Introduction

This vignette shows how to set up parallel workers running in Linux containers, e.g. Docker (https://www.docker.com/), Apptainer (https://apptainer.org/), and udocker (https://indigo-dc.github.io/udocker/).

Examples

Example: Two parallel workers running in Docker

This example sets up two parallel workers running Docker image ‘rocker/r-parallel’ (https://hub.docker.com/r/rocker/r-parallel).

library(parallelly)
cl <- makeClusterPSOCK(
  rep("localhost", times = 2L),
  ## Launch Rscript inside Linux container via Docker
  rscript = c(
    "docker", "run", "--net=host", "rocker/r-parallel",
    "Rscript"
  ),
  ## IMPORTANT: Because Docker runs inside a virtual machine (VM) on macOS
  ## and MS Windows (not Linux), when the R worker tries to connect back to
  ## the default 'localhost' it will fail, because the main R session is
  ## not running in the VM, but outside on the host.  To reach the host on
  ## macOS and MS Windows, make sure to use master = "host.docker.internal"
  master = if (.Platform$OS.type == "unix") NULL else "host.docker.internal",
)
print(cl)
#> Socket cluster with 2 nodes on host 'localhost' (R version 4.3.3
#> (2024-02-29), platform x86_64-pc-linux-gnu)

Example: Two parallel workers running in Apptainer

This example shows how to set up two parallel workers running Docker image ‘rocker/r-parallel’ (https://hub.docker.com/r/rocker/r-parallel) via Apptainer (<https://apptainer.org/).

library(parallelly)
cl <- makeClusterPSOCK(
  rep("localhost", times = 2L),
  ## Launch Rscript inside Linux container via Apptainer
  rscript = c(
    "apptainer", "exec", "docker://rocker/r-parallel",
    "Rscript"
  )
)
print(cl)
#> Socket cluster with 2 nodes on host 'localhost' (R version 3.6.1
#> (2019-07-05), platform x86_64-pc-linux-gnu)

Example: Two parallel workers running in udocker

This example shows how to set up two parallel workers running Docker image ‘rocker/r-parallel’ (https://hub.docker.com/r/rocker/r-parallel) via udocker (https://indigo-dc.github.io/udocker/).

library(parallelly)
cl <- makeClusterPSOCK(
  rep("localhost", times = 2L),
  ## Launch Rscript inside Linux container via Docker
  rscript = c(
    "udocker", "--quiet", "run", "rocker/r-parallel",
    "Rscript"
  )
)
print(cl)
#> Socket cluster with 2 nodes on host 'localhost' (R version 3.6.1
#> (2019-07-05), platform x86_64-pc-linux-gnu)