rush

Package website: release | dev

rush is a package designed to solve large-scale problems asynchronously across a distributed network. Employing a database centric model, rush enables workers to communicate tasks and their results over a shared Redis database. Key features include low task overhead, efficient caching, and robust error handling. The package powers asynchronous optimization algorithms in the bbotk and mlr3tuning packages.

Features

• Database centric model for robust scalability.
• Efficient communication between workers usingRedis.
• Maintains low overhead, limiting delays to just a millisecond per task.
• Reduces read/write operations with a lightweight and efficient caching system.
• Offers centralized system features, such as task queues.
• Provides fast data transformation from Redis to data.table.
• Simplifies local worker setup with processx.
• Enables scaling to large remote worker networks via the mirai package.
• Automatically detects and recovers from worker failures for high reliability.
• Logs worker messages directly into the Redis database using lgr.
• Designed with minimal dependencies for lightweight integration.

Install

Install the latest release from CRAN.

install.packages("rush")

Install the development version from GitHub.

pak::pak("mlr-org/rush")

And install Redis.

Related Work

• The rrq package is a task queue system for R using Redis.
• The future package provides a simple and uniform way of evaluating R expressions asynchronously across a range of backends.
• batchtools is a package for the execution of long-running tasks on high-performance computing clusters.
• The mirai package evaluates an R expression asynchronously in a parallel process, locally or distributed over the network.