Job Types

Supported Job Types

JuliaBatch

A Julia Batch Job.

Runs in a single allocation/container restricted to the specified amount of CPUs and memory. The Julia process started in the batch job is allowed to start other worker processes or threads, all within the specified limits. The upper bound of CPUs/memory is determined by the largest cluster node.

Scenarios:

Required parameters:

Optional parameters:

JuliaParBatch

A Julia Master-Slave Parallel Batch Job.

A designated master process and an array of worker processes, each of which runs in a separate allocation/container. All worker processes have a uniform resource allocation. The master process can have a different allocation.

The job as a whole can use all available CPU/memory in a cluster (within practical limitations imposed networking and such). The upper bound for each master/worker process is determined by the largest node in the cluster.

The master and worker processes are initialized for Julia parallel constructs. The number of workers can be checked and manipulated with the scale/scale! APIs. The number of worker processes actually running depends on the available cluster resources and may be different from the requested scale. The job is deemed running when the master process starts, and JuliaRun attempts to match number of worker processes to the requested scale continuously. The master process may check the number of actual workers with the nworkers API and wait for a minimum number of workers if needed by it.

Scenario:

Required parameters:

Optional parameters:

JuliaParBatchWorkers

A Julia Embarrassingly Parallel Batch Job.

An array of batch jobs that can be scaled as a single unit. Each unit in the job array can be likened to a JuliaBatch job. Since there is no interlinking between the units, this can be scaled faster and to much higher levels.

As a whole, the job can use all available CPU/memory in the cluster, but the upper bound for each worker process is determined by the largest node in the cluster.

Scenarios:

Required parameters:

Optional parameters:

Notebook

A Julia Interactive Notebook.

The notebook is accessible outside the cluster and can be protected by a password (or an external portal/proxy). The master container runs the Jupyter notebook process and the Julia kernels.

The Julia process running as the kernel is similar to a JuliaBatch. Workers can be provisioned (optionally) as:

Scenarios:

Required parameters:

Optional parameters:

PkgBuilder

Build / update a Julia package bundle.

It mounts the volume representing the package folder of the bundle in read-write mode, and launches a script to build them. It is similar to a JuilaBatch job.

Package bundles are a way of making Julia packages accessible to JuliaRun jobs. They are just folders with Julia packages, pre-compiled and along with all their dependencies. Package bundles can be built and tested separately and attached to multiple images, providing a way to dissociate their maintenance.

Attaching a package bundle to a JuliaRun job sets the appropriate environment variables (LOAD_PATH and such) so that Julia code can start using them seamlessly.

Required parameters:

Optional parameters:

Webserver

Webserver that can be used to serve API/UI/files for jobs.

Scenarios:

Required parameters:

Optional parameters:

MessageQ

Message Queue that can be used for communication between processes within or across jobs. Enables complex routing/broadcast of messages.

Scenarios:

Required parameters:

Optional parameters: