Distributed Computing

Multiple processes, possibly on multiple machines, with separate memory.

Clusters: addprocs(["host1", "host2"]) via SSH; ClusterManagers.jl (Slurm, PBS, Kubernetes); MPI.jl for HPC message passing; Dagger.jl for task graphs; DistributedArrays.jl.

using Distributed
addprocs(4)                          # local worker processes (or julia -p 4)
nprocs(), nworkers(), workers(), myid()

@everywhere function work(x)         # define on all processes
    sum(sin, 1:x)
end

pmap(work, 1:100)                    # parallel map with load balancing
@distributed (+) for i in 1:10^8     # parallel reduction
    rand()^2
end

r = remotecall(work, 2, 1000)        # run on worker 2
fetch(r)
f = @spawnat :any work(500); fetch(f)

rc = RemoteChannel(() -> Channel{Int}(10))
@everywhere using SharedArrays       # shared memory between local processes
S = SharedArray{Float64}(100)
rmprocs(workers())