I ran into memory overflow or close to memory overflow issues. Thanks to Francesco Nattino for pointing out that compute_mode_and_rms_of_subimages should be a static method. It runs in about 10s now and indeed in parallel. Previously it ran single threaded in about 17s for my 4K*4K test image. Still about twice as slow as the Ray version. However, this Dask version of kappa, sigma clipping distributes subimages over threads while the Ray version distributes rows of subimages over threads which may lead to a smaller overhead.
I ran into memory overflow or close to memory overflow issues. Thanks to Francesco Nattino for pointing out that compute_mode_and_rms_of_subimages should be a static method. It runs in about 10s now and indeed in parallel. Previously it ran single threaded in about 17s for my 4K*4K test image. Still about twice as slow as the Ray version. However, this Dask version of kappa, sigma clipping distributes subimages over threads while the Ray version distributes rows of subimages over threads which may lead to a smaller overhead.

transientskp/pyse,transientskp/pyse