cape-plugin is allocating heap memory within a cuda kernel so that maximum theta e values can be sorted properly. Default device heap memory size is 8MB, for global ECMWF we need N * (expected maxima) * 4 bytes =~ 250MB.
cape-plugin is allocating heap memory within a cuda kernel
so that maximum theta e values can be sorted properly.
Default device heap memory size is 8MB, for global ECMWF
we need N * (expected maxima) * 4 bytes =~ 250MB.

So what happened was that:

https://stackoverflow.com/a/44905235:

> At larger sizes, you are exceeding the size of that heap
> and then your two kernels are attempting to read and write
> from uninitialised memory. This produces a runtime error
> on the device

I'm guessing that in the development maching gpu card (GeForce GTX 1080)
the heap size is bigger, and when debugger is attached at production
server it slows things down just so that all threads are not
allocating memory at the same time.

A bit concerning is that the heap size can be adjusted before kernel
launch, but it can't be adjusted back anymore to the default value.

https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__DEVICE.html#group__CUDART__DEVICE_1g05956f16eaa47ef3a4efee84563ccb7d:

> Setting cudaLimitMallocHeapSize must not be performed after
> launching any kernel that uses the malloc() or free() device
> system calls - in such case cudaErrorInvalidValue will be returned.

I'm not liking this... so after some head-scratching I was able
to rewrite this part of the algorithm without the memory allocation altogether!

fmidev/himan,fmidev/himan,fmidev/himan