CUDA: multi GPUs issue

I'm trying to use multi-GPUs to train the model. When I increase the number of data, this issue happens.
Everything goes well if the size of train set is less than 10000.
Operating System:  Linux
CPU/GPU model:  GPU

@Brian0906 Thanks a lot using experimental CUDA implementation! I observe the same error even with 1 GPU executing simple_example.py: https://github.com/microsoft/LightGBM/pull/3424#issuecomment-702189313.

hi @StrikerRUS  what's the size of trainset in the simple_example.py? I fount that only if the size of dataset is large, this issue happens.

Dataset is very small in the example: 7000x28
https://github.com/microsoft/LightGBM/blob/master/examples/regression/regression.train

Is this the current issue for this problem?  I'm one of the members of the team that ported this CUDA code, and I'm ready to try to reproduce this problem in my environment, if someone can teach me how (preferably with the most simple possible dataset)...

Is this the current issue for this problem?  I'm one of the members of the team that ported this CUDA code, and I'm ready to try to reproduce this problem in my environment, if someone can teach me how (preferably with the most simple possible dataset)...

Hello @austinpagan !
Please refer to https://github.com/microsoft/LightGBM/pull/3428#issuecomment-747676987 for the self-contained repro via Docker. Please let me know if you need any additional details.

Hello @austinpagan !
Please refer to https://github.com/microsoft/LightGBM/pull/3428#issuecomment-747676987 for the self-contained repro via Docker. Please let me know if you need any additional details.

Let me apologize if my questions are seen as somehow inappropriate, as I'm rather new to the open source environment...
OK, so three things I'd like to understand, please:
(1) I could not find, in your link, any guidance that I could understand, can you point me more specifically to the recreate scenario?
(2) Is this problem only seen within Docker containers?
(3) I cannot see in this documentation whether you found this problem on a Power system or on an X86 system.  I know it's only happening when you run on CUDA, but we'd still like to understand the environment.

Let me apologize if my questions are seen as somehow inappropriate, as I'm rather new to the open source environment...
OK, so three things I'd like to understand, please:
(1) I could not find, in your link, any guidance that I could understand, can you point me more specifically to the recreate scenario?
(2) Is this problem only seen within Docker containers?
(3) I cannot see in this documentation whether you found this problem on a Power system or on an X86 system.  I know it's only happening when you run on CUDA, but we'd still like to understand the environment.

@austinpagan No need to apologize! Let me try to be more precise and do my best to answer your questions.

(2) Is this problem only seen within Docker containers?

No, this error can be reproduced w/ and w/o Docker. But I believe Docker is the easiest way to reproduce the error on your side as it ensures we are using the same environment.

(3) I cannot see in this documentation whether you found this problem on a Power system or on an X86 system.

We don't test Power systems. So we can be 100% sure only that X86 systems are affected.

(1) I could not find, in your link, any guidance that I could understand, can you point me more specifically to the recreate scenario?

Get machine (preferably x86, because we cannot guaranty that the bug is reproduced on Power machines) with NVIDIA GPU (we've tested with Tesla M60 and Tesla P100, but I don't think it matters).
Install Docker and NVIDIA Docker (nvidia-docker2) on your machine. https://docs.docker.com/engine/install/ubuntu/ and https://github.com/NVIDIA/nvidia-docker#getting-started can help with this.
Run the following command in your console to get the latest sources of LightGBM:git clone --recursive https://github.com/microsoft/LightGBM

Set environment variable named GITHUB_WORKSPACE to the path where you've downloaded  LightGBM repository at step #2. It will be something like export GITHUB_WORKSPACE=/home/yourUserName/Documents/LightGBM.

Please feel free to ping me if something is still doesn't clear for you or you face any errors during preparing the repro.

@austinpagan No need to apologize! Let me try to be more precise and do my best to answer your questions.

(2) Is this problem only seen within Docker containers?

No, this error can be reproduced w/ and w/o Docker. But I believe Docker is the easiest way to reproduce the error on your side as it ensures we are using the same environment.

(3) I cannot see in this documentation whether you found this problem on a Power system or on an X86 system.

We don't test Power systems. So we can be 100% sure only that X86 systems are affected.

(1) I could not find, in your link, any guidance that I could understand, can you point me more specifically to the recreate scenario?

Get machine (preferably x86, because we cannot guaranty that the bug is reproduced on Power machines) with NVIDIA GPU (we've tested with Tesla M60 and Tesla P100, but I don't think it matters).
Install Docker and NVIDIA Docker (nvidia-docker2) on your machine. https://docs.docker.com/engine/install/ubuntu/ and https://github.com/NVIDIA/nvidia-docker#getting-started can help with this.
Run the following command in your console to get the latest sources of LightGBM:git clone --recursive https://github.com/microsoft/LightGBM

Set environment variable named GITHUB_WORKSPACE to the path where you've downloaded  LightGBM repository at step #2. It will be something like export GITHUB_WORKSPACE=/home/yourUserName/Documents/LightGBM.

Please feel free to ping me if something is still doesn't clear for you or you face any errors during preparing the repro.

So, since we're not set up with Windows boxes here (imagine that), I decided to at least try to see if I could reproduce the problem on a Power system (since, after all, we did this exercise largely to allow folks on Power to access the GPUs, and did not contemplate that X86 folks would experiment with moving from OpenCL to direct CUDA).
INSIDE my docker container on my power box, I just ran the sample and the output looked like this:
(base) [root@58814263a195 python-guide]# python simple_example.py
Loading data...
Starting training...
[LightGBM] [Warning] CUDA currently requires double precision calculations.
[LightGBM] [Warning] Using sparse features with CUDA is currently not supported.
[LightGBM] [Warning] CUDA currently requires double precision calculations.
[LightGBM] [Warning] CUDA currently requires double precision calculations.
[1]	valid_0's l2: 0.244076	valid_0's l1: 0.493018
Training until validation scores don't improve for 5 rounds
[2]	valid_0's l2: 0.240297	valid_0's l1: 0.489056
[3]	valid_0's l2: 0.235733	valid_0's l1: 0.484089
[4]	valid_0's l2: 0.231352	valid_0's l1: 0.479088
[5]	valid_0's l2: 0.228939	valid_0's l1: 0.476159
[6]	valid_0's l2: 0.22593	valid_0's l1: 0.472664
[7]	valid_0's l2: 0.222515	valid_0's l1: 0.468425
[8]	valid_0's l2: 0.219569	valid_0's l1: 0.464594
[9]	valid_0's l2: 0.2168	valid_0's l1: 0.460795
[10]	valid_0's l2: 0.214371	valid_0's l1: 0.457276
[11]	valid_0's l2: 0.211988	valid_0's l1: 0.453923
[12]	valid_0's l2: 0.210264	valid_0's l1: 0.451235
[13]	valid_0's l2: 0.208926	valid_0's l1: 0.448992
[14]	valid_0's l2: 0.207403	valid_0's l1: 0.44634
[15]	valid_0's l2: 0.20601	valid_0's l1: 0.444016
[16]	valid_0's l2: 0.204447	valid_0's l1: 0.441362
[17]	valid_0's l2: 0.202712	valid_0's l1: 0.43891
[18]	valid_0's l2: 0.201066	valid_0's l1: 0.436192
[19]	valid_0's l2: 0.1998	valid_0's l1: 0.433884
[20]	valid_0's l2: 0.198063	valid_0's l1: 0.431129
Did not meet early stopping. Best iteration is:
[20]	valid_0's l2: 0.198063	valid_0's l1: 0.431129
Saving model...
Starting predicting...
[LightGBM] [Warning] CUDA currently requires double precision calculations.
The rmse of prediction is: 0.4450426449744025
(base) [root@58814263a195 python-guide]#
That message about "double precision calculations" is telling me we are using our code.  Is this a good result, or is there an error here?
I also wanted to try a raw run on a lightgbm repository completely outside of the Docker universe, so on a different Power box, I cloned the repository and did the following commands:
cd LightGBM
mkdir build ; cd build
cmake ..
make -j4
That all seemed to work, so I went into the directory with the program and ran it.  It gave me the following fundamental error:
[fossum@rain6p1 python-guide]$ pwd
/home/fossum/LightGBM/examples/python-guide
[fossum@rain6p1 python-guide]$ python3.8 simple_example.py
Traceback (most recent call last):
File "simple_example.py", line 2, in
import lightgbm as lgb
ModuleNotFoundError: No module named 'lightgbm'
[fossum@rain6p1 python-guide]$
I naively went back to the LightGBM and tried "make install" but that was a non-starter.
Not being a python expert, I figured I'd stop here and report my status, so maybe you could give me some pointers...

So, since we're not set up with Windows boxes here (imagine that), I decided to at least try to see if I could reproduce the problem on a Power system (since, after all, we did this exercise largely to allow folks on Power to access the GPUs, and did not contemplate that X86 folks would experiment with moving from OpenCL to direct CUDA).
INSIDE my docker container on my power box, I just ran the sample and the output looked like this:
(base) [root@58814263a195 python-guide]# python simple_example.py
Loading data...
Starting training...
[LightGBM] [Warning] CUDA currently requires double precision calculations.
[LightGBM] [Warning] Using sparse features with CUDA is currently not supported.
[LightGBM] [Warning] CUDA currently requires double precision calculations.
[LightGBM] [Warning] CUDA currently requires double precision calculations.
[1]	valid_0's l2: 0.244076	valid_0's l1: 0.493018
Training until validation scores don't improve for 5 rounds
[2]	valid_0's l2: 0.240297	valid_0's l1: 0.489056
[3]	valid_0's l2: 0.235733	valid_0's l1: 0.484089
[4]	valid_0's l2: 0.231352	valid_0's l1: 0.479088
[5]	valid_0's l2: 0.228939	valid_0's l1: 0.476159
[6]	valid_0's l2: 0.22593	valid_0's l1: 0.472664
[7]	valid_0's l2: 0.222515	valid_0's l1: 0.468425
[8]	valid_0's l2: 0.219569	valid_0's l1: 0.464594
[9]	valid_0's l2: 0.2168	valid_0's l1: 0.460795
[10]	valid_0's l2: 0.214371	valid_0's l1: 0.457276
[11]	valid_0's l2: 0.211988	valid_0's l1: 0.453923
[12]	valid_0's l2: 0.210264	valid_0's l1: 0.451235
[13]	valid_0's l2: 0.208926	valid_0's l1: 0.448992
[14]	valid_0's l2: 0.207403	valid_0's l1: 0.44634
[15]	valid_0's l2: 0.20601	valid_0's l1: 0.444016
[16]	valid_0's l2: 0.204447	valid_0's l1: 0.441362
[17]	valid_0's l2: 0.202712	valid_0's l1: 0.43891
[18]	valid_0's l2: 0.201066	valid_0's l1: 0.436192
[19]	valid_0's l2: 0.1998	valid_0's l1: 0.433884
[20]	valid_0's l2: 0.198063	valid_0's l1: 0.431129
Did not meet early stopping. Best iteration is:
[20]	valid_0's l2: 0.198063	valid_0's l1: 0.431129
Saving model...
Starting predicting...
[LightGBM] [Warning] CUDA currently requires double precision calculations.
The rmse of prediction is: 0.4450426449744025
(base) [root@58814263a195 python-guide]#
That message about "double precision calculations" is telling me we are using our code.  Is this a good result, or is there an error here?
I also wanted to try a raw run on a lightgbm repository completely outside of the Docker universe, so on a different Power box, I cloned the repository and did the following commands:
cd LightGBM
mkdir build ; cd build
cmake ..
make -j4
That all seemed to work, so I went into the directory with the program and ran it.  It gave me the following fundamental error:
[fossum@rain6p1 python-guide]$ pwd
/home/fossum/LightGBM/examples/python-guide
[fossum@rain6p1 python-guide]$ python3.8 simple_example.py
Traceback (most recent call last):
File "simple_example.py", line 2, in
import lightgbm as lgb
ModuleNotFoundError: No module named 'lightgbm'
[fossum@rain6p1 python-guide]$
I naively went back to the LightGBM and tried "make install" but that was a non-starter.
Not being a python expert, I figured I'd stop here and report my status, so maybe you could give me some pointers...

@austinpagan Am I right that you got successful run of the simple_example.py script by following my guide from https://github.com/microsoft/LightGBM/issues/3450#issuecomment-754327830 but without step #0?

That message about "double precision calculations" is telling me we are using our code.

What do you mean by "our code"? CUDA implementation your team contributed to LightGBM repository or some your internal code from a fork?

@austinpagan Am I right that you got successful run of the simple_example.py script by following my guide from https://github.com/microsoft/LightGBM/issues/3450#issuecomment-754327830 but without step #0?

That message about "double precision calculations" is telling me we are using our code.

What do you mean by "our code"?

Easy answer first:
"our code" means CUDA implementation our team contributed to LightGBM repository.
These warnings are only printed out when you run the code requesting the "cuda" device (as opposed to the OpenGL "gpu" device).
Yes, I ran "simple_example.py" following your guide, but skipping both steps 0 and 1, because we already have some Power boxes with functional docker containers, which already contained relatively recent clones of LightGBM, so I just went into one of them, and executed the "simple_example.py" program.
So, again, if you could help us figure out how to get the not-inside-a-container version running, we can hope to see the error there, and I can work on it.  Failing that, my backup suggestion COULD be that I could provide you with a debug version of one source file from our LightGBM, and you could compile that into your favorite local branch of LightGBM, and see what interesting debug data it prints out...

Easy answer first:
"our code" means CUDA implementation our team contributed to LightGBM repository.
These warnings are only printed out when you run the code requesting the "cuda" device (as opposed to the OpenGL "gpu" device).
Yes, I ran "simple_example.py" following your guide, but skipping both steps 0 and 1, because we already have some Power boxes with functional docker containers, which already contained relatively recent clones of LightGBM, so I just went into one of them, and executed the "simple_example.py" program.
So, again, if you could help us figure out how to get the not-inside-a-container version running, we can hope to see the error there, and I can work on it.  Failing that, my backup suggestion COULD be that I could provide you with a debug version of one source file from our LightGBM, and you could compile that into your favorite local branch of LightGBM, and see what interesting debug data it prints out...

Thanks for your prompt response!

which already contained relatively recent clones of LightGBM

Could you be more precise and tell based on what commit your local LightGBM version was compiled? You can check it by running
git rev-parse HEAD

inside your local clone of the repo. Before taking any further steps we should agree on version we will debug with. Because by continuing with different versions of source files we are making the whole debug process pointless.

Thanks for your prompt response!

which already contained relatively recent clones of LightGBM

Could you be more precise and tell based on what commit your local LightGBM version was compiled? You can check it by running
git rev-parse HEAD

inside your local clone of the repo. Before taking any further steps we should agree on version we will debug with. Because by continuing with different versions of source files we are making the whole debug process pointless.

Fortunately for both of us, I'm a morning person.  With the nine hour time difference between Москва and Austin, me being at my computer at 3PM your time will improve our productivity.  To the extent that you can work a bit into your evening, that helps as well!
(base) [root@58814263a195 LightGBM]# pwd
/home/builder/fossum/LightGBM
(base) [root@58814263a195 LightGBM]# git rev-parse HEAD
5d79ff20d1b7ae226531e2445b17d747b253a637
(base) [root@58814263a195 LightGBM]#

Now, if you want me to clone a fresh version of your choosing and try there, that will be fine, but you'll have to walk me through the process of building it to the point where my attempt to run the python test doesn't fail as I had indicated above on my other box.  (My strengths are algorithms and debugging and c coding, not building and installing.)

Fortunately for both of us, I'm a morning person.  With the nine hour time difference between Москва and Austin, me being at my computer at 3PM your time will improve our productivity.  To the extent that you can work a bit into your evening, that helps as well!
(base) [root@58814263a195 LightGBM]# pwd
/home/builder/fossum/LightGBM
(base) [root@58814263a195 LightGBM]# git rev-parse HEAD
5d79ff20d1b7ae226531e2445b17d747b253a637
(base) [root@58814263a195 LightGBM]#

Now, if you want me to clone a fresh version of your choosing and try there, that will be fine, but you'll have to walk me through the process of building it to the point where my attempt to run the python test doesn't fail as I had indicated above on my other box.  (My strengths are algorithms and debugging and c coding, not building and installing.)

I hope it's OK that we're more used to doing our work inside the docker container rather than issuing commands to the container from outside...

I hope it's OK that we're more used to doing our work inside the docker container rather than issuing commands to the container from outside...

Now, if you want me to clone a fresh version of your choosing and try there, that will be fine,

No thanks, I believe that 5d79ff20d1b7ae226531e2445b17d747b253a637 is a good candidate for the debugging! Let's continue with this commit.
Given that simple code runs OK on POWER machine but fails on many x86 ones, it is starting to look like the bug affects only x86 architecture. However, it is quite strange because we are speaking about CUDA code executing on NVIDIA cards here...
I think we can follow your suggestion

my backup suggestion COULD be that I could provide you with a debug version of one source file from our LightGBM, and you could compile that into your favorite local branch of LightGBM, and see what interesting debug data it prints out.

Let me compile LightGBM with the commit we agreed on and run the most verbose version of logs. Then I think you can suggest me some debug code injections and I'll recompile with them and get back with more info. I guess it will be the most efficient form of collaboration given that we do not have an easy access to POWER machines and you do not have a easy access to x86 ones. Please let me know WDYT.

Now, if you want me to clone a fresh version of your choosing and try there, that will be fine,

No thanks, I believe that 5d79ff20d1b7ae226531e2445b17d747b253a637 is a good candidate for the debugging! Let's continue with this commit.
Given that simple code runs OK on POWER machine but fails on many x86 ones, it is starting to look like the bug affects only x86 architecture. However, it is quite strange because we are speaking about CUDA code executing on NVIDIA cards here...
I think we can follow your suggestion

my backup suggestion COULD be that I could provide you with a debug version of one source file from our LightGBM, and you could compile that into your favorite local branch of LightGBM, and see what interesting debug data it prints out.

Let me compile LightGBM with the commit we agreed on and run the most verbose version of logs. Then I think you can suggest me some debug code injections and I'll recompile with them and get back with more info. I guess it will be the most efficient form of collaboration given that we do not have an easy access to POWER machines and you do not have a easy access to x86 ones. Please let me know WDYT.

I am happy with this plan!
I have a recommendation.  If you can try to run your "most verbose" test INSIDE the container as I do, as opposed to running it as a command from outside the container, we can remove that variable as well.  I have a dark suspicion that this may be a problem with Docker not doing a good job when GPUs are involved...
Also, I will just let you know that my plan would be to put more instrumentation around ALL of the CUDA-related memory allocation commands in our code, and they all exist in a single C file, but let's see what your log reports have to say.

I am happy with this plan!
I have a recommendation.  If you can try to run your "most verbose" test INSIDE the container as I do, as opposed to running it as a command from outside the container, we can remove that variable as well.  I have a dark suspicion that this may be a problem with Docker not doing a good job when GPUs are involved...
Also, I will just let you know that my plan would be to put more instrumentation around ALL of the CUDA-related memory allocation commands in our code, and they all exist in a single C file, but let's see what your log reports have to say.

Two more things.
(1) can you teach me how to PROPERLY rebuild LightGBM and the examples so that I'm not just running some old binary that HAPPENS to work?
(2) just FYI, when I type "python --version" it reports: "Python 3.6.9 :: Anaconda, Inc."  Don't know if this matters...

Two more things.
(1) can you teach me how to PROPERLY rebuild LightGBM and the examples so that I'm not just running some old binary that HAPPENS to work?
(2) just FYI, when I type "python --version" it reports: "Python 3.6.9 :: Anaconda, Inc."  Don't know if this matters...

OK, I have setup fresh and minimal environment to start debugging process.

If you can try to run your "most verbose" test INSIDE the container as I do, as opposed to running it as a command from outside the container, we can remove that variable as well. I have a dark suspicion that this may be a problem with Docker not doing a good job when GPUs are involved...

What variable do you mean? I run a bash script inside a docker. It's common practise to ask Docker to run something. It can't be a problem. More proofs come from other reports of the same error. I believe users reported them use pretty different scripts and maybe do not use Docker at all. And they for sure do not use any variables that I use.

(1) can you teach me how to PROPERLY rebuild LightGBM and the examples so that I can be sure I'm not just running some old binary that HAPPENS to work?

Yeah, that's why I've asked you to setup clean Docker environment. I was suspecting that you have some other version of LightGBM that works fine on your side. But now I'm quite confident with that. The thing is that that commit you've told me your version of LightGBM is compiled from simply cannot be compiled. CMake reports the following error.
...
[ 77%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/data_parallel_tree_learner.cpp.o
/LightGBM/src/treelearner/cuda_tree_learner.cpp: In member function 'LightGBM::Tree* LightGBM::CUDATreeLearner::Train(const score_t*, const score_t*)':
/LightGBM/src/treelearner/cuda_tree_learner.cpp:538:59: error: no matching function for call to 'LightGBM::CUDATreeLearner::Train(const score_t*&, const score_t*&)'
  538 |   Tree *ret = SerialTreeLearner::Train(gradients, hessians);
      |                                                           ^
In file included from /LightGBM/src/treelearner/cuda_tree_learner.h:25,
                 from /LightGBM/src/treelearner/cuda_tree_learner.cpp:6:
/LightGBM/src/treelearner/serial_tree_learner.h:78:9: note: candidate: 'virtual LightGBM::Tree* LightGBM::SerialTreeLearner::Train(const score_t*, const score_t*, bool)'
   78 |   Tree* Train(const score_t* gradients, const score_t *hessians, bool is_first_tree) override;
      |         ^~~~~
/LightGBM/src/treelearner/serial_tree_learner.h:78:9: note:   candidate expects 3 arguments, 2 provided
[ 80%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/feature_parallel_tree_learner.cpp.o
make[3]: *** [CMakeFiles/_lightgbm.dir/build.make:407: CMakeFiles/_lightgbm.dir/src/treelearner/cuda_tree_learner.cpp.o] Error 1
make[3]: *** Waiting for unfinished jobs....
make[2]: *** [CMakeFiles/Makefile2:304: CMakeFiles/_lightgbm.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:311: CMakeFiles/_lightgbm.dir/rule] Error 2
make: *** [Makefile:274: _lightgbm] Error 2

This happens due to the following recent changes in LightGBM codebase: fcfd4132e6d40a22d52023396329c41fd3de4a42 (but those changes came before the commit we agreed on).
So you should rebuild LightGBM to match the commit you've specified  (and ensure that compilation fails), or tell me another (older) commit that your LightGBM version is really built from.
However, I went ahead and fixed the error which didn't allow to compile the library.

fcdeb10535d340143f1fe54fc9785dbfd655469c
5eee55cc3e5a531e24530cbcd4f027a4b44ebcdd

These fixes allowed me to successfully compile the library with the commit you've mentioned (5d79ff20d1b7ae226531e2445b17d747b253a637).
Then I specified verbose=4 in simple_example.py to get debug logs from cpp code but unfortunately this didn't help. The error is still the same as before without no additional info.
2021-01-07T15:06:02.5788235Z Loading data...
2021-01-07T15:06:02.5789446Z
2021-01-07T15:06:02.5789792Z Starting training...
2021-01-07T15:06:02.5790650Z [LightGBM] [Warning] CUDA currently requires double precision calculations.
2021-01-07T15:06:02.5791552Z [LightGBM] [Warning] Using sparse features with CUDA is currently not supported.
2021-01-07T15:06:02.5792427Z [LightGBM] [Warning] CUDA currently requires double precision calculations.
2021-01-07T15:06:02.5798769Z Traceback (most recent call last):
2021-01-07T15:06:02.5799483Z   File "simple_example.py", line 38, in <module>
2021-01-07T15:06:02.5799965Z     early_stopping_rounds=5)
2021-01-07T15:06:02.5801170Z   File "/root/.local/lib/python3.6/site-packages/lightgbm/engine.py", line 228, in train
2021-01-07T15:06:02.5801839Z     booster = Booster(params=params, train_set=train_set)
2021-01-07T15:06:02.5802709Z   File "/root/.local/lib/python3.6/site-packages/lightgbm/basic.py", line 2076, in __init__
2021-01-07T15:06:02.5803309Z     ctypes.byref(self.handle)))
2021-01-07T15:06:02.5804122Z   File "/root/.local/lib/python3.6/site-packages/lightgbm/basic.py", line 52, in _safe_call
2021-01-07T15:06:02.5805012Z     raise LightGBMError(_LIB.LGBM_GetLastError().decode('utf-8'))
2021-01-07T15:06:02.5811139Z lightgbm.basic.LightGBMError: [CUDA] invalid argument /LightGBM/src/treelearner/cuda_tree_learner.cpp 414
2021-01-07T15:06:02.5811782Z
2021-01-07T15:06:05.3524322Z ##[error]Process completed with exit code 1.

So I will really appreciate your suggestions for

to put more instrumentation around ALL of the CUDA-related memory allocation commands in our code, and they all exist in a single C file

Speaking about to how re-compile and reinstall LightGBM, it is quite simple.
Commands to compile the dynamic library:
https://github.com/microsoft/LightGBM/blob/5eee55cc3e5a531e24530cbcd4f027a4b44ebcdd/.github/workflows/cuda.yml#L76-L80
Command to install python package with just compiled library:
https://github.com/microsoft/LightGBM/blob/5eee55cc3e5a531e24530cbcd4f027a4b44ebcdd/.github/workflows/cuda.yml#L81
Here is the full script that is used to install and setup Docker, clone repository, install CMake, Python and so on:
https://github.com/microsoft/LightGBM/blob/test_cuda/.github/workflows/cuda.yml

(2) just FYI, when I type "python --version" it reports: "Python 3.6.9 :: Anaconda, Inc." Don't know if this matters...

Thanks! I setup the same Python version (3.6) to mimic your environment.

OK, I have setup fresh and minimal environment to start debugging process.

If you can try to run your "most verbose" test INSIDE the container as I do, as opposed to running it as a command from outside the container, we can remove that variable as well. I have a dark suspicion that this may be a problem with Docker not doing a good job when GPUs are involved...

What variable do you mean? I run a bash script inside a docker. It's common practise to ask Docker to run something. It can't be a problem. More proofs come from other reports of the same error. I believe users reported them use pretty different scripts and maybe do not use Docker at all. And they for sure do not use any variables that I use.

(1) can you teach me how to PROPERLY rebuild LightGBM and the examples so that I can be sure I'm not just running some old binary that HAPPENS to work?

Yeah, that's why I've asked you to setup clean Docker environment. I was suspecting that you have some other version of LightGBM that works fine on your side. But now I'm quite confident with that. The thing is that that commit you've told me your version of LightGBM is compiled from simply cannot be compiled. CMake reports the following error.
...
[ 77%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/data_parallel_tree_learner.cpp.o
/LightGBM/src/treelearner/cuda_tree_learner.cpp: In member function 'LightGBM::Tree* LightGBM::CUDATreeLearner::Train(const score_t*, const score_t*)':
/LightGBM/src/treelearner/cuda_tree_learner.cpp:538:59: error: no matching function for call to 'LightGBM::CUDATreeLearner::Train(const score_t*&, const score_t*&)'
  538 |   Tree *ret = SerialTreeLearner::Train(gradients, hessians);
      |                                                           ^
In file included from /LightGBM/src/treelearner/cuda_tree_learner.h:25,
                 from /LightGBM/src/treelearner/cuda_tree_learner.cpp:6:
/LightGBM/src/treelearner/serial_tree_learner.h:78:9: note: candidate: 'virtual LightGBM::Tree* LightGBM::SerialTreeLearner::Train(const score_t*, const score_t*, bool)'
   78 |   Tree* Train(const score_t* gradients, const score_t *hessians, bool is_first_tree) override;
      |         ^~~~~
/LightGBM/src/treelearner/serial_tree_learner.h:78:9: note:   candidate expects 3 arguments, 2 provided
[ 80%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/feature_parallel_tree_learner.cpp.o
make[3]: *** [CMakeFiles/_lightgbm.dir/build.make:407: CMakeFiles/_lightgbm.dir/src/treelearner/cuda_tree_learner.cpp.o] Error 1
make[3]: *** Waiting for unfinished jobs....
make[2]: *** [CMakeFiles/Makefile2:304: CMakeFiles/_lightgbm.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:311: CMakeFiles/_lightgbm.dir/rule] Error 2
make: *** [Makefile:274: _lightgbm] Error 2

This happens due to the following recent changes in LightGBM codebase: fcfd4132e6d40a22d52023396329c41fd3de4a42 (but those changes came before the commit we agreed on).
So you should rebuild LightGBM to match the commit you've specified  (and ensure that compilation fails), or tell me another (older) commit that your LightGBM version is really built from.
However, I went ahead and fixed the error which didn't allow to compile the library.

fcdeb10535d340143f1fe54fc9785dbfd655469c
5eee55cc3e5a531e24530cbcd4f027a4b44ebcdd

These fixes allowed me to successfully compile the library with the commit you've mentioned (5d79ff20d1b7ae226531e2445b17d747b253a637).
Then I specified verbose=4 in simple_example.py to get debug logs from cpp code but unfortunately this didn't help. The error is still the same as before without no additional info.
2021-01-07T15:06:02.5788235Z Loading data...
2021-01-07T15:06:02.5789446Z
2021-01-07T15:06:02.5789792Z Starting training...
2021-01-07T15:06:02.5790650Z [LightGBM] [Warning] CUDA currently requires double precision calculations.
2021-01-07T15:06:02.5791552Z [LightGBM] [Warning] Using sparse features with CUDA is currently not supported.
2021-01-07T15:06:02.5792427Z [LightGBM] [Warning] CUDA currently requires double precision calculations.
2021-01-07T15:06:02.5798769Z Traceback (most recent call last):
2021-01-07T15:06:02.5799483Z   File "simple_example.py", line 38, in <module>
2021-01-07T15:06:02.5799965Z     early_stopping_rounds=5)
2021-01-07T15:06:02.5801170Z   File "/root/.local/lib/python3.6/site-packages/lightgbm/engine.py", line 228, in train
2021-01-07T15:06:02.5801839Z     booster = Booster(params=params, train_set=train_set)
2021-01-07T15:06:02.5802709Z   File "/root/.local/lib/python3.6/site-packages/lightgbm/basic.py", line 2076, in __init__
2021-01-07T15:06:02.5803309Z     ctypes.byref(self.handle)))
2021-01-07T15:06:02.5804122Z   File "/root/.local/lib/python3.6/site-packages/lightgbm/basic.py", line 52, in _safe_call
2021-01-07T15:06:02.5805012Z     raise LightGBMError(_LIB.LGBM_GetLastError().decode('utf-8'))
2021-01-07T15:06:02.5811139Z lightgbm.basic.LightGBMError: [CUDA] invalid argument /LightGBM/src/treelearner/cuda_tree_learner.cpp 414
2021-01-07T15:06:02.5811782Z
2021-01-07T15:06:05.3524322Z ##[error]Process completed with exit code 1.

So I will really appreciate your suggestions for

to put more instrumentation around ALL of the CUDA-related memory allocation commands in our code, and they all exist in a single C file

Speaking about to how re-compile and reinstall LightGBM, it is quite simple.
Commands to compile the dynamic library:
https://github.com/microsoft/LightGBM/blob/5eee55cc3e5a531e24530cbcd4f027a4b44ebcdd/.github/workflows/cuda.yml#L76-L80
Command to install python package with just compiled library:
https://github.com/microsoft/LightGBM/blob/5eee55cc3e5a531e24530cbcd4f027a4b44ebcdd/.github/workflows/cuda.yml#L81
Here is the full script that is used to install and setup Docker, clone repository, install CMake, Python and so on:
https://github.com/microsoft/LightGBM/blob/test_cuda/.github/workflows/cuda.yml

(2) just FYI, when I type "python --version" it reports: "Python 3.6.9 :: Anaconda, Inc." Don't know if this matters...

Thanks! I setup the same Python version (3.6) to mimic your environment.

Give me like 10 minutes, and I'll do a quick suggestion for some debug around that line 414 in src/treelearner/cuda_tree_learner.cpp...

Give me like 10 minutes, and I'll do a quick suggestion for some debug around that line 414 in src/treelearner/cuda_tree_learner.cpp...

Just for "synchronization" here's the sum check on my cuda_tree_learner.cpp, before I add debug to it:
(base) [root@58814263a195 treelearner]# sum cuda_tree_learner.cpp
36657    40
(base) [root@58814263a195 treelearner]#

Just for "synchronization" here's the sum check on my cuda_tree_learner.cpp, before I add debug to it:
(base) [root@58814263a195 treelearner]# sum cuda_tree_learner.cpp
36657    40
(base) [root@58814263a195 treelearner]#

Give me like 10 minutes, and I'll do a quick suggestion for some debug around that line 414 in

Thank you very much!

Just for "synchronization" here's the sum check on my cuda_tree_learner.cpp, before I add debug to it:

Have you applied two those fixes?

However, I went ahead and fixed the error which didn't allow to compile the library.

fcdeb10
5eee55c

Give me like 10 minutes, and I'll do a quick suggestion for some debug around that line 414 in

Thank you very much!

Just for "synchronization" here's the sum check on my cuda_tree_learner.cpp, before I add debug to it:

Have you applied two those fixes?

However, I went ahead and fixed the error which didn't allow to compile the library.

fcdeb10
5eee55c

This may or may not end up being a "fix" if it helps, but it's useful information to have, and it's an easy change.
Please replace line 414 of src/treelearner/cuda_tree_learner with a different line, as follows:
Current line:
CUDASUCCESS_OR_FATAL(cudaMemcpyAsync(&device_features[copied_feature * num_data_], tmp_data, sizes_in_byte, cudaMemcpyHostToDevice, stream_[device_id]));

Suggested new line:
CUDASUCCESS_OR_FATAL(cudaMemcpy(&device_features[copied_feature * num_data_], tmp_data, sizes_in_byte, cudaMemcpyHostToDevice));

This may or may not end up being a "fix" if it helps, but it's useful information to have, and it's an easy change.
Please replace line 414 of src/treelearner/cuda_tree_learner with a different line, as follows:
Current line:
CUDASUCCESS_OR_FATAL(cudaMemcpyAsync(&device_features[copied_feature * num_data_], tmp_data, sizes_in_byte, cudaMemcpyHostToDevice, stream_[device_id]));

Suggested new line:
CUDASUCCESS_OR_FATAL(cudaMemcpy(&device_features[copied_feature * num_data_], tmp_data, sizes_in_byte, cudaMemcpyHostToDevice));

Have you applied two those fixes?

Here what I'm getting after a patch:
Check sum of cuda_tree_learner.cpp
15848    40

Have you applied two those fixes?

Here what I'm getting after a patch:
Check sum of cuda_tree_learner.cpp
15848    40

Sorry, I don't know how to "apply" a fix.

Sorry, I don't know how to "apply" a fix.

Oh, never mind.  I see now.  Give me a couple minutes.

Oh, never mind.  I see now.  Give me a couple minutes.

Suggested new line:

CUDASUCCESS_OR_FATAL(cudaMemcpy(&device_features[copied_feature * num_data_], tmp_data, sizes_in_byte, cudaMemcpyHostToDevice));

Done! Building...

Sorry, I don't know how to "apply" a fix.

Simply change lines #537-538 in file src/treelearner/cuda_tree_learner.cpp
from
Tree* CUDATreeLearner::Train(const score_t* gradients, const score_t *hessians) {
  Tree *ret = SerialTreeLearner::Train(gradients, hessians);

to
Tree* CUDATreeLearner::Train(const score_t* gradients, const score_t *hessians, bool is_first_tree) {
  Tree *ret = SerialTreeLearner::Train(gradients, hessians, is_first_tree);

and line #48 in file src/treelearner/cuda_tree_learner.h
from
    Tree* Train(const score_t* gradients, const score_t *hessians);

to
    Tree* Train(const score_t* gradients, const score_t *hessians, bool is_first_tree);

Suggested new line:

CUDASUCCESS_OR_FATAL(cudaMemcpy(&device_features[copied_feature * num_data_], tmp_data, sizes_in_byte, cudaMemcpyHostToDevice));

Done! Building...

Sorry, I don't know how to "apply" a fix.

Simply change lines #537-538 in file src/treelearner/cuda_tree_learner.cpp
from
Tree* CUDATreeLearner::Train(const score_t* gradients, const score_t *hessians) {
  Tree *ret = SerialTreeLearner::Train(gradients, hessians);

to
Tree* CUDATreeLearner::Train(const score_t* gradients, const score_t *hessians, bool is_first_tree) {
  Tree *ret = SerialTreeLearner::Train(gradients, hessians, is_first_tree);

and line #48 in file src/treelearner/cuda_tree_learner.h
from
    Tree* Train(const score_t* gradients, const score_t *hessians);

to
    Tree* Train(const score_t* gradients, const score_t *hessians, bool is_first_tree);

Done! Building...

Built! Unfortunately, no changes...
I'm afraid that I can't get more debug info due to the following our issue: #3641.

Done! Building...

Built! Unfortunately, no changes...
I'm afraid that I can't get more debug info due to the following our issue: #3641.

And, on my end, I can't get this code to build.  So frustrating...

And, on my end, I can't get this code to build.  So frustrating...

so, when you say "unfortunately, no changes" you mean the error reported is exactly the same, even with the change I proposed?  That would be good news, because it means the error is actually in THAT CALL, and not in some previous call in the same "async thread"...

so, when you say "unfortunately, no changes" you mean the error reported is exactly the same, even with the change I proposed?  That would be good news, because it means the error is actually in THAT CALL, and not in some previous call in the same "async thread"...

still claiming the problem is in line 414, right?

still claiming the problem is in line 414, right?

And, on my end, I can't get this code to build. So frustrating...

"this code" = code with these fixes https://github.com/microsoft/LightGBM/issues/3450#issuecomment-756209798?
Maybe you don't have all source files? Could you please try to re-clone the repo and only after that apply a fix?
git clone --recursive https://github.com/microsoft/LightGBM.git
cd LightGBM
git checkout 5d79ff20d1b7ae226531e2445b17d747b253a637

<apply fixes to src/treelearner/cuda_tree_learner.h and src/treelearner/cuda_tree_learner.cpp>

so, when you say "unfortunately, no changes" you mean the error reported is exactly the same, even with the change I proposed?

Yes, absolutely right.

still claiming the problem is in line 414, right?

I guess so. At least the error comes from line 414...

And, on my end, I can't get this code to build. So frustrating...

"this code" = code with these fixes https://github.com/microsoft/LightGBM/issues/3450#issuecomment-756209798?
Maybe you don't have all source files? Could you please try to re-clone the repo and only after that apply a fix?
git clone --recursive https://github.com/microsoft/LightGBM.git
cd LightGBM
git checkout 5d79ff20d1b7ae226531e2445b17d747b253a637

<apply fixes to src/treelearner/cuda_tree_learner.h and src/treelearner/cuda_tree_learner.cpp>

so, when you say "unfortunately, no changes" you mean the error reported is exactly the same, even with the change I proposed?

Yes, absolutely right.

still claiming the problem is in line 414, right?

I guess so. At least the error comes from line 414...

so when I try to build, it's trying to get files from the "external_libs" directory, but in my clone, that directory just contains two empty sub-directories...  any idea whether I'm missing some piece of the build that populates those directories?  It looks like there's a "setup.py" file that mentions this directory, but I don't know who is supposed to execute that setup command...

so when I try to build, it's trying to get files from the "external_libs" directory, but in my clone, that directory just contains two empty sub-directories...  any idea whether I'm missing some piece of the build that populates those directories?  It looks like there's a "setup.py" file that mentions this directory, but I don't know who is supposed to execute that setup command...

We are investigating, but I figured it wouldn't hurt to ask you if you just know the answer off the top of your head...

We are investigating, but I figured it wouldn't hurt to ask you if you just know the answer off the top of your head...

that directory just contains two empty sub-directories...

Please make sure you don't forgot --recursive flag during cloning the repo.
git clone --recursive https://github.com/microsoft/LightGBM.git

that directory just contains two empty sub-directories...

Please make sure you don't forgot --recursive flag during cloning the repo.
git clone --recursive https://github.com/microsoft/LightGBM.git

I've tried and can confirm that we can reproduce the error with simple command-line program. I simplified reproducible example so that it no longer requests Python installation. I believe it will help to sync environments.
Fortunately, the error is still the same. But we do not need a proxy of Python layer anymore. Now we run simple regression example from the repository directly via CLI version of LightGBM. Previously we run it via our Python-package.
Please take a look at greatly simplified script (no Python, no any env. variables) we run inside a Docker to reproduce the error:
https://github.com/microsoft/LightGBM/blob/bcc3f291c8470bd680aa0c332cfaa3b1a0d01bdd/.github/workflows/cuda.yml#L43-L62
This script

ensures we have NVIDIA card available inside a Docker via nvidia-smi
installs curl and CMake to compile LightGBM
changes default device type from cpu to cuda in source config file (we will see later [Warning] CUDA currently requires double precision calculations. warning that proves successful change)
checks sum of cuda_tree_learner.cpp
compiles lightgbm executable programm
runs regression example

I've tried and can confirm that we can reproduce the error with simple command-line program. I simplified reproducible example so that it no longer requests Python installation. I believe it will help to sync environments.
Fortunately, the error is still the same. But we do not need a proxy of Python layer anymore. Now we run simple regression example from the repository directly via CLI version of LightGBM. Previously we run it via our Python-package.
Please take a look at greatly simplified script (no Python, no any env. variables) we run inside a Docker to reproduce the error:
https://github.com/microsoft/LightGBM/blob/bcc3f291c8470bd680aa0c332cfaa3b1a0d01bdd/.github/workflows/cuda.yml#L43-L62
This script

ensures we have NVIDIA card available inside a Docker via nvidia-smi
installs curl and CMake to compile LightGBM
changes default device type from cpu to cuda in source config file (we will see later [Warning] CUDA currently requires double precision calculations. warning that proves successful change)
checks sum of cuda_tree_learner.cpp
compiles lightgbm executable programm
runs regression example

And here are more verbose logs from the run after applying your proposed change in 414 line of src/treelearner/cuda_tree_learner.cpp file https://github.com/microsoft/LightGBM/issues/3450#issuecomment-756205729:
2021-01-07T18:54:57.1318861Z [LightGBM] [Warning] CUDA currently requires double precision calculations.
2021-01-07T18:54:57.1320390Z [LightGBM] [Info] Finished loading parameters
2021-01-07T18:54:57.1320991Z [LightGBM] [Debug] Loading train file...
2021-01-07T18:54:57.1405940Z [LightGBM] [Info] Loading initial scores...
2021-01-07T18:54:57.1597220Z [LightGBM] [Warning] Using sparse features with CUDA is currently not supported.
2021-01-07T18:54:58.2787014Z [LightGBM] [Debug] Loading validation file #1...
2021-01-07T18:54:58.2879002Z [LightGBM] [Info] Loading initial scores...
2021-01-07T18:54:58.2964932Z [LightGBM] [Info] Finished loading data in 1.165807 seconds
2021-01-07T18:54:58.2965532Z [LightGBM] [Info] LightGBM using CUDA trainer with DP float!!
2021-01-07T18:54:58.2971585Z [LightGBM] [Info] Total Bins 6132
2021-01-07T18:54:58.2981032Z [LightGBM] [Info] Number of data points in the train set: 7000, number of used features: 28
2021-01-07T18:54:58.2981689Z [LightGBM] [Debug] device_bin_size_ = 256
2021-01-07T18:54:58.2982161Z [LightGBM] [Debug] Resized feature masks
2021-01-07T18:54:58.2982684Z [LightGBM] [Debug] Memset pinned_feature_masks_
2021-01-07T18:54:58.2983679Z [LightGBM] [Debug] Allocated device_features_ addr=0x7ff5aaa00000 sz=196000
2021-01-07T18:54:58.2985727Z [LightGBM] [Debug] Memset device_data_indices_
2021-01-07T18:54:58.2991002Z [LightGBM] [Fatal] [CUDA] invalid argument /LightGBM/src/treelearner/cuda_tree_learner.cpp 414
2021-01-07T18:54:58.2995493Z [LightGBM] [Debug] created device_subhistograms_: 0x7ff5ab000000
2021-01-07T18:54:58.3027139Z
2021-01-07T18:54:58.3027684Z [LightGBM] [Debug] Started copying dense features from CPU to GPU
2021-01-07T18:54:58.3028247Z Met Exceptions:
2021-01-07T18:54:58.3028802Z [CUDA] invalid argument /LightGBM/src/treelearner/cuda_tree_learner.cpp 414
2021-01-07T18:54:58.3029237Z
2021-01-07T18:54:58.3030255Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 1
2021-01-07T18:54:58.3031103Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3031917Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3032773Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3033581Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3034408Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3035216Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3036038Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3036843Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3037660Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3038459Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3039263Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3040077Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3041108Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3041993Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3042794Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3043607Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3044405Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3045225Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3046029Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3046847Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3047646Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3048447Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3049264Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3050082Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3050902Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3051702Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3052521Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3053318Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3054138Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3054939Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3055754Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3056550Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3057347Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3058351Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3059161Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3059976Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3060773Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3061589Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3062382Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:59.6885338Z ##[error]Process completed with exit code 255.

Hope they will help somehow. Please let me know how can I modify the source code of CUDA treelearner more to get useful info that will help to narrow the problem.

And here are more verbose logs from the run after applying your proposed change in 414 line of src/treelearner/cuda_tree_learner.cpp file https://github.com/microsoft/LightGBM/issues/3450#issuecomment-756205729:
2021-01-07T18:54:57.1318861Z [LightGBM] [Warning] CUDA currently requires double precision calculations.
2021-01-07T18:54:57.1320390Z [LightGBM] [Info] Finished loading parameters
2021-01-07T18:54:57.1320991Z [LightGBM] [Debug] Loading train file...
2021-01-07T18:54:57.1405940Z [LightGBM] [Info] Loading initial scores...
2021-01-07T18:54:57.1597220Z [LightGBM] [Warning] Using sparse features with CUDA is currently not supported.
2021-01-07T18:54:58.2787014Z [LightGBM] [Debug] Loading validation file #1...
2021-01-07T18:54:58.2879002Z [LightGBM] [Info] Loading initial scores...
2021-01-07T18:54:58.2964932Z [LightGBM] [Info] Finished loading data in 1.165807 seconds
2021-01-07T18:54:58.2965532Z [LightGBM] [Info] LightGBM using CUDA trainer with DP float!!
2021-01-07T18:54:58.2971585Z [LightGBM] [Info] Total Bins 6132
2021-01-07T18:54:58.2981032Z [LightGBM] [Info] Number of data points in the train set: 7000, number of used features: 28
2021-01-07T18:54:58.2981689Z [LightGBM] [Debug] device_bin_size_ = 256
2021-01-07T18:54:58.2982161Z [LightGBM] [Debug] Resized feature masks
2021-01-07T18:54:58.2982684Z [LightGBM] [Debug] Memset pinned_feature_masks_
2021-01-07T18:54:58.2983679Z [LightGBM] [Debug] Allocated device_features_ addr=0x7ff5aaa00000 sz=196000
2021-01-07T18:54:58.2985727Z [LightGBM] [Debug] Memset device_data_indices_
2021-01-07T18:54:58.2991002Z [LightGBM] [Fatal] [CUDA] invalid argument /LightGBM/src/treelearner/cuda_tree_learner.cpp 414
2021-01-07T18:54:58.2995493Z [LightGBM] [Debug] created device_subhistograms_: 0x7ff5ab000000
2021-01-07T18:54:58.3027139Z
2021-01-07T18:54:58.3027684Z [LightGBM] [Debug] Started copying dense features from CPU to GPU
2021-01-07T18:54:58.3028247Z Met Exceptions:
2021-01-07T18:54:58.3028802Z [CUDA] invalid argument /LightGBM/src/treelearner/cuda_tree_learner.cpp 414
2021-01-07T18:54:58.3029237Z
2021-01-07T18:54:58.3030255Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 1
2021-01-07T18:54:58.3031103Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3031917Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3032773Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3033581Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3034408Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3035216Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3036038Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3036843Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3037660Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3038459Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3039263Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3040077Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3041108Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3041993Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3042794Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3043607Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3044405Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3045225Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3046029Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3046847Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3047646Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3048447Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3049264Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3050082Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3050902Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3051702Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3052521Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3053318Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3054138Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3054939Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3055754Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3056550Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3057347Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3058351Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3059161Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3059976Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3060773Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:58.3061589Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 3
2021-01-07T18:54:58.3062382Z [LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
2021-01-07T18:54:59.6885338Z ##[error]Process completed with exit code 255.

Hope they will help somehow. Please let me know how can I modify the source code of CUDA treelearner more to get useful info that will help to narrow the problem.

So, sorry for the delay in response.  My colleague seems to be close to figuring out how we can reproduce this problem on Power systems.  You can rest easy for now, because if he is successful, we can handle it from here on out...

So, sorry for the delay in response.  My colleague seems to be close to figuring out how we can reproduce this problem on Power systems.  You can rest easy for now, because if he is successful, we can handle it from here on out...

Oh, great news! Thank you very much!

Oh, great news! Thank you very much!

And, it is confirmed.  On my power system, I get this now:
(base) [root@58814263a195 python-guide]# python simple_example.py
Loading data...
Starting training...
[LightGBM] [Warning] CUDA currently requires double precision calculations.
[LightGBM] [Warning] Using sparse features with CUDA is currently not supported.
[LightGBM] [Warning] CUDA currently requires double precision calculations.
[LightGBM] [Fatal] [CUDA] invalid argument /home/builder/fossum/LightGBM/src/treelearner/cuda_tree_learner.cpp 414
Traceback (most recent call last):
File "simple_example.py", line 39, in
early_stopping_rounds=5)
File "/opt/anaconda3/lib/python3.6/site-packages/lightgbm/engine.py", line 228, in train
booster = Booster(params=params, train_set=train_set)
File "/opt/anaconda3/lib/python3.6/site-packages/lightgbm/basic.py", line 2076, in init
ctypes.byref(self.handle)))
File "/opt/anaconda3/lib/python3.6/site-packages/lightgbm/basic.py", line 52, in _safe_call
raise LightGBMError(_LIB.LGBM_GetLastError().decode('utf-8'))
lightgbm.basic.LightGBMError: [CUDA] invalid argument /home/builder/fossum/LightGBM/src/treelearner/cuda_tree_learner.cpp 414
(base) [root@58814263a195 python-guide]#

And, it is confirmed.  On my power system, I get this now:
(base) [root@58814263a195 python-guide]# python simple_example.py
Loading data...
Starting training...
[LightGBM] [Warning] CUDA currently requires double precision calculations.
[LightGBM] [Warning] Using sparse features with CUDA is currently not supported.
[LightGBM] [Warning] CUDA currently requires double precision calculations.
[LightGBM] [Fatal] [CUDA] invalid argument /home/builder/fossum/LightGBM/src/treelearner/cuda_tree_learner.cpp 414
Traceback (most recent call last):
File "simple_example.py", line 39, in
early_stopping_rounds=5)
File "/opt/anaconda3/lib/python3.6/site-packages/lightgbm/engine.py", line 228, in train
booster = Booster(params=params, train_set=train_set)
File "/opt/anaconda3/lib/python3.6/site-packages/lightgbm/basic.py", line 2076, in init
ctypes.byref(self.handle)))
File "/opt/anaconda3/lib/python3.6/site-packages/lightgbm/basic.py", line 52, in _safe_call
raise LightGBMError(_LIB.LGBM_GetLastError().decode('utf-8'))
lightgbm.basic.LightGBMError: [CUDA] invalid argument /home/builder/fossum/LightGBM/src/treelearner/cuda_tree_learner.cpp 414
(base) [root@58814263a195 python-guide]#

So, again, I can pursue this now, without pestering you.  Wish me luck!

So, again, I can pursue this now, without pestering you.  Wish me luck!

Hah, in any other situation people shouldn't be happy when someone another gets errors from software, but right now I'm happy! 😄
Hope it won't be hard to find a root cause for you.
Again, if you are not comfortable using Python, please check this my message where I show how to reproduce the same error with LightGBM's executable binary from command line interface. Feel free to ask for any details if something is not clear.

Hah, in any other situation people shouldn't be happy when someone another gets errors from software, but right now I'm happy! 😄
Hope it won't be hard to find a root cause for you.
Again, if you are not comfortable using Python, please check this my message where I show how to reproduce the same error with LightGBM's executable binary from command line interface. Feel free to ask for any details if something is not clear.

This problem is that the non-CUDA vector allocators were changed to use kAlignedSize with VirtualFileWriter::AlignedSize between 3.0 and 3.1.  Therefore the CUDA vector allocator wasn't allocating enough space in some instances.  Here is a purpose change to fix the CUDA allocator.  simple_example.py and advanced_example.py work with this change.
diff --git a/include/LightGBM/cuda/vector_cudahost.h b/include/LightGBM/cuda/vector_cudahost.h
index 03db338..46698d0 100644
--- a/include/LightGBM/cuda/vector_cudahost.h
+++ b/include/LightGBM/cuda/vector_cudahost.h
@@ -42,6 +42,7 @@ struct CHAllocator {
   T* allocate(std::size_t n) {
     T* ptr;
     if (n == 0) return NULL;
+    n = (n + kAlignedSize) & ~kAlignedSize;
     #ifdef USE_CUDA
       if (LGBM_config_::current_device == lgbm_device_cuda) {
         cudaError_t ret = cudaHostAlloc(&ptr, n*sizeof(T), cudaHostAllocPortable);

This problem is that the non-CUDA vector allocators were changed to use kAlignedSize with VirtualFileWriter::AlignedSize between 3.0 and 3.1.  Therefore the CUDA vector allocator wasn't allocating enough space in some instances.  Here is a purpose change to fix the CUDA allocator.  simple_example.py and advanced_example.py work with this change.
diff --git a/include/LightGBM/cuda/vector_cudahost.h b/include/LightGBM/cuda/vector_cudahost.h
index 03db338..46698d0 100644
--- a/include/LightGBM/cuda/vector_cudahost.h
+++ b/include/LightGBM/cuda/vector_cudahost.h
@@ -42,6 +42,7 @@ struct CHAllocator {
   T* allocate(std::size_t n) {
     T* ptr;
     if (n == 0) return NULL;
+    n = (n + kAlignedSize) & ~kAlignedSize;
     #ifdef USE_CUDA
       if (LGBM_config_::current_device == lgbm_device_cuda) {
         cudaError_t ret = cudaHostAlloc(&ptr, n*sizeof(T), cudaHostAllocPortable);

@austinpagan @ChipKerchner Awesome! I can confirm that this fix helps to get rid from errors on X86 machines as well. Many thanks for the research you've done and providing the fix!
Would you like to contribute this fix from your account so that GitHub will associate fixing commit with you?  I can confirm that this fix helps to get rid from errors on X86 machines as well. Many thanks for the research you've done and providing the fix!
Would you like to contribute this fix from your account so that GitHub will associate fixing commit with you? Or maybe it's not very important for you and you prefer to let do this to someone from LightGBM maintainers to save your time?

Fixed via #3748.

Fixed via #3748.

This should fix the remaining CUDA failures.  Let me know if you see any issues.
diff --git a/src/treelearner/cuda_tree_learner.cpp b/src/treelearner/cuda_tree_learner.cpp
index 16569ee..4495578 100644
--- a/src/treelearner/cuda_tree_learner.cpp
+++ b/src/treelearner/cuda_tree_learner.cpp
@@ -408,7 +408,7 @@ void CUDATreeLearner::copyDenseFeature() {
     // looking for dword_features_ non-sparse feature-groups
     if (!train_data_->IsMultiGroup(i)) {
       dense_feature_group_map_.push_back(i);
-      auto sizes_in_byte = train_data_->FeatureGroupSizesInByte(i);
+      auto sizes_in_byte = std::min(train_data_->FeatureGroupSizesInByte(i), static_cast<size_t>(num_data_));
       void* tmp_data = train_data_->FeatureGroupData(i);
       Log::Debug("Started copying dense features from CPU to GPU - 2");
       CUDASUCCESS_OR_FATAL(cudaMemcpyAsync(&device_features[copied_feature * num_data_], tmp_data, sizes_in_byte, cu
@@ -534,8 +534,8 @@ void CUDATreeLearner::InitGPU(int num_gpu) {
   copyDenseFeature();
 }

diff --git a/include/LightGBM/cuda/vector_cudahost.h b/include/LightGBM/cuda/vector_cudahost.h
index bd488d7..6685b3a 100644
--- a/include/LightGBM/cuda/vector_cudahost.h
+++ b/include/LightGBM/cuda/vector_cudahost.h
@@ -42,7 +42,7 @@ struct CHAllocator {
   T* allocate(std::size_t n) {
     T* ptr;
     if (n == 0) return NULL;
-    n = (n + kAlignedSize - 1) & -kAlignedSize;
+    n = (n + kAlignedSize - 1) & -static_cast<size_t>(kAlignedSize);
     #ifdef USE_CUDA
       if (LGBM_config_::current_device == lgbm_device_cuda) {
         cudaError_t ret = cudaHostAlloc(&ptr, n*sizeof(T), cudaHostAllocPortable);

This should fix the remaining CUDA failures.  Let me know if you see any issues.
diff --git a/src/treelearner/cuda_tree_learner.cpp b/src/treelearner/cuda_tree_learner.cpp
index 16569ee..4495578 100644
--- a/src/treelearner/cuda_tree_learner.cpp
+++ b/src/treelearner/cuda_tree_learner.cpp
@@ -408,7 +408,7 @@ void CUDATreeLearner::copyDenseFeature() {
     // looking for dword_features_ non-sparse feature-groups
     if (!train_data_->IsMultiGroup(i)) {
       dense_feature_group_map_.push_back(i);
-      auto sizes_in_byte = train_data_->FeatureGroupSizesInByte(i);
+      auto sizes_in_byte = std::min(train_data_->FeatureGroupSizesInByte(i), static_cast<size_t>(num_data_));
       void* tmp_data = train_data_->FeatureGroupData(i);
       Log::Debug("Started copying dense features from CPU to GPU - 2");
       CUDASUCCESS_OR_FATAL(cudaMemcpyAsync(&device_features[copied_feature * num_data_], tmp_data, sizes_in_byte, cu
@@ -534,8 +534,8 @@ void CUDATreeLearner::InitGPU(int num_gpu) {
   copyDenseFeature();
 }

diff --git a/include/LightGBM/cuda/vector_cudahost.h b/include/LightGBM/cuda/vector_cudahost.h
index bd488d7..6685b3a 100644
--- a/include/LightGBM/cuda/vector_cudahost.h
+++ b/include/LightGBM/cuda/vector_cudahost.h
@@ -42,7 +42,7 @@ struct CHAllocator {
   T* allocate(std::size_t n) {
     T* ptr;
     if (n == 0) return NULL;
-    n = (n + kAlignedSize - 1) & -kAlignedSize;
+    n = (n + kAlignedSize - 1) & -static_cast<size_t>(kAlignedSize);
     #ifdef USE_CUDA
       if (LGBM_config_::current_device == lgbm_device_cuda) {
         cudaError_t ret = cudaHostAlloc(&ptr, n*sizeof(T), cudaHostAllocPortable);

@ChipKerchner After applying this fix all but two tests are passed! Very nice indeed!
Failures in two remaining plotting tests are not related to CUDA implementation. I believe it is the same graphviz environment issue as in https://github.com/microsoft/LightGBM/pull/3672#issuecomment-757642931.
============================= test session starts ==============================
platform linux -- Python 3.8.2, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /LightGBM
collected 238 items

../tests/c_api_test/test_.py ..                                          [  0%]
../tests/python_package_test/test_basic.py .............                 [  6%]
../tests/python_package_test/test_consistency.py ......                  [  8%]
../tests/python_package_test/test_dask.py ............................   [ 20%]
../tests/python_package_test/test_dual.py s                              [ 21%]
../tests/python_package_test/test_engine.py ............................ [ 32%]
.......................................                                  [ 49%]
../tests/python_package_test/test_plotting.py F...F                      [ 51%]
../tests/python_package_test/test_sklearn.py ........................... [ 62%]
......x.........................................x....................... [ 92%]
.................                                                        [100%]
= 2 failed, 233 passed, 1 skipped, 2 xfailed, 74 warnings in 195.32s (0:03:15) =

@ChipKerchner After applying this fix all but two tests are passed! Very nice indeed!
Failures in two remaining plotting tests are not related to CUDA implementation. I believe it is the same graphviz environment issue as in https://github.com/microsoft/LightGBM/pull/3672#issuecomment-757642931.
============================= test session starts ==============================
platform linux -- Python 3.8.2, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /LightGBM
collected 238 items

../tests/c_api_test/test_.py ..                                          [  0%]
../tests/python_package_test/test_basic.py .............                 [  6%]
../tests/python_package_test/test_consistency.py ......                  [  8%]
../tests/python_package_test/test_dask.py ............................   [ 20%]
../tests/python_package_test/test_dual.py s                              [ 21%]
../tests/python_package_test/test_engine.py ............................ [ 32%]
.......................................                                  [ 49%]
../tests/python_package_test/test_plotting.py F...F                      [ 51%]
../tests/python_package_test/test_sklearn.py ........................... [ 62%]
......x.........................................x....................... [ 92%]
.................                                                        [100%]
= 2 failed, 233 passed, 1 skipped, 2 xfailed, 74 warnings in 195.32s (0:03:15) =

Failures in two remaining plotting tests are not related to CUDA implementation. I believe it is the same graphviz environment issue as in #3672 (comment).
============================= test session starts ==============================
platform linux -- Python 3.8.2, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /LightGBM
collected 238 items

../tests/python_package_test/test_plotting.py F...F                      [ 51%]

In my branch , test_plotting passes all tests.
python -m unittest tests/python_package_test/test_plotting.py
.../test_plotting.py:156: UserWarning: More than one metric available, picking one to plot.
  ax0 = lgb.plot_metric(evals_result0)
..s
----------------------------------------------------------------------
Ran 5 tests in 1.956s

OK (skipped=2)

Failures in two remaining plotting tests are not related to CUDA implementation. I believe it is the same graphviz environment issue as in #3672 (comment).
============================= test session starts ==============================
platform linux -- Python 3.8.2, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /LightGBM
collected 238 items

../tests/python_package_test/test_plotting.py F...F                      [ 51%]

In my branch , test_plotting passes all tests.
python -m unittest tests/python_package_test/test_plotting.py
.../test_plotting.py:156: UserWarning: More than one metric available, picking one to plot.
  ax0 = lgb.plot_metric(evals_result0)
..s
----------------------------------------------------------------------
Ran 5 tests in 1.956s

OK (skipped=2)

woo! Thanks @ChipKerchner . Like @StrikerRUS mentioned, I think it's very very unlikely that the two failing plotting tests are related to your changes. I found in https://github.com/microsoft/LightGBM/pull/3672#issuecomment-757642931 that there might be some issues with the conda-forge recipe for graphviz.

woo! Thanks @ChipKerchner . Like @StrikerRUS mentioned, I think it's very very unlikely that the two failing plotting tests are related to your changes. I found in https://github.com/microsoft/LightGBM/pull/3672#issuecomment-757642931 that there might be some issues with the conda-forge recipe for graphviz.

Yeah, thanks for the info about tests @ChipKerchner ! I'm 100% sure that 2 failing plotting tests on our side is related to our environment. And I'll fix this environment issue during working on making CUDA builds run on a regular basis.

Yeah, thanks for the info about tests @ChipKerchner ! I'm 100% sure that 2 failing plotting tests on our side is related to our environment. And I'll fix this environment issue during working on making CUDA builds run on a regular basis.

  Thank you!

  Thank you!

@austinpagan Thanks a lot for all your hard work!

@austinpagan Thanks a lot for all your hard work!


Jest doesn't exit after test is completed

I currently use jest --forceExit in my test script because it appears that my mongodb connection refuses to close.
I would assume that I should close the connection using either mongoose.disconnect() or mongoose.connection.close()
Either solution so far has failed for me.
I welcome any solution for this.

https://github.com/HyperTHD/BTS530-TS-Backend/commit/38f4fe28d554ddb2774039ff20ebcd4e8a689b72 switches from mock-mongoose to mongodb-memory-server. Package works just as fine with less code and as a result, the jest exiting problem has been solved and this issue can now be successfully closed.


Unable to edit review

I created a review with one picture in it
Later on I attempted to edit it and add another picture.
The review was not saved.
See attached screen movie
Expected Behavior
Expected existing review updated with new picture
Current Behavior
I am guessing that the upload buttons are not properly connected to a model so they dont know that there is already a picture in this review, anyway, currently just does not work
https://user-images.githubusercontent.com/53953892/113946434-7401c280-97d6-11eb-92cf-804e995aabbc.mov

Duplicate issue as #93


More questions than issues...

I have more questions than issues. Thanks for your help in clarifying this.

Why is port 53 (DNS) opened on UFW for incoming traffic? If it's to connect to Cloudflare - that port default is 853? The reason I ask about this is because I've considered running WG on port 53 for some scenarios.
Does the CoreDNS stuff need to be enabled? If so, can I download the install script and remove that? If I would do that, what would break?
Any plans for IPv6 support?
Any plans for accounts to have a little more restriction - ex. Help Desk person being able to create and delete peers but not change the server config?
Plans to support additional WG interfaces wg1, wg2, etc.?

Also, how does one 'update'?

oh uh.... the DNS port should only be open on the wg0 interface! It is used for coredns so a secure dns can be provided over the vpn.
No you don't need to enable coredns. But then you have to ensure to use another dns server through the vpn tunnel to avoid dns leaks or other funky stuff. Also coredns uses DNS-over-TLS to encrypt the dns traffic between your server and Cloudflare.
No not currently. But we are open for PRs here :)
No because the dashboard is only reachable via ssh and letting an untrusted user connect to ssh has its whole own pack of security considerations that we didn't want to include in the project scope.
No not for now. What would be your intended use case here?

I took a stab at the updater with #22


License file in repo?

Hi,
Would you mind adding an MIT license file into the body of this repository? Your NPM registry and package.json declare it as MIT licensed, but I don't see one in the repository, and I don't think I can use this code base without that. I think Github has a button on the repo settings to do that for you.
Thanks!

Hey, thanks for the feedback. I added the corresponding LICENCE file.

Thank you!


Upgrade to Debian 12.5

I don't know why this wasn't merged in, but at this point, it would be easier to just create a new patch. Sorry, I have to close this.


IsAnyDialogOpen not working for Childwindows

IsAnyDialogOpen property is set to false although a child window is shown using the ShowChildWindowAsync by the MahApps.Metro.SimpleChildWindow extension
What steps will reproduce this issue?
var result = await this.ShowChildWindowAsync<bool>(new UserControl { IsModal = false });

// somewhere else in code

if (this.IsAnyDialogOpen == false)
 MessageBox.Show("you shouldn' see this);

Expected outcome
Property should return TRUE if there is a child window shown. Otherwise there should be another property for it.
Environment

MahApps.Metro 1.6.1
MahApps.Metro.SimpleChildWindow 1.5
Windows OS Win10
Visual Studio 2017
.NET Framework 4.7

@Symbai moved this issue to https://github.com/punker76/MahApps.Metro.SimpleChildWindow/issues/73


Updating labels on existing metrics

Hi, I've got a quick question.
If I collect a metric and have done so for some time, but now decide I want to add an additional label to samples going forward, what would be the intended way to do so?
As far as I can tell, the Collector throws an exception if the label count doesn't match, but that practically never happens as the labels aren't loaded when a new process tries to update a sample with different labels, leading to undefined behavior.
With main1.php (bootstrapping excluded for brevity):
$registry = new CollectorRegistry(new Redis());

$counter = $registry->getOrRegisterCounter('test', 'some_counter', 'it increases', ['shape']);
$counter->incBy(3, ['square']);

$renderer = new RenderTextFormat();
$result = $renderer->render($registry->getMetricFamilySamples());

echo $result;

and main2.php:
$registry = new CollectorRegistry(new Redis());

$counter = $registry->getOrRegisterCounter('test', 'some_counter', 'it increases still', ['shape', 'color']);
$counter->incBy(3, ['square', 'blue']);

$renderer = new RenderTextFormat();
$result = $renderer->render($registry->getMetricFamilySamples());

echo $result;

it renders as follows:
$ php main1.php
# HELP test_some_counter it increases
# TYPE test_some_counter counter
test_some_counter{shape="square"} 3
$ php main2.php
PHP Warning:  array_combine(): Both parameters should have an equal number of elements in <snip>/vendor/endclothing/prometheus_client_php/src/Prometheus/RenderTextFormat.php on line 43
PHP Warning:  Invalid argument supplied for foreach() in <snip>/vendor/endclothing/prometheus_client_php/src/Prometheus/RenderTextFormat.php on line 44
# HELP test_some_counter it increases still
# TYPE test_some_counter counter
test_some_counter{} 3
test_some_counter{shape="square",color="blue"} 3

Imo, I should be able to add new dimensions to my samples as my application grows.

You will need to clear the registry after deploying the code with the new labels


Toplist signs still not working

Detailed description of the issue: Signs still not working after update, this time the signs just look empty when writing Toplist signs. Join signs work tho.

Jobs version: 4.17.2
Server Type (Spigot/Paper/etc): paper
Server Version (using /ver): 567

My mistake, had to delete the old jobs folder


QoS Class should be Guaranteed

Describe the bug
The QoS class of MySQL pods is Burstable whereas it is preferred to be Guaranteed.
$ kubectl -n dev-dbre describe pod moco-stage0-fujita-1-0 | grep -i qos
QoS Class:                   Burstable
$

Environments

Version: v0.10.9
OS: Flatcar Container Linux (Stable)

To Reproduce
Steps to reproduce the behavior:

Deploy MySQLCluster
kubectl describe pod PODNAME | grep QoS

Expected behavior
QoS Class is Guaranteed.
Additional context
According to our experience of running MOCO for the past half year in our development environment, the max memory usages by containers except for mysqld are as follows. The memory usages are observed by container_memory_working_set_bytes from cAdvisor.

container
max memory usage

agent (moco-agent)
60 MiB

slow-log (fluent-bit)
9 MiB

mysqld_exporter
38 MiB

@arosh Could you move this to #235 ?

@ymmt2005
I transcribed the max memory usages to #235. Thanks.


model_selection/iterative_stratification.py throws AttributeError: 'numpy.ndarray' object has no attribute 'iloc'

The method iterative_stratification throws the error AttributeError: 'numpy.ndarray' object has no attribute 'iloc'.
In this method, X and y are used in the following function:
train_indexes, test_indexes = next(stratifier.split(X, y))
train_indexes, test_indexes are numpy arrays, not pandas dataframes.  Thus, .iloc in the following code block (lines 105-106) results in the error:
    X_train, y_train = X.iloc[train_indexes, :], y.iloc[train_indexes, :]
    X_test, y_test = X.iloc[test_indexes, :], y.iloc[test_indexes, :]
Removing '.iloc' from this code fixes the error.
It looks like the four instances of '.iloc' should be removed from these two lines of code.  (Note:  This is the first time I've submitted an issue here; please let me know if I need to provide anything else.)

if X and y are pandas dataframes and train_indexes,test_indexes are numpy arrays/list then removing slice should do the job. Try this:
 X_train, y_train = X.iloc[train_indexes], y.iloc[train_indexes]
X_test, y_test = X.iloc[test_indexes], y.iloc[test_indexes]

Thanks, jayant-yadav.  I had already fixed the file to run in my notebook, but raised the issue so that someone could make a fix to the code so that when people use this library, they don't also run into the same error.


Add Python Type comments throughout the library

Add Python Type comments throughout the library, include a py.typed file in the main package, and verify all typing with MyPy, so that PySOA can be used with MyPy in other projects.

Done in d7b39041d3253b702e4c532c6326fb275fa075a9.


Update access for cas_admin and cas_analyst

Dependencies
Blocked by #2493 and #2494 (not technically blocked, but will avoid conflicts and be easier to wrap our heads around)
Blocked by #2522
Description:
As a cas-admin or cas_analyst user, I should have edit and view-only functionality as listed in this table.
Acceptance Criteria:
cas_admin:

[ ] Only editable piece is internal user access
[x] View-only access to the external user grid
[x] can't assign boro or bcghg id
[x] Check and update the endpoint and service permissions for this new role to ensure the user has the correct access.
cas_analyst
[ ] Manage first-time operator admin role approvals
[ ] Manage Transfers
[x] view only everything else
both
[x] reg 1 still needs to be accessible by cas_admin and cas_analyst, same as before

Development Checklist:

[x] update conditionality to the operator admin role approvals (added in #2494) so that cas_admin cannot manage first-time operator admin role approvals (only cas_director and cas_analyst can)

[x] can't assign boro or bcghg id (this should already be done as part of #2493

[x] make sure reg 1 users still have access

[x] confirm that cas_admin can/can't only do the things in the miro board

[x] all cas users can view everything, so check if we need the internal admin vs internal dashboard, or if we can just use one

[ ] [ ] Meets the DOD

Definition of Ready (Note: If any of these points are not applicable, mark N/A)

[ ] User story is included
[ ] User role and type are identified
[ ] Acceptance criteria are included
[ ] Wireframes are included (if required)
[ ] Design / Solution is accepted by Product Owner
[ ] Dependencies are identified (technical, business, regulatory/policy)
[ ] Story has been estimated (under 13 pts)

·Definition of Done (Note: If any of these points are not applicable, mark N/A)

[ ] Acceptance criteria are tested by the CI pipeline
[ ] UI meets accessibility requirements
[ ] Configuration changes are documented, documentation and designs are updated
[ ] Passes code peer-review
[ ] Passes QA of Acceptance Criteria with verification in Dev and Test
[ ] Ticket is ready to be merged to main branch
[ ] Can be demoed in Sprint Review
[ ] Bugs or future work cards are identified and created
[ ] Reviewed and approved by Product Owner

Notes:

Manage first-time operator admin role approvals (#2059) and Manage Transfers (#2427) are in-progress, so no role can do either of these at the moment. Giving access to the correct roles is part of those tickets

Manage first-time operator admin role approvals (https://github.com/bcgov/cas-registration/issues/2059) and Manage Transfers (https://github.com/bcgov/cas-registration/issues/2427) are in-progress are in progress so the functionality should be tested as part of those tickets. I can assume the roles and navigate in the dev environment


increase test coverage

adds test for post and pre methods
adds missing and unknown attribute tests for card

Coverage increased (+1.9%) to 76.852% when pulling 3bd5238d341fefb18f21b3d47777533910253538 on js/missingCoverage into a16feed03fe51c5f80484754856c074f6ecdef37 on master.


📝📑📗 [Docs] (code + docs) Add package documentation

Target

Add package documentation.
Unified the coding style of docstring in source code.

Modify Code Scope

Source Code

sub-package in module _/utils
module crawler
module election
sub-package model

Documentation

Add RST format files in directory docs/.

Scripts

Add sample code.

Effecting Scope

Nothing, it adds new feature of this package --- documentation of this package.

Description

Add or modify the docstring in source code for package documentation.
Add Sphinx format documentation.
Add sample code in script directory.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

38 of 38 (100.0%)  changed or added relevant lines in 5 files are covered.
2 unchanged lines in 1 file lost coverage.
Overall coverage increased (+0.04%) to 94.363%

Files with Coverage Reduction
New Missed Lines
%

smoothcrawler_cluster/crawler.py
2
88.25%

Totals

Change from base Build<PHONE_NUMBER>:
0.04%

Covered Lines:
1088

Relevant Lines:
1153

💛  - Coveralls


Unary Attributes (OPTION SELECTED)

Looking in the docs, I'm struggling with how to add a selected attribute to a SELECT OPTION.
OPTION({value: 'eager', selected: i.evaluate.to.true}, "Pick me, pick me");

This doesn't seem to work.

What you did there should work (assuming that the selected attribute does indeed evaluate to true at some point). Alternatively, you can set the value on the select element itself. Either should work.

Well, assigning the value to the SELECT element worked. Will close this. Thank you!


Install guides are unclear that Tectonic is pure Kubernetes

Feature Request
Environment
Baremetal
Desired Feature
Make direct links to guides clear about Tectonic being 100% pure upstream Kubernetes.
https://coreos.com/tectonic/docs/latest/install/bare-metal/index.html
Other Information
I often link directly to Tectonic AWS installer or Tectonic Baremetal when I get questions about how to install Kubernetes. Why? Because the installer is so easy / good!
I'm realizing that these docs may not make it clear that Tectonic is pure Kubernetes, though. Should we add a line to the top of the docs (or somewhere else) that clarifies that Tectonic is a pure implementation of k8s?

Sure, want to open a PR?

Thanks @pgburt - we'll make sure to clarify that in our docs!

@pgburt Should we just repost this image?

(found on coreos.com/tectonic landing page)

Going to close this issue for now, as we've updated our docs to better clarify this.


AutoLinks option to open in new window

Hi,
I noticed that JiraLinks has this feature, but not AutoLinks. Would you consider a PR for this?
Thanks.

sure.

When I open the solution with VS2015 (it looks like the .sln file was made by VS2015) I cannot load the Markdig project. I get:
Markdig.csproj : error  : The default XML namespace of the project must be the MSBuild XML namespace. If the project is authored in the MSBuild 2003 format, please add xmlns="http://schemas.microsoft.com/developer/msbuild/2003" to the <Project> element. If the project has been authored in the old 1.0 or 1.2 format, please convert it to MSBuild 2003 format.

If I try with VS2017, when compiling Markdig, I get:
Severity	Code	Description	Project	File	Line	Suppression State
Error	APPX3217	SDK folder containing 'UAP.props' for 'UAP 10.0.10240.0' cannot be located. See http://go.microsoft.com/fwlink/?LinkID=798187 for more information.	Markdig	C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\Microsoft\VisualStudio\v15.0\AppxPackage\Microsoft.AppXPackage.Targets	874

Any ideas?

You need most likely to install UWP components (10240) from the Visual Studio 2017 installer. Don't recall exactly the option... I should add a part in the readme about this.

I installed Universal Windows Platform, but Windows 10 SDK (10.0.10240.0) was marked as optional. After checking / installing the option I'm now able to build. Thanks.

JiraLinks has the OpenInNewWindow property true by default. As far as I can tell, the tests only check for the true value.
In the case of AutoLink, setting a similar value to true would be a breaking change, so we should probably have it false by default. As far as testing, would you like a clone of AutoLinks.md that has target="blank" or just have hand written tests (i.e. not auto generated from .md files) that would test the new option on AutoLink?
Thanks


aiohttp 3.10 typed signature incompatibility

The problem
aiohttp 3.10 added fine-grained types to the request wrapper methods' kwargs, which you can see here.  Currently, calls which specify expire_after do not type check.
Expected behavior
No type errors.
Steps to reproduce the behavior
Type check e.g. client.get('foo', expire_after=0).
Workarounds
Call request instead of one of the wrapper methods, though the fact its kwargs are not typed is probably an oversight.
Environment

aiohttp-client-cache version: 0.11.0
Python version: 3.11+
Platform: n/a

Thank you for opening an issue.
I would like to clarify a few things to make sure we are talking about the same thing.

No type errors.

Perhaps you wanted to say: Expected behavior: to see type errors?
Because now Pyright works correctly

but Mypy shows no warnings when I pass fake_argument="foobar".

client.get('foo', expire_after=0)

This is far a way from a minimal reproducible example. It would be great to get a full snippet that can be copied and tested.
Can you show a type error that you get? Are you talking about Mypy? About Pyright?

@JWCook With aiohttp==3.10.0, do you know how to get Mypy to display the same warning as Pyright in my screenshot above?

Yes, Pyright is behaving correctly based on the function's definition.  However, passing arbitrary keyword arguments to session.get and friends did not produce type errors prior to aiohttp 3.10, because they were annotated to accept Any. Now that their parameters are spelled out, passing expire_after, which is valid with a CachedSession, produces a type error.

By the way, expire_after is a backend argument.

It isn't advertised in the docs, but it is possible to pass expire_after to request methods, which forwards it via **kwargs to CachedSession._request. (added in #49)
@layday  I can see what you're talking about from looking at aiohttp's client.py. I'd be willing to add typed wrapper methods for get(), post(), etc. Is that something you'd like to submit a PR for?
I can't reproduce this locally, though. What mypy settings are you using?
For example:
from asyncio import run
from aiohttp_client_cache import CachedSession, SQLiteBackend

async def test():
    async with CachedSession(cache=SQLiteBackend('demo_cache')) as session:
        await session.get('http://httpbin.org/get', expire_after=360)

run(test())

$ pip install -U aiohttp-client-cache[sqlite] 'aiohttp>=3.10' mypy
$ mypy test.py
'Success: no issues found in 1 source file'

As an aside, aiohttp's usage of typing.Unpack with a TypedDict is new to me! Looks like a nice way to reuse verbose function signatures. I'm currently doing something similar here for the docs (example) using python-forge , but I might like this approach better for simple cases like adding a single extra argument.

https://github.com/requests-cache/aiohttp-client-cache/pull/249

I can't reproduce this locally, though. What mypy settings are you using?

I'm using Pyright. It looks like MyPy's not registering this as a type error, oddly.

Your changes are in 0.11.1. Thanks!


Shopping cart visual clickthrough elements

Adding initial shopping cart hooks

I think I am going to wait till after demo to merge this ;-)


FEATURE: Remove Liquidity from any Uniswapv2 Pool to ETH or any ERC20

As a user, I want to call a VolumeFi vyper contract to remove liquidity from any Uniswap pool by liquiditing my LP tokens at the cost of a flat rate of 0.005 ETH on the Uniswap pool and then exchange my tokens for either ETH or any ERC20, that may include USDC or USDT or other.
Requirements

Secure for SafeMath and checking for re-entrancy type issues.

Read LP token reserves in Uniswap Pool.

Swap LP tokens for ETH or ERC20

Withdraw ETH or ERC20 to the withdrawal address.

Check for valid amounts and valid exchange

The contract will charge 0.005 ETH fee per transaction for all successful.

Acceptance Criteria

Test coverage should be available on brownie for test
Optimize for gas costs to be lower than solidity variants

Closing as delivered.


Demote severity of ExtendsAutoValue and RefersToDaggerCodegen to a warning to not break Java cross compilation in Bazel

Both checks appear to be a breaking changes for Bazel EP integration.
It broke all Bazel users who rely on cross compilation: [1], [2], and was the reason, why the first attempt to bump EP version to 2.3.3 in Bazel was reverted: [3], [4].
The way Bazel cross compilation works is a bit convoluted, but the reason for false positive boils down to: the missing annotation generation is using -source 8 -target 8 with a JDK 11 -bootclasspath (because it's targeting the remote JDK 11). So AutoValue processor is trying to use javax.annotation.Generated based on the source level, but it isn't available because the JDK 11 bootclasspath only contains javax.annotation.processing.Generated.
Demote the severity to warning for the checks that rely on having
JDK 8's: javax.annotation.Generated or JDK 11's javax.annotation.processing.Generated on the classpath today, and consider to promote it to error severity again later, when Java 8 is gone.
That would substantially simplify the adoption of new EP releases, because it wouldn't require the addition of javax.annotation library to the build tool chain.
[1] https://github.com/bazelbuild/rules_closure/issues/483
[2] https://github.com/bazelbuild/rules_kotlin/issues/321
[3] https://github.com/bazelbuild/bazel/commit/d990746
[4] https://github.com/bazelbuild/bazel/issues/10023

#1348


Add Travis-CI configuration

This allows the project to be automatically built and tested with the latest version of Perl6 on the Travis-CI continuous integration service.

Thank you!


mbed new requires mercurial to be installed, even though it is not necessary

I installed mbed-cli using pip and tried to create a new project. Since I want to use git, I didn't think I needed to install hg.
However, the mbed new --offline --scm git -v test_project command fails, when it tries to clone the mbed-os using hg, and when hg is not found, it aborts...
Ironically, if I install hg, the clone command still fails as below, but the setup then tries to use git and succeeds.
[mbed] Exec "hg clone https://github.com/ARMmbed/mbed-os C:\Users\owner\Desktop\mbed\test_project\mbed-os" in "C:\Users\owner\Desktop\mbed\test_project"
abort: 'https://github.com/ARMmbed/mbed-os' does not appear to be an hg repository:
---%<--- (text/html; charset=utf-8)

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
  <link rel="dns-prefetch" href="https://github.githubassets.com">
  <link rel="dns-prefetch" href="https://avatars0.githubusercontent.com">
  <link rel="dns-prefetch" href="https://avatars1.githubusercontent.com">
  <link rel="dns-prefetch" href="https://avatars2.githubusercontent.com">
  <link rel="dns-prefetch" href="https://avatars3.githubusercontent.com">
  <link rel="dns-prefetch" href="https://github-cloud.s3.amazonaws.com">
  <link rel="dns-prefetch" href="https://user-images.githubusercontent.com/">

  <link crossorigin="anonymous" media="all" integrity="sha512-R+Vpkv86him5JZcqAEuQRUGOKqH897w6q7uJ1P65tQR+9Hxar5vU4wpEd4uvcXT8ooRZ7zsNftrjnCemEt2u2Q==" rel="stylesheet" href="https://github.githubassets.com/assets/frameworks-f4557b27209914aa4705202b188165b5.css" />
  <link crossorigin="anonymous" media="all" integrity="sha512-fr6+cq8eNA1ibps13PoHVqLSAKa6rd6vyZrEvJW9cUbARpp4qjMA0+WqELgIBXVgveT9zDkjUtoguP1rjKR4tA==" rel="styleshee
---%<---
!
[mbed] Exec "git clone https://github.com/ARMmbed/mbed-os C:\Users\owner\Desktop\mbed\test_project\mbed-os" in "C:\Users\owner\Desktop\mbed\test_project"

Internal Jira reference: https://jira.arm.com/browse/MBOCUSTRIA-981

This is documented here https://os.mbed.com/docs/mbed-os/v5.11/tools/linux.html
Sorry, but we have no plans to change this behavior.


DeepTile does not arrange tiles correctly on an nd2 file

Hi folks, I am trying to use deep to perform 2D stitching on a multi-position nd2 file acquired on our spinning disc. Below is what is use to run DeepTile but I noticed that the FOVs are not in the correct position in the stitched dataset. Let me also mention that my raw multi-position .nd2 file is not a regular shape. How does DeepTile read stage positions from the .nd2 file. Here is my script that i currently use.
import numpy as np
import tifffile
from deeptile import load
from deeptile.extensions import stitch
from pathlib import Path
import zarr
from skimage.transform import pyramid_gaussian

# Define the input file path
image_path = '/Processed/33975/33975-far label.nd2'
image = Path(image_path)

# Load the image and get tiles
dt = load(str(image))
tiles = dt.get_tiles()

# Stitch the image with blending
stitched = stitch.stitch_image(tiles, blend=True, sigma=10)

# Create a multi-resolution pyramid
pyramid = tuple(pyramid_gaussian(stitched, max_layer=4, downscale=2, multichannel=True))

# Define the output file path for the pyramid
output_path_pyramid = image.parent / (image.stem + '_stitched_pyramid.zarr')

# Save the pyramid as a Zarr file with zlib compression
zarr_group = zarr.open_group(str(output_path_pyramid), mode='w')
compressor = zarr.Zlib(level=3)  # Set the compression level
for i, level in enumerate(pyramid):
    zarr_group.create_dataset(f'level_{i}', data=level, chunks=True, compressor=compressor)

print(f"Multi-resolution pyramid saved to {output_path_pyramid}")

@Eddymorphling Do your .nd2 files have tiles that have a consistent overlap with each other? If possible, it would be helpful to provide a sample file so I could debug further!


test: check if default bucket object quota is applied

Resolves: rhbz#1391500
Signed-off-by: Ali Maredia<EMAIL_ADDRESS>

@andrewschoen once this is merged I will cherry-pick this commit and the other rgw testing related one to stable-2.2

jenkins test jewel-ansible2.2-docker_cluster

@andrewschoen are we good to go with this one?


Re-enable variable name shadowing

Currently, variable name shadowing is disabled due to the difficulty in unifying different backend.
After #16 is done, name shadowing can be re-enabled.

Closed by #24