[CLI]: wandb created two sessions for the same run on a slurm cluster

Describe the bug
My python script calls wandb.init() only once (at the beginning of the python script). On a normal remote machine or a local machine, the python script runs without any problems, one launch of the python script created one wandb session.
However, when I launch the same python script on a cluster (which is managed through slurm), one run of the script creates two wandb sessions. Each of these two wandb sessions has its own run name, for example, glowing-wind-255 and pretty-energy-256. These two wandb sessions started at exactly the same time, but their duration can differ by some seconds, this is shown in the screenshot below:

The (beginning of) terminal logging of one run which created two wandb sessions is shown as follows:

How can I prevent one run from creating two duplicate wandb sessions?
Additional Files
No response
Environment
WandB version: 0.13.5
OS: Linux-3.10.0-1160.11.1.el7.x86_64-x86_64-with-glibc2.10
Python version: 3.8.5
Versions of relevant libraries:
Additional Context
No response

Hi @SunHaozhe thank you for reporting this. May I please ask how you're executing the experiment on the Slurm cluster, are you using a Slurm job script and would it be possible to share this (excluding any private info such as keys)? could you please also describe your use case, your training framework and if you want to do distributed training in the cluster?

I am using a slurm/bash script to launch job arrays, this script is launched by the command sbatch. My slurm/bash script is shown as follows (excluding some sensitive information):
#!/bin/bash
#SBATCH --job-name=XXX
#SBATCH --partition=YYY
#SBATCH --qos=ZZZ
#SBATCH --mem=16g
#SBATCH --nodes=1
#SBATCH --cpus-per-task=1
#SBATCH --open-mode=append
#SBATCH --signal=B:USR1@120
#SBATCH --requeue
#SBATCH --array=0-15
#SBATCH --output=logs/%A_%a.stdout
#SBATCH --error=logs/%A_%a.stderr
#SBATCH --exclude=n[1-5]

SECONDS=0

restart(){
	echo "Calling restart"
	scontrol requeue $SLURM_JOB_ID
	echo "Scheduled job for restart"
}

ignore(){
	echo "Ignored SIGTERM"
}
trap restart USR1
trap ignore TERM

date

args=()

for SEED in "1" "2" "3" "4" "5" "6" "7" "8" "9" "10" "11" "12" "13" "14" "15" "16"
do
	args+=("run.py --random_seed ${SEED}")
done

echo "Starting python ${args[${SLURM_ARRAY_TASK_ID}]}"

srun python ${args[${SLURM_ARRAY_TASK_ID}]}

echo "End python ${args[${SLURM_ARRAY_TASK_ID}]}"

DURATION=$SECONDS

echo "End of the program! $(($DURATION / 60)) minutes and $(($DURATION % 60)) seconds elapsed."

My use case is that I want to run the same experiment with multiple random seeds to report the average performance of my model. Each call to run.py is one independent experiment, there are no dependency or interaction between experiments.
My training framework is PyTorch 1.8.1.
I am not sure what you meant by distributed training. As mentioned above, in my case, each call to run.py is an independent experiment, experiments have no interaction between them at all. Each call to run.py uses 0 or 1 GPU (GPUs are not shared among experiments). So I think this is not "distributed training".
run.py calls wandb.init() only once (at its beginning).
The slurm cluster is in a protected network, accessing Internet from within the slurm cluster requires to go through a specific proxy host, which supports http and https transfers.
Don't hesitate to let me know if you need further information!

I just add one recent observation.
In very rare cases, one of the two duplicate wandb sessions got the error wandb.errors.UsageError: Error communicating with wandb process in the very beginning of the process. However (and luckily), this error does not interrupt the other wandb process nor the whole training process.
One example of such terminal logs (only stderr, excluded stdout) are shown as follows:
wandb: Currently logged in as: sunhaozhe. Use `wandb login --relogin` to force relogin
wandb: Currently logged in as: sunhaozhe. Use `wandb login --relogin` to force relogin
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandwandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: Network error (ReadTimeout), entering retry loop.
b.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: Network error (ReadTimeout), entering retry loop.
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: ERROR Error while calling W&B API: internal database error (<Response [500]>)
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: ERROR Error while calling W&B API: internal database error (<Response [500]>)
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: ERROR Error while calling W&B API: internal database error (<Response [500]>)
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: Tracking run with wandb version 0.13.5
wandb: Run data is saved locally in wandb_logs/wandb/run-20221128_181056-1l023j3h
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run sparkling-durian-9
wandb: ⭐ View project at https://wandb.ai/sunhaozhe/XXX
wandb: 🚀 View run at https://wandb.ai/sunhaozhe/XXX/runs/1l023j3h
wandb: Network error resolved after 0:00:53.691939, resuming normal operation.
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: ERROR Error communicating with wandb process
wandb: ERROR For more info see: https://docs.wandb.ai/library/init#init-start-error
wandb: Waiting for W&B process to finish... (failed 1). Press Control-C to abort syncing.
wandb: ERROR Error while calling W&B API: internal database error (<Response [500]>)
wandb: Synced 2 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: wandb_logs/wandb/run-20221128_181056-3ccmf6d6/logs
Traceback (most recent call last):
  File "run.py", line 664, in <module>
    project_name, group_name, wandb_run_name = init_wandb(args)
  File "run.py", line 638, in init_wandb
    wandb.init(config=args, project=project_name,
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/wandb/sdk/wandb_init.py", line 1078, in init
    run = wi.init()
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/wandb/sdk/wandb_init.py", line 719, in init
    raise UsageError(error_message)
wandb.errors.UsageError: Error communicating with wandb process
For more info see: https://docs.wandb.ai/library/init#init-start-error
srun: error: n51: task 1: Exited with exit code 1
wandb: Waiting for W&B process to finish... (success).
wandb:
wandb: Run history:
wandb:                 epoch ▁▃▅▆█
wandb:      time_ratio_optim █▄▁▁▇
wandb:             train_acc ▁▄▆▇█
wandb:      train_epoch_time ▆▁▇█▄
wandb:            train_loss █▅▃▂▁
wandb: train_optim_batch_avg █▁▇█▅
wandb:       true_optim_time █▁▇█▅
wandb:
wandb: Run summary:
wandb:               PyTorch_version 1.8.1
wandb:    count_trainable_parameters 466138
wandb:                         epoch 5
wandb:             nb_available_GPUs 0
wandb:                    nb_classes 20
wandb:                  overall_time 833.9291
wandb:                         shift 13
wandb:                      test_acc 49.75
wandb:               test_epoch_time 16.24823
wandb: test_inference_time_batch_avg 1.23273
wandb:              time_ratio_optim 0.90322
wandb:                  timm_version 0.5.4
wandb:                 torch_version 1.8.1
wandb:                     train_acc 42.73438
wandb:              train_epoch_time 9.34155
wandb:                    train_loss 1.73
wandb:         train_optim_batch_avg 0.64903
wandb:               true_optim_time 8.43744
wandb:
wandb: Synced sparkling-durian-9: https://wandb.ai/sunhaozhe/XXX/runs/1l023j3h
wandb: Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: wandb_logs/wandb/run-20221128_181056-1l023j3h/logs

Hi @SunHaozhe thank you for the detailed information. For the initial issue, could you please try to specify the GPU as you would do when running Sweep agents on a multi-GPU machine. Would it work for you if you were setting the CUDA_VISIBLE_DEVICES?
Regarding the new issue, it would help if you could attach the debug.log and debug-internal.log files of both runs (which can be found in the two directories that are mentioned in the terminal log above). Thanks!

Thanks for following up on these two issues. I will first provide information for the second issue.
I am initializing 6 python runs at the same time at most because our slurm cluster set the limit to 6. However, as each python run created two wandb sessions, this could mean 12 runs from the point of view of wandb server. Some of the python runs are rather quick, so it is possible that more than 6 python runs are initialized within one minute.
Below I provide debug.log and debug-internal.log files of both runs.
For the wandb session run-20221128_181056-3ccmf6d6 (which failed), debug.log is
2022-11-28 18:10:56,808 INFO    MainThread:22891 [wandb_setup.py:_flush():68] Configure stats pid to 22891
2022-11-28 18:10:56,813 INFO    MainThread:22891 [wandb_setup.py:_flush():68] Loading settings from /mnt/beegfs/home/sun/.config/wandb/settings
2022-11-28 18:10:56,814 INFO    MainThread:22891 [wandb_setup.py:_flush():68] Loading settings from /mnt/beegfs/home/sun/MoE_private/exp_20220804_github/wandb/settings
2022-11-28 18:10:56,814 INFO    MainThread:22891 [wandb_setup.py:_flush():68] Loading settings from environment variables: {'_require_service': 'True'}
2022-11-28 18:10:56,814 INFO    MainThread:22891 [wandb_setup.py:_flush():68] Inferring run settings from compute environment: {'program_relpath': 'exp_20220804_github/run.py', 'program': 'run.py'}
2022-11-28 18:10:56,815 INFO    MainThread:22891 [wandb_init.py:_log_setup():476] Logging user logs to wandb_logs/wandb/run-20221128_181056-3ccmf6d6/logs/debug.log
2022-11-28 18:10:56,816 INFO    MainThread:22891 [wandb_init.py:_log_setup():477] Logging internal logs to wandb_logs/wandb/run-20221128_181056-3ccmf6d6/logs/debug-internal.log
2022-11-28 18:10:56,816 INFO    MainThread:22891 [wandb_init.py:init():516] calling init triggers
2022-11-28 18:10:56,817 INFO    MainThread:22891 [wandb_init.py:init():519] wandb.init called with sweep_config: {}
config: {'wandb_project_name': 'new_RRR_CPU_time_benchmark_nb_branch_LogScale', 'random_seed': 5, 'epochs': 5, 'batch_size': 32, 'num_workers': 0, 'lr': 0.001, 'weight_decay': 0.01, 'lr_scheduler': 'none', 'optimizer': 'AdamW', 'loss': 'cross_entropy', 'img_size': 128, 'no_wandb': False, 'do_validation': False, 'dataset': 'MetaAlbumMicro_BCT_Micro', 'train_examples_per_class': 20, 'val_examples_per_class': 0, 'test_examples_per_class': 20, 'divergence_idx': 15, 'nb_branch': 16, 'split_pretrained_weights': False, 'warm_stem_epochs': 0, 'nb_blocks': [1, 1, 1, 1], 'pretrained_model': 'resnet152', 'ConcatenatedModel_train_layers': 'freeze_stem_train_branch', 'ema_epochs': None, 'run_baseline': False, 'baseline_model': 'resnet152', 'train_layers': 'only_last', 'run_FGE': False, 'fge_lr_1': 0.001, 'fge_lr_2': 0.0005, 'dropout_2d_replace_branch': 1000, 'use_wandb': True}
2022-11-28 18:10:56,818 INFO    MainThread:22891 [wandb_init.py:init():569] starting backend
2022-11-28 18:10:56,818 INFO    MainThread:22891 [wandb_init.py:init():573] setting up manager
2022-11-28 18:10:56,825 INFO    MainThread:22891 [backend.py:_multiprocessing_setup():102] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
2022-11-28 18:10:56,848 INFO    MainThread:22891 [wandb_init.py:init():580] backend started and connected
2022-11-28 18:10:56,862 INFO    MainThread:22891 [wandb_init.py:init():658] updated telemetry
2022-11-28 18:10:56,898 INFO    MainThread:22891 [wandb_init.py:init():693] communicating run to backend with 60 second timeout
2022-11-28 18:11:56,969 ERROR   MainThread:22891 [wandb_init.py:init():704] backend process timed out
2022-11-28 18:11:56,969 ERROR   MainThread:22891 [wandb_init.py:init():713] encountered error: Error communicating with wandb process
For more info see: https://docs.wandb.ai/library/init#init-start-error
2022-11-28 18:12:47,504 WARNING MsgRouterThr:22891 [router.py:message_loop():77] message_loop has been closed

For the wandb session run-20221128_181056-3ccmf6d6 (which failed), debug-internal.log is
2022-11-28 18:10:56,836 INFO    StreamThr :23157 [internal.py:wandb_internal():88] W&B internal server running at pid: 23157, started at: 2022-11-28 18:10:56.835156
2022-11-28 18:10:56,838 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: status
2022-11-28 18:10:56,847 DEBUG   SenderThread:23157 [sender.py:send_request():317] send_request: status
2022-11-28 18:10:56,863 DEBUG   SenderThread:23157 [sender.py:send():303] send: header
2022-11-28 18:10:56,863 INFO    WriterThread:23157 [datastore.py:open_for_write():75] open: wandb_logs/wandb/run-20221128_181056-3ccmf6d6/run-3ccmf6d6.wandb
2022-11-28 18:10:56,901 DEBUG   SenderThread:23157 [sender.py:send():303] send: run
2022-11-28 18:11:01,904 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:11:06,905 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:11:11,907 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:11:16,908 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:11:21,910 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:11:26,912 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:11:30,622 ERROR   SenderThread:23157 [retry.py:__call__():154] Retry attempt failed:
Traceback (most recent call last):
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/http/client.py", line 1347, in getresponse
    response.begin()
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/urllib3/util/retry.py", line 410, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/urllib3/connectionpool.py", line 428, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/urllib3/connectionpool.py", line 335, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='api.wandb.ai', port=443): Read timed out. (read timeout=10)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/wandb/sdk/lib/retry.py", line 113, in __call__
    result = self._call_fn(*args, **kwargs)
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/wandb/sdk/internal/internal_api.py", line 210, in execute
    return self.client.execute(*args, **kwargs)  # type: ignore
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py", line 52, in execute
    result = self._get_result(document, *args, **kwargs)
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py", line 60, in _get_result
    return self.transport.execute(document, *args, **kwargs)
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/transport/requests.py", line 38, in execute
    request = requests.post(self.url, **post_args)
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.wandb.ai', port=443): Read timed out. (read timeout=10)
2022-11-28 18:11:31,918 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:11:36,927 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:11:41,866 ERROR   SenderThread:23157 [internal_api.py:execute():213] 500 response executing GraphQL.
2022-11-28 18:11:41,867 ERROR   SenderThread:23157 [internal_api.py:execute():214] {"errors":[{"message":"internal database error","path":["upsertBucket"]}],"data":{"upsertBucket":null}}
2022-11-28 18:11:41,936 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:11:46,948 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:11:50,573 ERROR   SenderThread:23157 [internal_api.py:execute():213] 500 response executing GraphQL.
2022-11-28 18:11:50,574 ERROR   SenderThread:23157 [internal_api.py:execute():214] {"errors":[{"message":"internal database error","path":["upsertBucket"]}],"data":{"upsertBucket":null}}
2022-11-28 18:11:51,958 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:11:58,072 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: poll_exit
2022-11-28 18:12:03,075 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:12:08,076 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:12:08,380 ERROR   SenderThread:23157 [internal_api.py:execute():213] 500 response executing GraphQL.
2022-11-28 18:12:08,382 ERROR   SenderThread:23157 [internal_api.py:execute():214] {"errors":[{"message":"internal database error","path":["upsertBucket"]}],"data":{"upsertBucket":null}}
2022-11-28 18:12:13,078 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:12:18,079 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:12:23,081 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:12:28,082 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:12:33,085 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:12:38,087 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:12:43,093 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:12:43,679 INFO    SenderThread:23157 [dir_watcher.py:__init__():216] watching files in: wandb_logs/wandb/run-20221128_181056-3ccmf6d6/files
2022-11-28 18:12:43,680 INFO    SenderThread:23157 [sender.py:_start_run_threads():928] run started: 3ccmf6d6 with start time<PHONE_NUMBER>.850641
2022-11-28 18:12:43,680 DEBUG   SenderThread:23157 [sender.py:send():303] send: exit
2022-11-28 18:12:43,681 INFO    SenderThread:23157 [sender.py:send_exit():442] handling exit code: 1
2022-11-28 18:12:43,682 INFO    SenderThread:23157 [sender.py:send_exit():444] handling runtime: 0
2022-11-28 18:12:43,686 INFO    SenderThread:23157 [sender.py:_save_file():1171] saving file wandb-summary.json with policy end
2022-11-28 18:12:43,687 INFO    SenderThread:23157 [sender.py:send_exit():450] send defer
2022-11-28 18:12:43,688 DEBUG   SenderThread:23157 [sender.py:send_request():317] send_request: poll_exit
2022-11-28 18:12:43,688 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: defer
2022-11-28 18:12:43,690 INFO    HandlerThread:23157 [handler.py:handle_request_defer():162] handle defer: 0
2022-11-28 18:12:43,690 DEBUG   SenderThread:23157 [sender.py:send_request():317] send_request: defer
2022-11-28 18:12:43,691 INFO    SenderThread:23157 [sender.py:send_request_defer():459] handle sender defer: 0
2022-11-28 18:12:43,692 INFO    SenderThread:23157 [sender.py:transition_state():463] send defer: 1
2022-11-28 18:12:43,693 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: defer
2022-11-28 18:12:43,693 INFO    HandlerThread:23157 [handler.py:handle_request_defer():162] handle defer: 1
2022-11-28 18:12:43,694 DEBUG   SenderThread:23157 [sender.py:send_request():317] send_request: defer
2022-11-28 18:12:43,695 INFO    SenderThread:23157 [sender.py:send_request_defer():459] handle sender defer: 1
2022-11-28 18:12:43,696 INFO    SenderThread:23157 [sender.py:transition_state():463] send defer: 2
2022-11-28 18:12:43,697 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: defer
2022-11-28 18:12:43,697 INFO    HandlerThread:23157 [handler.py:handle_request_defer():162] handle defer: 2
2022-11-28 18:12:43,698 DEBUG   SenderThread:23157 [sender.py:send_request():317] send_request: defer
2022-11-28 18:12:43,699 INFO    SenderThread:23157 [sender.py:send_request_defer():459] handle sender defer: 2
2022-11-28 18:12:43,700 INFO    SenderThread:23157 [sender.py:transition_state():463] send defer: 3
2022-11-28 18:12:43,701 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: defer
2022-11-28 18:12:43,701 INFO    HandlerThread:23157 [handler.py:handle_request_defer():162] handle defer: 3
2022-11-28 18:12:43,702 DEBUG   SenderThread:23157 [sender.py:send_request():317] send_request: defer
2022-11-28 18:12:43,703 INFO    SenderThread:23157 [sender.py:send_request_defer():459] handle sender defer: 3
2022-11-28 18:12:43,703 INFO    SenderThread:23157 [sender.py:transition_state():463] send defer: 4
2022-11-28 18:12:43,704 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: defer
2022-11-28 18:12:43,705 INFO    HandlerThread:23157 [handler.py:handle_request_defer():162] handle defer: 4
2022-11-28 18:12:43,706 DEBUG   SenderThread:23157 [sender.py:send():303] send: summary
2022-11-28 18:12:43,709 INFO    SenderThread:23157 [sender.py:_save_file():1171] saving file wandb-summary.json with policy end
2022-11-28 18:12:43,710 DEBUG   SenderThread:23157 [sender.py:send_request():317] send_request: defer
2022-11-28 18:12:43,711 INFO    SenderThread:23157 [sender.py:send_request_defer():459] handle sender defer: 4
2022-11-28 18:12:43,712 INFO    SenderThread:23157 [sender.py:transition_state():463] send defer: 5
2022-11-28 18:12:43,712 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: defer
2022-11-28 18:12:43,713 INFO    HandlerThread:23157 [handler.py:handle_request_defer():162] handle defer: 5
2022-11-28 18:12:43,714 DEBUG   SenderThread:23157 [sender.py:send_request():317] send_request: defer
2022-11-28 18:12:43,714 INFO    SenderThread:23157 [sender.py:send_request_defer():459] handle sender defer: 5
2022-11-28 18:12:43,715 INFO    SenderThread:23157 [sender.py:transition_state():463] send defer: 6
2022-11-28 18:12:43,716 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: defer
2022-11-28 18:12:43,716 INFO    HandlerThread:23157 [handler.py:handle_request_defer():162] handle defer: 6
2022-11-28 18:12:43,717 DEBUG   SenderThread:23157 [sender.py:send_request():317] send_request: defer
2022-11-28 18:12:43,718 INFO    SenderThread:23157 [sender.py:send_request_defer():459] handle sender defer: 6
2022-11-28 18:12:43,718 INFO    SenderThread:23157 [sender.py:transition_state():463] send defer: 7
2022-11-28 18:12:43,719 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: defer
2022-11-28 18:12:43,720 INFO    HandlerThread:23157 [handler.py:handle_request_defer():162] handle defer: 7
2022-11-28 18:12:43,721 DEBUG   SenderThread:23157 [sender.py:send_request():317] send_request: defer
2022-11-28 18:12:43,722 INFO    SenderThread:23157 [sender.py:send_request_defer():459] handle sender defer: 7
2022-11-28 18:12:43,722 INFO    SenderThread:23157 [dir_watcher.py:finish():362] shutting down directory watcher
2022-11-28 18:12:44,683 INFO    SenderThread:23157 [dir_watcher.py:_on_file_created():275] file/dir created: wandb_logs/wandb/run-20221128_181056-3ccmf6d6/files/wandb-summary.json
2022-11-28 18:12:44,685 INFO    SenderThread:23157 [dir_watcher.py:finish():392] scan: wandb_logs/wandb/run-20221128_181056-3ccmf6d6/files
2022-11-28 18:12:44,687 INFO    SenderThread:23157 [dir_watcher.py:finish():406] scan save: wandb_logs/wandb/run-20221128_181056-3ccmf6d6/files/config.yaml config.yaml
2022-11-28 18:12:44,688 INFO    SenderThread:23157 [dir_watcher.py:finish():406] scan save: wandb_logs/wandb/run-20221128_181056-3ccmf6d6/files/wandb-summary.json wandb-summary.json
2022-11-28 18:12:44,699 INFO    SenderThread:23157 [sender.py:transition_state():463] send defer: 8
2022-11-28 18:12:44,700 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: defer
2022-11-28 18:12:44,706 INFO    HandlerThread:23157 [handler.py:handle_request_defer():162] handle defer: 8
2022-11-28 18:12:44,709 DEBUG   SenderThread:23157 [sender.py:send_request():317] send_request: defer
2022-11-28 18:12:44,709 INFO    SenderThread:23157 [sender.py:send_request_defer():459] handle sender defer: 8
2022-11-28 18:12:44,710 INFO    SenderThread:23157 [file_pusher.py:finish():168] shutting down file pusher
2022-11-28 18:12:45,344 INFO    Thread-15 :23157 [upload_job.py:push():143] Uploaded file wandb_logs/wandb/run-20221128_181056-3ccmf6d6/files/config.yaml
2022-11-28 18:12:45,369 INFO    Thread-16 :23157 [upload_job.py:push():143] Uploaded file wandb_logs/wandb/run-20221128_181056-3ccmf6d6/files/wandb-summary.json
2022-11-28 18:12:45,571 INFO    Thread-12 :23157 [sender.py:transition_state():463] send defer: 9
2022-11-28 18:12:45,572 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: defer
2022-11-28 18:12:45,573 INFO    HandlerThread:23157 [handler.py:handle_request_defer():162] handle defer: 9
2022-11-28 18:12:45,574 DEBUG   SenderThread:23157 [sender.py:send_request():317] send_request: defer
2022-11-28 18:12:45,575 INFO    SenderThread:23157 [sender.py:send_request_defer():459] handle sender defer: 9
2022-11-28 18:12:45,575 INFO    SenderThread:23157 [file_pusher.py:join():173] waiting for file pusher
2022-11-28 18:12:45,576 INFO    SenderThread:23157 [sender.py:transition_state():463] send defer: 10
2022-11-28 18:12:45,577 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: defer
2022-11-28 18:12:45,578 INFO    HandlerThread:23157 [handler.py:handle_request_defer():162] handle defer: 10
2022-11-28 18:12:45,579 DEBUG   SenderThread:23157 [sender.py:send_request():317] send_request: defer
2022-11-28 18:12:45,579 INFO    SenderThread:23157 [sender.py:send_request_defer():459] handle sender defer: 10
2022-11-28 18:12:46,087 INFO    SenderThread:23157 [sender.py:transition_state():463] send defer: 11
2022-11-28 18:12:46,088 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: defer
2022-11-28 18:12:46,089 INFO    HandlerThread:23157 [handler.py:handle_request_defer():162] handle defer: 11
2022-11-28 18:12:46,090 DEBUG   SenderThread:23157 [sender.py:send_request():317] send_request: defer
2022-11-28 18:12:46,091 INFO    SenderThread:23157 [sender.py:send_request_defer():459] handle sender defer: 11
2022-11-28 18:12:46,091 INFO    SenderThread:23157 [sender.py:transition_state():463] send defer: 12
2022-11-28 18:12:46,092 DEBUG   SenderThread:23157 [sender.py:send():303] send: final
2022-11-28 18:12:46,093 DEBUG   SenderThread:23157 [sender.py:send():303] send: footer
2022-11-28 18:12:46,092 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: defer
2022-11-28 18:12:46,094 INFO    HandlerThread:23157 [handler.py:handle_request_defer():162] handle defer: 12
2022-11-28 18:12:46,095 DEBUG   SenderThread:23157 [sender.py:send_request():317] send_request: defer
2022-11-28 18:12:46,095 INFO    SenderThread:23157 [sender.py:send_request_defer():459] handle sender defer: 12
2022-11-28 18:12:46,097 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: poll_exit
2022-11-28 18:12:46,098 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: server_info
2022-11-28 18:12:46,099 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: get_summary
2022-11-28 18:12:46,099 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: sampled_history
2022-11-28 18:12:46,098 DEBUG   SenderThread:23157 [sender.py:send_request():317] send_request: poll_exit
2022-11-28 18:12:46,101 DEBUG   SenderThread:23157 [sender.py:send_request():317] send_request: server_info
2022-11-28 18:12:46,496 INFO    MainThread:23157 [wandb_run.py:_footer_history_summary_info():3377] rendering history
2022-11-28 18:12:46,497 INFO    MainThread:23157 [wandb_run.py:_footer_history_summary_info():3409] rendering summary
2022-11-28 18:12:46,498 INFO    MainThread:23157 [wandb_run.py:_footer_sync_info():3333] logging synced files
2022-11-28 18:12:46,499 DEBUG   HandlerThread:23157 [handler.py:handle_request():139] handle_request: shutdown
2022-11-28 18:12:46,500 INFO    HandlerThread:23157 [handler.py:finish():814] shutting down handler
2022-11-28 18:12:47,093 INFO    WriterThread:23157 [datastore.py:close():279] close: wandb_logs/wandb/run-20221128_181056-3ccmf6d6/run-3ccmf6d6.wandb
2022-11-28 18:12:47,497 INFO    SenderThread:23157 [sender.py:finish():1331] shutting down sender
2022-11-28 18:12:47,498 INFO    SenderThread:23157 [file_pusher.py:finish():168] shutting down file pusher
2022-11-28 18:12:47,499 INFO    SenderThread:23157 [file_pusher.py:join():173] waiting for file pusher
2022-11-28 18:12:47,504 INFO    MainThread:23157 [internal.py:handle_exit():78] Internal process exited

For the wandb session run-20221128_181056-1l023j3h (which succeeded), debug.log is
2022-11-28 18:10:56,783 INFO    MainThread:22890 [wandb_setup.py:_flush():68] Configure stats pid to 22890
2022-11-28 18:10:56,789 INFO    MainThread:22890 [wandb_setup.py:_flush():68] Loading settings from /mnt/beegfs/home/sun/.config/wandb/settings
2022-11-28 18:10:56,790 INFO    MainThread:22890 [wandb_setup.py:_flush():68] Loading settings from /mnt/beegfs/home/sun/MoE_private/exp_20220804_github/wandb/settings
2022-11-28 18:10:56,790 INFO    MainThread:22890 [wandb_setup.py:_flush():68] Loading settings from environment variables: {'_require_service': 'True'}
2022-11-28 18:10:56,790 INFO    MainThread:22890 [wandb_setup.py:_flush():68] Inferring run settings from compute environment: {'program_relpath': 'exp_20220804_github/run.py', 'program': 'run.py'}
2022-11-28 18:10:56,791 INFO    MainThread:22890 [wandb_init.py:_log_setup():476] Logging user logs to wandb_logs/wandb/run-20221128_181056-1l023j3h/logs/debug.log
2022-11-28 18:10:56,792 INFO    MainThread:22890 [wandb_init.py:_log_setup():477] Logging internal logs to wandb_logs/wandb/run-20221128_181056-1l023j3h/logs/debug-internal.log
2022-11-28 18:10:56,793 INFO    MainThread:22890 [wandb_init.py:init():516] calling init triggers
2022-11-28 18:10:56,793 INFO    MainThread:22890 [wandb_init.py:init():519] wandb.init called with sweep_config: {}
config: {'wandb_project_name': 'new_RRR_CPU_time_benchmark_nb_branch_LogScale', 'random_seed': 5, 'epochs': 5, 'batch_size': 32, 'num_workers': 0, 'lr': 0.001, 'weight_decay': 0.01, 'lr_scheduler': 'none', 'optimizer': 'AdamW', 'loss': 'cross_entropy', 'img_size': 128, 'no_wandb': False, 'do_validation': False, 'dataset': 'MetaAlbumMicro_BCT_Micro', 'train_examples_per_class': 20, 'val_examples_per_class': 0, 'test_examples_per_class': 20, 'divergence_idx': 15, 'nb_branch': 16, 'split_pretrained_weights': False, 'warm_stem_epochs': 0, 'nb_blocks': [1, 1, 1, 1], 'pretrained_model': 'resnet152', 'ConcatenatedModel_train_layers': 'freeze_stem_train_branch', 'ema_epochs': None, 'run_baseline': False, 'baseline_model': 'resnet152', 'train_layers': 'only_last', 'run_FGE': False, 'fge_lr_1': 0.001, 'fge_lr_2': 0.0005, 'dropout_2d_replace_branch': 1000, 'use_wandb': True}
2022-11-28 18:10:56,794 INFO    MainThread:22890 [wandb_init.py:init():569] starting backend
2022-11-28 18:10:56,795 INFO    MainThread:22890 [wandb_init.py:init():573] setting up manager
2022-11-28 18:10:56,803 INFO    MainThread:22890 [backend.py:_multiprocessing_setup():102] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
2022-11-28 18:10:56,819 INFO    MainThread:22890 [wandb_init.py:init():580] backend started and connected
2022-11-28 18:10:56,833 INFO    MainThread:22890 [wandb_init.py:init():658] updated telemetry
2022-11-28 18:10:56,873 INFO    MainThread:22890 [wandb_init.py:init():693] communicating run to backend with 60 second timeout
2022-11-28 18:11:50,798 INFO    MainThread:22890 [wandb_run.py:_on_init():2000] communicating current version
2022-11-28 18:11:50,911 INFO    MainThread:22890 [wandb_run.py:_on_init():2004] got version response
2022-11-28 18:11:50,912 INFO    MainThread:22890 [wandb_init.py:init():728] starting run threads in backend
2022-11-28 18:11:55,627 INFO    MainThread:22890 [wandb_run.py:_console_start():1980] atexit reg
2022-11-28 18:11:55,628 INFO    MainThread:22890 [wandb_run.py:_redirect():1838] redirect: SettingsConsole.WRAP_RAW
2022-11-28 18:11:55,632 INFO    MainThread:22890 [wandb_run.py:_redirect():1903] Wrapping output streams.
2022-11-28 18:11:55,633 INFO    MainThread:22890 [wandb_run.py:_redirect():1925] Redirects installed.
2022-11-28 18:11:55,635 INFO    MainThread:22890 [wandb_init.py:init():765] run started, returning control to user process
2022-11-28 18:24:51,688 WARNING MsgRouterThr:22890 [router.py:message_loop():77] message_loop has been closed

For the wandb session run-20221128_181056-1l023j3h (which succeeded), debug-internal.log is
2022-11-28 18:10:56,814 INFO    StreamThr :23158 [internal.py:wandb_internal():88] W&B internal server running at pid: 23158, started at: 2022-11-28 18:10:56.812859
2022-11-28 18:10:56,818 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: status
2022-11-28 18:10:56,831 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: status
2022-11-28 18:10:56,835 DEBUG   SenderThread:23158 [sender.py:send():303] send: header
2022-11-28 18:10:56,835 INFO    WriterThread:23158 [datastore.py:open_for_write():75] open: wandb_logs/wandb/run-20221128_181056-1l023j3h/run-1l023j3h.wandb
2022-11-28 18:10:56,877 DEBUG   SenderThread:23158 [sender.py:send():303] send: run
2022-11-28 18:11:01,893 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:11:06,895 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:11:11,897 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:11:16,899 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:11:21,901 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:11:26,902 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:11:30,785 ERROR   SenderThread:23158 [retry.py:__call__():154] Retry attempt failed:
Traceback (most recent call last):
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/http/client.py", line 1347, in getresponse
    response.begin()
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/urllib3/util/retry.py", line 410, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/urllib3/connectionpool.py", line 428, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/urllib3/connectionpool.py", line 335, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='api.wandb.ai', port=443): Read timed out. (read timeout=10)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/wandb/sdk/lib/retry.py", line 113, in __call__
    result = self._call_fn(*args, **kwargs)
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/wandb/sdk/internal/internal_api.py", line 210, in execute
    return self.client.execute(*args, **kwargs)  # type: ignore
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py", line 52, in execute
    result = self._get_result(document, *args, **kwargs)
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py", line 60, in _get_result
    return self.transport.execute(document, *args, **kwargs)
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/transport/requests.py", line 38, in execute
    request = requests.post(self.url, **post_args)
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/mnt/beegfs/home/sun/miniconda3/lib/python3.8/site-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.wandb.ai', port=443): Read timed out. (read timeout=10)
2022-11-28 18:11:31,904 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:11:36,910 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:11:41,869 ERROR   SenderThread:23158 [internal_api.py:execute():213] 500 response executing GraphQL.
2022-11-28 18:11:41,870 ERROR   SenderThread:23158 [internal_api.py:execute():214] {"errors":[{"message":"internal database error","path":["upsertBucket"]}],"data":{"upsertBucket":null}}
2022-11-28 18:11:41,919 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:11:46,933 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: keepalive
2022-11-28 18:11:50,799 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: check_version
2022-11-28 18:11:50,812 INFO    SenderThread:23158 [dir_watcher.py:__init__():216] watching files in: wandb_logs/wandb/run-20221128_181056-1l023j3h/files
2022-11-28 18:11:50,812 INFO    SenderThread:23158 [sender.py:_start_run_threads():928] run started: 1l023j3h with start time<PHONE_NUMBER>.820478
2022-11-28 18:11:50,813 DEBUG   SenderThread:23158 [sender.py:send():303] send: summary
2022-11-28 18:11:50,817 INFO    SenderThread:23158 [sender.py:_save_file():1171] saving file wandb-summary.json with policy end
2022-11-28 18:11:50,818 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: check_version
2022-11-28 18:11:50,940 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: run_start
2022-11-28 18:11:50,946 DEBUG   HandlerThread:23158 [system_info.py:__init__():31] System info init
2022-11-28 18:11:50,946 DEBUG   HandlerThread:23158 [system_info.py:__init__():46] System info init done
2022-11-28 18:11:50,947 INFO    HandlerThread:23158 [system_monitor.py:start():150] Starting system monitor
2022-11-28 18:11:50,947 INFO    SystemMonitor:23158 [system_monitor.py:_start():116] Starting system asset monitoring threads
2022-11-28 18:11:50,947 INFO    HandlerThread:23158 [system_monitor.py:probe():168] Collecting system info
2022-11-28 18:11:50,948 INFO    SystemMonitor:23158 [interfaces.py:start():168] Started cpu
2022-11-28 18:11:50,961 INFO    SystemMonitor:23158 [interfaces.py:start():168] Started disk
2022-11-28 18:11:50,962 INFO    SystemMonitor:23158 [interfaces.py:start():168] Started gpu
2022-11-28 18:11:50,962 INFO    SystemMonitor:23158 [interfaces.py:start():168] Started memory
2022-11-28 18:11:50,963 INFO    SystemMonitor:23158 [interfaces.py:start():168] Started network
2022-11-28 18:11:50,978 DEBUG   HandlerThread:23158 [system_info.py:probe():195] Probing system
2022-11-28 18:11:50,998 DEBUG   HandlerThread:23158 [system_info.py:_probe_git():180] Probing git
2022-11-28 18:11:51,041 DEBUG   HandlerThread:23158 [system_info.py:_probe_git():188] Probing git done
2022-11-28 18:11:51,042 DEBUG   HandlerThread:23158 [system_info.py:probe():241] Probing system done
2022-11-28 18:11:51,042 DEBUG   HandlerThread:23158 [system_monitor.py:probe():177] {'os': 'Linux-3.10.0-1160.11.1.el7.x86_64-x86_64-with-glibc2.10', 'python': '3.8.5', 'heartbeatAt': '2022-11-28T17:11:50.991692', 'startedAt': '2022-11-28T17:10:56.488005', 'docker': None, 'cuda': None, 'args': ('--wandb_project_name', 'new_RRR_CPU_time_benchmark_nb_branch_LogScale', '--nb_blocks', '1,1,1,1', '--nb_branch', '16', '--ConcatenatedModel_train_layers', 'freeze_stem_train_branch', '--epochs', '5', '--dataset', 'MetaAlbumMicro_BCT_Micro', '--random_seed', '1', '--pretrained_model', 'resnet152', '--divergence_idx', '15', '--warm_stem_epochs', '0', '--img_size', '128', '--loss', 'cross_entropy', '--weight_decay', '0.01', '--num_workers', '0', '--lr', '0.001', '--batch_size', '32', '--optimizer', 'AdamW'), 'state': 'running', 'program': 'run.py', 'codePath': 'exp_20220804_github/run.py', 'git': {'remote': 'https://github.com/SunHaozhe/MoE_private.git', 'commit': '1698e73f6c68ddd10f3478aa399ffe6b4cf87e7b'}, 'email': None, 'root': '/mnt/beegfs/home/sun/MoE_private', 'host': 'n51.lab-ia.local', 'username': 'sun', 'executable': '/mnt/beegfs/home/sun/miniconda3/bin/python', 'cpu_count': 28, 'cpu_count_logical': 56, 'cpu_freq': {'current': 2200.0, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}, {'current': 2200.0, 'min': 0.0, 'max': 0.0}], 'disk': {'total':<PHONE_NUMBER>5219727, 'used': 38.751991271972656}, 'gpu': 'Tesla V100-PCIE-32GB', 'gpu_count': 3, 'gpu_devices': [{'name': 'Tesla V100-PCIE-32GB', 'memory_total':<PHONE_NUMBER>8}, {'name': 'Tesla V100-PCIE-32GB', 'memory_total':<PHONE_NUMBER>8}, {'name': 'Tesla V100-PCIE-32GB', 'memory_total':<PHONE_NUMBER>8}], 'memory': {'total':<PHONE_NUMBER>3188477}}
2022-11-28 18:11:51,043 INFO    HandlerThread:23158 [system_monitor.py:probe():178] Finished collecting system info
2022-11-28 18:11:51,043 INFO    HandlerThread:23158 [system_monitor.py:probe():181] Publishing system info
2022-11-28 18:11:51,043 DEBUG   HandlerThread:23158 [system_info.py:_save_pip():51] Saving list of pip packages installed into the current environment
2022-11-28 18:11:51,048 DEBUG   HandlerThread:23158 [system_info.py:_save_pip():67] Saving pip packages done
2022-11-28 18:11:51,048 DEBUG   HandlerThread:23158 [system_info.py:_save_conda():74] Saving list of conda packages installed into the current environment
2022-11-28 18:11:51,815 INFO    Thread-13 :23158 [dir_watcher.py:_on_file_created():275] file/dir created: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/requirements.txt
2022-11-28 18:11:51,816 INFO    Thread-13 :23158 [dir_watcher.py:_on_file_created():275] file/dir created: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/conda-environment.yaml
2022-11-28 18:11:51,817 INFO    Thread-13 :23158 [dir_watcher.py:_on_file_created():275] file/dir created: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/wandb-summary.json
2022-11-28 18:11:55,422 DEBUG   HandlerThread:23158 [system_info.py:_save_conda():86] Saving conda packages done
2022-11-28 18:11:55,428 INFO    HandlerThread:23158 [system_monitor.py:probe():183] Finished publishing system info
2022-11-28 18:11:55,613 DEBUG   SenderThread:23158 [sender.py:send():303] send: files
2022-11-28 18:11:55,614 INFO    SenderThread:23158 [sender.py:_save_file():1171] saving file wandb-metadata.json with policy now
2022-11-28 18:11:55,627 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:11:55,628 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:11:55,824 INFO    Thread-13 :23158 [dir_watcher.py:_on_file_modified():292] file/dir modified: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/conda-environment.yaml
2022-11-28 18:11:55,825 INFO    Thread-13 :23158 [dir_watcher.py:_on_file_created():275] file/dir created: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/wandb-metadata.json
2022-11-28 18:11:55,831 DEBUG   SenderThread:23158 [sender.py:send():303] send: telemetry
2022-11-28 18:11:55,834 DEBUG   SenderThread:23158 [sender.py:send():303] send: summary
2022-11-28 18:11:55,840 INFO    SenderThread:23158 [sender.py:_save_file():1171] saving file wandb-summary.json with policy end
2022-11-28 18:11:55,841 DEBUG   SenderThread:23158 [sender.py:send():303] send: summary
2022-11-28 18:11:55,843 INFO    SenderThread:23158 [sender.py:_save_file():1171] saving file wandb-summary.json with policy end
2022-11-28 18:11:55,844 DEBUG   SenderThread:23158 [sender.py:send():303] send: summary
2022-11-28 18:11:55,846 INFO    SenderThread:23158 [sender.py:_save_file():1171] saving file wandb-summary.json with policy end
2022-11-28 18:11:55,847 DEBUG   SenderThread:23158 [sender.py:send():303] send: summary
2022-11-28 18:11:55,849 INFO    SenderThread:23158 [sender.py:_save_file():1171] saving file wandb-summary.json with policy end
2022-11-28 18:11:56,174 INFO    Thread-16 :23158 [upload_job.py:push():143] Uploaded file /tmp/tmprb0df5c7wandb/3er5i3q1-wandb-metadata.json
2022-11-28 18:11:56,826 INFO    Thread-13 :23158 [dir_watcher.py:_on_file_modified():292] file/dir modified: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/wandb-summary.json
2022-11-28 18:11:56,827 INFO    Thread-13 :23158 [dir_watcher.py:_on_file_created():275] file/dir created: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/output.log
2022-11-28 18:11:57,497 DEBUG   SenderThread:23158 [sender.py:send():303] send: summary
2022-11-28 18:11:57,499 INFO    SenderThread:23158 [sender.py:_save_file():1171] saving file wandb-summary.json with policy end
2022-11-28 18:11:57,828 INFO    Thread-13 :23158 [dir_watcher.py:_on_file_modified():292] file/dir modified: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/wandb-summary.json
2022-11-28 18:11:58,834 INFO    Thread-13 :23158 [dir_watcher.py:_on_file_modified():292] file/dir modified: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/output.log
2022-11-28 18:12:01,171 DEBUG   SenderThread:23158 [sender.py:send():303] send: summary
2022-11-28 18:12:01,174 INFO    SenderThread:23158 [sender.py:_save_file():1171] saving file wandb-summary.json with policy end
2022-11-28 18:12:01,175 DEBUG   SenderThread:23158 [sender.py:send():303] send: summary
2022-11-28 18:12:01,178 INFO    SenderThread:23158 [sender.py:_save_file():1171] saving file wandb-summary.json with policy end
2022-11-28 18:12:01,179 DEBUG   SenderThread:23158 [sender.py:send():303] send: summary
2022-11-28 18:12:01,182 INFO    SenderThread:23158 [sender.py:_save_file():1171] saving file wandb-summary.json with policy end
2022-11-28 18:12:01,183 DEBUG   SenderThread:23158 [sender.py:send():303] send: summary
2022-11-28 18:12:01,186 INFO    SenderThread:23158 [sender.py:_save_file():1171] saving file wandb-summary.json with policy end
2022-11-28 18:12:01,840 INFO    Thread-13 :23158 [dir_watcher.py:_on_file_modified():292] file/dir modified: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/wandb-summary.json
2022-11-28 18:12:02,843 INFO    Thread-13 :23158 [dir_watcher.py:_on_file_modified():292] file/dir modified: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/output.log
2022-11-28 18:12:10,835 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:12:10,836 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:12:21,889 INFO    Thread-13 :23158 [dir_watcher.py:_on_file_modified():292] file/dir modified: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/config.yaml
2022-11-28 18:12:26,190 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:12:26,191 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:12:41,383 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:12:41,384 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:12:50,964 DEBUG   SystemMonitor:23158 [system_monitor.py:_start():130] Starting system metrics aggregation loop
2022-11-28 18:12:50,967 DEBUG   SenderThread:23158 [sender.py:send():303] send: stats
2022-11-28 18:12:56,575 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:12:56,576 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:13:11,774 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:13:11,775 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:13:20,968 DEBUG   SenderThread:23158 [sender.py:send():303] send: stats
2022-11-28 18:13:26,968 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:13:26,969 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:13:42,164 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:13:42,164 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:13:50,969 DEBUG   SenderThread:23158 [sender.py:send():303] send: stats
2022-11-28 18:13:57,353 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:13:57,354 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:14:12,550 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:14:12,551 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:14:20,970 DEBUG   SenderThread:23158 [sender.py:send():303] send: stats
2022-11-28 18:14:27,743 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:14:27,743 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:14:31,139 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: partial_history
2022-11-28 18:14:31,141 DEBUG   SenderThread:23158 [sender.py:send():303] send: history
2022-11-28 18:14:31,141 DEBUG   SenderThread:23158 [sender.py:send():303] send: summary
2022-11-28 18:14:31,145 INFO    SenderThread:23158 [sender.py:_save_file():1171] saving file wandb-summary.json with policy end
2022-11-28 18:14:31,204 INFO    Thread-13 :23158 [dir_watcher.py:_on_file_modified():292] file/dir modified: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/wandb-summary.json
2022-11-28 18:14:32,207 INFO    Thread-13 :23158 [dir_watcher.py:_on_file_modified():292] file/dir modified: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/output.log
2022-11-28 18:14:42,967 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:14:42,968 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:14:50,972 DEBUG   SenderThread:23158 [sender.py:send():303] send: stats
2022-11-28 18:14:58,162 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:14:58,163 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:15:13,371 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:15:13,372 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:15:20,973 DEBUG   SenderThread:23158 [sender.py:send():303] send: stats
2022-11-28 18:15:28,558 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:15:28,559 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:15:43,743 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:15:43,743 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:15:50,975 DEBUG   SenderThread:23158 [sender.py:send():303] send: stats
2022-11-28 18:15:58,943 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:15:58,944 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:16:14,145 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:16:14,146 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:16:20,976 DEBUG   SenderThread:23158 [sender.py:send():303] send: stats
2022-11-28 18:16:29,340 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:16:29,341 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:16:45,114 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:16:45,115 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:16:50,977 DEBUG   SenderThread:23158 [sender.py:send():303] send: stats
2022-11-28 18:17:00,160 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: partial_history
2022-11-28 18:17:00,161 DEBUG   SenderThread:23158 [sender.py:send():303] send: history
2022-11-28 18:17:00,162 DEBUG   SenderThread:23158 [sender.py:send():303] send: summary
2022-11-28 18:17:00,166 INFO    SenderThread:23158 [sender.py:_save_file():1171] saving file wandb-summary.json with policy end
2022-11-28 18:17:00,314 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:17:00,315 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:17:00,572 INFO    Thread-13 :23158 [dir_watcher.py:_on_file_modified():292] file/dir modified: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/wandb-summary.json
2022-11-28 18:17:02,577 INFO    Thread-13 :23158 [dir_watcher.py:_on_file_modified():292] file/dir modified: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/output.log
2022-11-28 18:17:15,521 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:17:15,522 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:17:20,979 DEBUG   SenderThread:23158 [sender.py:send():303] send: stats
2022-11-28 18:17:30,716 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:17:30,718 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:17:45,909 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:17:45,910 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:17:50,980 DEBUG   SenderThread:23158 [sender.py:send():303] send: stats
2022-11-28 18:18:01,118 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:18:01,119 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:18:16,314 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:18:16,315 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:18:20,981 DEBUG   SenderThread:23158 [sender.py:send():303] send: stats
2022-11-28 18:18:31,508 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:18:31,509 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:18:46,931 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:18:46,932 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:18:50,982 DEBUG   SenderThread:23158 [sender.py:send():303] send: stats
2022-11-28 18:19:02,138 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:19:02,140 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:19:17,335 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:19:17,336 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:19:20,984 DEBUG   SenderThread:23158 [sender.py:send():303] send: stats
2022-11-28 18:19:30,344 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: partial_history
2022-11-28 18:19:30,345 DEBUG   SenderThread:23158 [sender.py:send():303] send: history
2022-11-28 18:19:30,346 DEBUG   SenderThread:23158 [sender.py:send():303] send: summary
2022-11-28 18:19:30,349 INFO    SenderThread:23158 [sender.py:_save_file():1171] saving file wandb-summary.json with policy end
2022-11-28 18:19:31,029 INFO    Thread-13 :23158 [dir_watcher.py:_on_file_modified():292] file/dir modified: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/wandb-summary.json
2022-11-28 18:19:32,539 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:19:32,540 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:19:33,034 INFO    Thread-13 :23158 [dir_watcher.py:_on_file_modified():292] file/dir modified: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/output.log
2022-11-28 18:19:47,735 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:19:47,736 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:19:50,985 DEBUG   SenderThread:23158 [sender.py:send():303] send: stats
2022-11-28 18:20:02,931 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:20:02,932 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:20:18,138 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:20:18,139 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:20:20,986 DEBUG   SenderThread:23158 [sender.py:send():303] send: stats
2022-11-28 18:20:33,333 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:20:33,334 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:20:48,540 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:20:48,541 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:20:50,987 DEBUG   SenderThread:23158 [sender.py:send():303] send: stats
2022-11-28 18:21:03,733 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:21:03,734 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:21:18,923 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:21:18,924 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:21:20,989 DEBUG   SenderThread:23158 [sender.py:send():303] send: stats
2022-11-28 18:21:34,141 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:21:34,142 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:21:49,346 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:21:49,347 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:21:50,990 DEBUG   SenderThread:23158 [sender.py:send():303] send: stats
2022-11-28 18:22:00,681 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: partial_history
2022-11-28 18:22:00,682 DEBUG   SenderThread:23158 [sender.py:send():303] send: history
2022-11-28 18:22:00,683 DEBUG   SenderThread:23158 [sender.py:send():303] send: summary
2022-11-28 18:22:00,686 INFO    SenderThread:23158 [sender.py:_save_file():1171] saving file wandb-summary.json with policy end
2022-11-28 18:22:01,425 INFO    Thread-13 :23158 [dir_watcher.py:_on_file_modified():292] file/dir modified: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/wandb-summary.json
2022-11-28 18:22:03,431 INFO    Thread-13 :23158 [dir_watcher.py:_on_file_modified():292] file/dir modified: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/output.log
2022-11-28 18:22:04,549 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:22:04,550 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:22:19,750 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:22:19,751 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:22:20,991 DEBUG   SenderThread:23158 [sender.py:send():303] send: stats
2022-11-28 18:22:34,947 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:22:34,948 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:22:50,137 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:22:50,138 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:22:50,992 DEBUG   SenderThread:23158 [sender.py:send():303] send: stats
2022-11-28 18:23:05,381 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:23:05,382 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:23:20,570 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:23:20,571 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:23:20,998 DEBUG   SenderThread:23158 [sender.py:send():303] send: stats
2022-11-28 18:23:35,758 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:23:35,759 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:23:50,958 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:23:50,959 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:23:51,160 DEBUG   SenderThread:23158 [sender.py:send():303] send: stats
2022-11-28 18:24:06,162 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:24:06,163 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:24:21,001 DEBUG   SenderThread:23158 [sender.py:send():303] send: stats
2022-11-28 18:24:21,348 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:24:21,349 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:24:30,149 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: partial_history
2022-11-28 18:24:30,150 DEBUG   SenderThread:23158 [sender.py:send():303] send: history
2022-11-28 18:24:30,151 DEBUG   SenderThread:23158 [sender.py:send():303] send: summary
2022-11-28 18:24:30,156 INFO    SenderThread:23158 [sender.py:_save_file():1171] saving file wandb-summary.json with policy end
2022-11-28 18:24:30,852 INFO    Thread-13 :23158 [dir_watcher.py:_on_file_modified():292] file/dir modified: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/wandb-summary.json
2022-11-28 18:24:30,853 INFO    Thread-13 :23158 [dir_watcher.py:_on_file_modified():292] file/dir modified: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/output.log
2022-11-28 18:24:36,566 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: stop_status
2022-11-28 18:24:36,567 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: stop_status
2022-11-28 18:24:46,404 DEBUG   SenderThread:23158 [sender.py:send():303] send: summary
2022-11-28 18:24:46,407 INFO    SenderThread:23158 [sender.py:_save_file():1171] saving file wandb-summary.json with policy end
2022-11-28 18:24:46,408 DEBUG   SenderThread:23158 [sender.py:send():303] send: summary
2022-11-28 18:24:46,411 INFO    SenderThread:23158 [sender.py:_save_file():1171] saving file wandb-summary.json with policy end
2022-11-28 18:24:46,412 DEBUG   SenderThread:23158 [sender.py:send():303] send: summary
2022-11-28 18:24:46,414 INFO    SenderThread:23158 [sender.py:_save_file():1171] saving file wandb-summary.json with policy end
2022-11-28 18:24:46,415 DEBUG   SenderThread:23158 [sender.py:send():303] send: summary
2022-11-28 18:24:46,418 INFO    SenderThread:23158 [sender.py:_save_file():1171] saving file wandb-summary.json with policy end
2022-11-28 18:24:46,418 DEBUG   SenderThread:23158 [sender.py:send():303] send: exit
2022-11-28 18:24:46,419 INFO    SenderThread:23158 [sender.py:send_exit():442] handling exit code: 0
2022-11-28 18:24:46,420 INFO    SenderThread:23158 [sender.py:send_exit():444] handling runtime: 775
2022-11-28 18:24:46,422 INFO    SenderThread:23158 [sender.py:_save_file():1171] saving file wandb-summary.json with policy end
2022-11-28 18:24:46,423 INFO    SenderThread:23158 [sender.py:send_exit():450] send defer
2022-11-28 18:24:46,424 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: defer
2022-11-28 18:24:46,424 INFO    HandlerThread:23158 [handler.py:handle_request_defer():162] handle defer: 0
2022-11-28 18:24:46,425 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: defer
2022-11-28 18:24:46,425 INFO    SenderThread:23158 [sender.py:send_request_defer():459] handle sender defer: 0
2022-11-28 18:24:46,425 INFO    SenderThread:23158 [sender.py:transition_state():463] send defer: 1
2022-11-28 18:24:46,426 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: defer
2022-11-28 18:24:46,426 INFO    HandlerThread:23158 [handler.py:handle_request_defer():162] handle defer: 1
2022-11-28 18:24:46,427 INFO    HandlerThread:23158 [system_monitor.py:finish():160] Stopping system monitor
2022-11-28 18:24:46,429 DEBUG   SystemMonitor:23158 [system_monitor.py:_start():137] Finished system metrics aggregation loop
2022-11-28 18:24:46,434 DEBUG   SystemMonitor:23158 [system_monitor.py:_start():141] Publishing last batch of metrics
2022-11-28 18:24:46,436 INFO    HandlerThread:23158 [interfaces.py:finish():173] Joined cpu
2022-11-28 18:24:46,449 INFO    HandlerThread:23158 [interfaces.py:finish():173] Joined disk
2022-11-28 18:24:46,882 INFO    HandlerThread:23158 [interfaces.py:finish():173] Joined gpu
2022-11-28 18:24:46,883 INFO    HandlerThread:23158 [interfaces.py:finish():173] Joined memory
2022-11-28 18:24:46,883 INFO    HandlerThread:23158 [interfaces.py:finish():173] Joined network
2022-11-28 18:24:46,884 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: defer
2022-11-28 18:24:46,885 INFO    SenderThread:23158 [sender.py:send_request_defer():459] handle sender defer: 1
2022-11-28 18:24:46,885 INFO    SenderThread:23158 [sender.py:transition_state():463] send defer: 2
2022-11-28 18:24:46,885 DEBUG   SenderThread:23158 [sender.py:send():303] send: stats
2022-11-28 18:24:46,886 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: defer
2022-11-28 18:24:46,887 INFO    HandlerThread:23158 [handler.py:handle_request_defer():162] handle defer: 2
2022-11-28 18:24:46,887 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: defer
2022-11-28 18:24:46,888 INFO    SenderThread:23158 [sender.py:send_request_defer():459] handle sender defer: 2
2022-11-28 18:24:46,888 INFO    SenderThread:23158 [sender.py:transition_state():463] send defer: 3
2022-11-28 18:24:46,889 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: defer
2022-11-28 18:24:46,889 INFO    HandlerThread:23158 [handler.py:handle_request_defer():162] handle defer: 3
2022-11-28 18:24:46,889 INFO    Thread-13 :23158 [dir_watcher.py:_on_file_modified():292] file/dir modified: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/wandb-summary.json
2022-11-28 18:24:46,890 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: defer
2022-11-28 18:24:46,891 INFO    SenderThread:23158 [sender.py:send_request_defer():459] handle sender defer: 3
2022-11-28 18:24:46,891 INFO    SenderThread:23158 [sender.py:transition_state():463] send defer: 4
2022-11-28 18:24:46,892 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: defer
2022-11-28 18:24:46,892 INFO    HandlerThread:23158 [handler.py:handle_request_defer():162] handle defer: 4
2022-11-28 18:24:46,893 DEBUG   SenderThread:23158 [sender.py:send():303] send: summary
2022-11-28 18:24:46,895 INFO    SenderThread:23158 [sender.py:_save_file():1171] saving file wandb-summary.json with policy end
2022-11-28 18:24:46,896 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: defer
2022-11-28 18:24:46,896 INFO    SenderThread:23158 [sender.py:send_request_defer():459] handle sender defer: 4
2022-11-28 18:24:46,897 INFO    SenderThread:23158 [sender.py:transition_state():463] send defer: 5
2022-11-28 18:24:46,897 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: defer
2022-11-28 18:24:46,898 INFO    HandlerThread:23158 [handler.py:handle_request_defer():162] handle defer: 5
2022-11-28 18:24:46,898 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: defer
2022-11-28 18:24:46,898 INFO    SenderThread:23158 [sender.py:send_request_defer():459] handle sender defer: 5
2022-11-28 18:24:46,899 INFO    SenderThread:23158 [sender.py:transition_state():463] send defer: 6
2022-11-28 18:24:46,899 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: defer
2022-11-28 18:24:46,900 INFO    HandlerThread:23158 [handler.py:handle_request_defer():162] handle defer: 6
2022-11-28 18:24:46,900 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: defer
2022-11-28 18:24:46,900 INFO    SenderThread:23158 [sender.py:send_request_defer():459] handle sender defer: 6
2022-11-28 18:24:47,403 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: poll_exit
2022-11-28 18:24:48,715 INFO    SenderThread:23158 [sender.py:transition_state():463] send defer: 7
2022-11-28 18:24:48,716 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: poll_exit
2022-11-28 18:24:48,717 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: defer
2022-11-28 18:24:48,719 INFO    HandlerThread:23158 [handler.py:handle_request_defer():162] handle defer: 7
2022-11-28 18:24:48,720 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: defer
2022-11-28 18:24:48,720 INFO    SenderThread:23158 [sender.py:send_request_defer():459] handle sender defer: 7
2022-11-28 18:24:48,721 INFO    SenderThread:23158 [dir_watcher.py:finish():362] shutting down directory watcher
2022-11-28 18:24:48,894 INFO    SenderThread:23158 [dir_watcher.py:_on_file_modified():292] file/dir modified: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/output.log
2022-11-28 18:24:48,896 INFO    SenderThread:23158 [dir_watcher.py:finish():392] scan: wandb_logs/wandb/run-20221128_181056-1l023j3h/files
2022-11-28 18:24:48,897 INFO    SenderThread:23158 [dir_watcher.py:finish():406] scan save: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/config.yaml config.yaml
2022-11-28 18:24:48,898 INFO    SenderThread:23158 [dir_watcher.py:finish():406] scan save: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/conda-environment.yaml conda-environment.yaml
2022-11-28 18:24:48,906 INFO    SenderThread:23158 [dir_watcher.py:finish():406] scan save: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/wandb-summary.json wandb-summary.json
2022-11-28 18:24:48,912 INFO    SenderThread:23158 [dir_watcher.py:finish():406] scan save: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/output.log output.log
2022-11-28 18:24:48,917 INFO    SenderThread:23158 [dir_watcher.py:finish():406] scan save: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/wandb-metadata.json wandb-metadata.json
2022-11-28 18:24:48,917 INFO    SenderThread:23158 [dir_watcher.py:finish():406] scan save: wandb_logs/wandb/run-20221128_181056-1l023j3h/files/requirements.txt requirements.txt
2022-11-28 18:24:48,921 INFO    SenderThread:23158 [sender.py:transition_state():463] send defer: 8
2022-11-28 18:24:48,925 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: defer
2022-11-28 18:24:48,926 INFO    HandlerThread:23158 [handler.py:handle_request_defer():162] handle defer: 8
2022-11-28 18:24:48,927 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: defer
2022-11-28 18:24:48,927 INFO    SenderThread:23158 [sender.py:send_request_defer():459] handle sender defer: 8
2022-11-28 18:24:48,928 INFO    SenderThread:23158 [file_pusher.py:finish():168] shutting down file pusher
2022-11-28 18:24:49,521 INFO    Thread-18 :23158 [upload_job.py:push():143] Uploaded file wandb_logs/wandb/run-20221128_181056-1l023j3h/files/conda-environment.yaml
2022-11-28 18:24:49,546 INFO    Thread-19 :23158 [upload_job.py:push():143] Uploaded file wandb_logs/wandb/run-20221128_181056-1l023j3h/files/wandb-summary.json
2022-11-28 18:24:49,551 INFO    Thread-20 :23158 [upload_job.py:push():143] Uploaded file wandb_logs/wandb/run-20221128_181056-1l023j3h/files/output.log
2022-11-28 18:24:49,562 INFO    Thread-21 :23158 [upload_job.py:push():143] Uploaded file wandb_logs/wandb/run-20221128_181056-1l023j3h/files/requirements.txt
2022-11-28 18:24:49,586 INFO    Thread-17 :23158 [upload_job.py:push():143] Uploaded file wandb_logs/wandb/run-20221128_181056-1l023j3h/files/config.yaml
2022-11-28 18:24:49,788 INFO    Thread-12 :23158 [sender.py:transition_state():463] send defer: 9
2022-11-28 18:24:49,789 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: defer
2022-11-28 18:24:49,790 INFO    HandlerThread:23158 [handler.py:handle_request_defer():162] handle defer: 9
2022-11-28 18:24:49,790 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: defer
2022-11-28 18:24:49,791 INFO    SenderThread:23158 [sender.py:send_request_defer():459] handle sender defer: 9
2022-11-28 18:24:49,791 INFO    SenderThread:23158 [file_pusher.py:join():173] waiting for file pusher
2022-11-28 18:24:49,792 INFO    SenderThread:23158 [sender.py:transition_state():463] send defer: 10
2022-11-28 18:24:49,793 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: defer
2022-11-28 18:24:49,793 INFO    HandlerThread:23158 [handler.py:handle_request_defer():162] handle defer: 10
2022-11-28 18:24:49,794 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: defer
2022-11-28 18:24:49,794 INFO    SenderThread:23158 [sender.py:send_request_defer():459] handle sender defer: 10
2022-11-28 18:24:50,260 INFO    SenderThread:23158 [sender.py:transition_state():463] send defer: 11
2022-11-28 18:24:50,261 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: defer
2022-11-28 18:24:50,262 INFO    HandlerThread:23158 [handler.py:handle_request_defer():162] handle defer: 11
2022-11-28 18:24:50,263 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: defer
2022-11-28 18:24:50,263 INFO    SenderThread:23158 [sender.py:send_request_defer():459] handle sender defer: 11
2022-11-28 18:24:50,264 INFO    SenderThread:23158 [sender.py:transition_state():463] send defer: 12
2022-11-28 18:24:50,265 DEBUG   SenderThread:23158 [sender.py:send():303] send: final
2022-11-28 18:24:50,266 DEBUG   SenderThread:23158 [sender.py:send():303] send: footer
2022-11-28 18:24:50,265 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: defer
2022-11-28 18:24:50,266 INFO    HandlerThread:23158 [handler.py:handle_request_defer():162] handle defer: 12
2022-11-28 18:24:50,267 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: defer
2022-11-28 18:24:50,267 INFO    SenderThread:23158 [sender.py:send_request_defer():459] handle sender defer: 12
2022-11-28 18:24:50,270 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: poll_exit
2022-11-28 18:24:50,271 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: server_info
2022-11-28 18:24:50,272 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: get_summary
2022-11-28 18:24:50,272 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: sampled_history
2022-11-28 18:24:50,271 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: poll_exit
2022-11-28 18:24:50,274 DEBUG   SenderThread:23158 [sender.py:send_request():317] send_request: server_info
2022-11-28 18:24:50,678 INFO    MainThread:23158 [wandb_run.py:_footer_history_summary_info():3377] rendering history
2022-11-28 18:24:50,680 INFO    MainThread:23158 [wandb_run.py:_footer_history_summary_info():3409] rendering summary
2022-11-28 18:24:50,681 INFO    MainThread:23158 [wandb_run.py:_footer_sync_info():3333] logging synced files
2022-11-28 18:24:50,683 DEBUG   HandlerThread:23158 [handler.py:handle_request():139] handle_request: shutdown
2022-11-28 18:24:50,683 INFO    HandlerThread:23158 [handler.py:finish():814] shutting down handler
2022-11-28 18:24:51,266 INFO    WriterThread:23158 [datastore.py:close():279] close: wandb_logs/wandb/run-20221128_181056-1l023j3h/run-1l023j3h.wandb
2022-11-28 18:24:51,679 INFO    SenderThread:23158 [sender.py:finish():1331] shutting down sender
2022-11-28 18:24:51,680 INFO    SenderThread:23158 [file_pusher.py:finish():168] shutting down file pusher
2022-11-28 18:24:51,680 INFO    SenderThread:23158 [file_pusher.py:join():173] waiting for file pusher
2022-11-28 18:24:51,689 INFO    MainThread:23158 [internal.py:handle_exit():78] Internal process exited

For the first (initial) issue, I do not understand what you mean by "Sweep agent". My wandb workflow did not involve "sweep", below I show the snippet code of how I used wandb:
import os
import wandb

project_name = "XXX"
group_name = ""
wandb_dir = "wandb_logs"
if not os.path.exists(wandb_dir):
    os.makedirs(wandb_dir)

wandb.init(config=args, project=project_name, group=group_name, dir=wandb_dir)

# in the training loop
for epoch in range(epochs):
    for X, y in PyTorch_dataloader:
        ...
        wandb.log({"epoch": epoch+1, "YYY": yyy})
        wandb.run.summary["ZZZ"] = zzz

As seen above, my only calls to wandb APIs are wandb.init, wandb.log, and wandb.run.summary, nothing else. I do not know what you meant by "sweep agent".
On the other hand, our slurm cluster does not use the environment variable CUDA_VISIBLE_DEVICES to allocate GPUs, it is instead automatically managed by slurm. For example, adding #SBATCH --gres=gpu:1 in the beginning of the bash/slurm script would ask slurm to allocate one GPU. In our slurm cluster, each node contains 3 or 4 GPUs.
In my experiments, half of my python runs used 0 GPUs (CPU only), the other half used exactly 1 GPU (automatically assigned by slurm). In both cases (with or without GPUs), one python run created two wandb sessions.
I do not know whether the information I provided here is meaningful to solve the first issue? Do you need further information?

Hi @SunHaozhe thank you for the detailed update. Can you please try to also include a line wandb.finish() at the end? it seems that you're not terminating them from the code snippet you provided above. Would this work for you?

Hi @SunHaozhe I wanted to follow up with you about this issue, did you try adding the wandb.finish() call and does it still occur for you?

Hi @thanos-wandb , sorry for the delay.
I tried adding the wandb.finish() call at the end of my python program but nothing changed, each call to python run still creates two wandb sessions (which started at exactly the same timestamp).
I think this result is not surprising because the duplicate wandb session is created at the very beginning of the python script, it does not know whether wandb.finish() will be called at the end of the python script.
In general wandb.finish() is automatically done when my script exits, no? This is written in the document.

Hi @SunHaozhe sorry for the late reply here. Do you still get this issue? and would it be possible to run a simple python script printing just some random text, to check if the issue is related to your Slurm job script that will initialize two concurrent runs? thanks!

Hi @thanos-wandb I tried running a simple python script and I found that the issue is still here. Two concurrent runs are always created for a single call of the python script.
Below I show the complete code for the python script and the slurm job script.
Python script (named "test.py"):
import os
import wandb

def parse_arguments():
    import argparse
    parser = argparse.ArgumentParser()
    args = parser.parse_args()
    return args

if __name__ == "__main__":

    args = parse_arguments()

    project_name = "test_concurrent_wandb_runs"

    group_name = "".format()

    wandb_dir = "wandb_logs"
    if not os.path.exists(wandb_dir):
        os.makedirs(wandb_dir)
    wandb.init(config=args, project=project_name,
                group=group_name, dir=wandb_dir)

    epochs = 2

    for epoch in range(epochs):
        for x in range(10):
            wandb.log({"epoch": epoch+1, "x": x})
            wandb.run.summary["x"] = x

Slurm job script:
#!/bin/bash
#SBATCH --job-name=XXX
#SBATCH --partition=all
#SBATCH --qos=default
#SBATCH --mem=16g
#SBATCH --nodes=1
#SBATCH --cpus-per-task=1
#SBATCH --open-mode=append
#SBATCH --signal=B:USR1@120
#SBATCH --requeue
#SBATCH --array=0-2
#SBATCH --output=logs/%A_%a.stdout
#SBATCH --error=logs/%A_%a.stderr

SECONDS=0

restart(){
	echo "Calling restart"
	scontrol requeue $SLURM_JOB_ID
	echo "Scheduled job for restart"
}

ignore(){
	echo "Ignored SIGTERM"
}
trap restart USR1
trap ignore TERM

date

# Main work starts

args=()

for SEED in "1" "2" "3"
do
	args+=("test.py")
done

echo "Starting python ${args[${SLURM_ARRAY_TASK_ID}]}"

srun python ${args[${SLURM_ARRAY_TASK_ID}]}

echo "End python ${args[${SLURM_ARRAY_TASK_ID}]}"

# Main work ends

DURATION=$SECONDS

echo "End of the program! $(($DURATION / 60)) minutes and $(($DURATION % 60)) seconds elapsed."

As you can see, the slurm job script called the python script "test.py" three times. However I got 6 wandb sessions in the end.
Could you please let me know how I could get rid of this concurrent-run issue?

Hi @SunHaozhe,
Apologies for the delay here. We have tried reproducing this issue on 2 separate slurm clusters with the code you have provided, without any luck. Is there a slurm.conf file that you can share which defines the properties of your slurm cluster?
This error looks to be coming from some interaction between the W&B SDK and your slurm cluster here. It would also br great if you could try running this training process on another slurm cluster and let us know if you had any luck running into the same error.
Thanks,
Ramit

Hey @SunHaozhe,
I wanted to reach out again to check if you would be able to provide some configuration details regarding your slurm cluster. We can not reproduce without the configuration and will not have a path forward to resolve this without it.
Thanks,
Ramit

Hi @SunHaozhe,
Since we have not heard back from you, we will be closing this issue for now. In case you run into this issue again and need assistance debugging this, please reach back out to us and I will be happy to continue looking into this for you.

Hi @SunHaozhe,
Apologies for the delay here. We have tried reproducing this issue on 2 separate slurm clusters with the code you have provided, without any luck. Is there a slurm.conf file that you can share which defines the properties of your slurm cluster?
This error looks to be coming from some interaction between the W&B SDK and your slurm cluster here. It would also br great if you could try running this training process on another slurm cluster and let us know if you had any luck running into the same error.
Thanks, Ramit

I also encountered the same problem, wandb.init() will lag can not complete the initialization, input as previously provided by @SunHaoZhe, but we are using the public supercomputing center can not get slurm.conf file.


[bug]: Service does not enter "Restarting" state when config is changed on a "Running" service

Prerequisites

[X] I have searched for existing issues that already report this problem.

EmbassyOS Version
<IP_ADDRESS>
Device
Laptop/Desktop
Device OS
Linux
Device OS Version
5.18
Browser
Firefox
Browser Version
102
Current Behavior
It is not obvious that a service is restarting when you reconfigure it (while in a Running state)
Expected Behavior
Enter Restarting state
Steps to Reproduce

Start Bitcoin service (for example)
Change config while running
Service will restart, but this is not indicated in the UI

Anything else?
No response

100% we need this fixed

Dupe of #958


Add this repository to hacs default

In the readme you show that this repository is in the hacs default list via the badge "hacs:default".
But in fact this repository is not in the default list and must be added via the custom url.
I think it would be easier if you add it to the default list.
When you repository is part of the default list, a new user must only search for the name and doesn't need to add the custom url.
What do you think?
P.S. adding to the default list can only be done by the owner or a big contributor

Thank you for your comment. My request is still pending.
https://github.com/home-assistant/brands/pull/2818

No Stress 😊

The validation has been done


Update provision README and files for vector store/embedder

Added examples of how to setup with openai+qdrant and grafana/vectorapi for vector provisioning
Removed unneeded provisioning for openai vector embedder (uses openai completion settings by default)
updated docker-compose and provisioning local dev to use grafana/vectorapi by default.

We want to help OSS users (or internal users testing locally) to get the PromQL query advisor running -- that'll be really hard with Qdrant! I've updated the readme to guide people better. More iterations needed still.

@edwardcqian I am merging this for now, so we can have a more up to date README for the day. We will iterate on the steps within the day.


Put files in MMPT & do file revisions properly

Put file contents in MMPT
properly revision files & contents in MMPT
add members() method to MMPT for debugging
add localOnly flag for filesystem for local testing purposes

To test:

Run quotes app locally
use<EMAIL_ADDRESS>login to Quotes & create a new account
add a couple quotes, navigate etc
wait for DNS to propagate (can take awhile)
Exit out of the original quotes app (we don't want the second page to make a browser hop, we want to ensure that the gateway has all of the needed content
open quotes app in a separate browser (or incognito window)
login to quotes app & see if content loads

Updated https://quotes.fission.app to v0.19-alpha, seems to be working smoothly now 🎉

@dholms Should we merge this? 👀

Oops, didn't mean to leave it as a draft. Yup! It's good to go whenever :ok_hand: