torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 12.72 GiB. GPU 0 has a total capacity of 14.75 GiB of which 1.07 GiB is free. Process 34044 has 13.68 GiB memory in use. Of the allocated memory 13.54 GiB is allocated by PyTorch, and 10.24 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Trial _train_tune_4003e_00012 errored after 0 iterations at 2024-05-15 18:49:56. Total running time: 2min 40s
Error file: /tmp/ray/session_2024-05-15_18-47-11_316201_352/artifacts/2024-05-15_18-47-15/_train_tune_2024-05-15_18-47-11/driver_artifacts/_train_tune_4003e_00012_12_2024-05-15_18-47-16/error.txt
(_train_tune pid=3163) /usr/local/lib/python3.10/dist-packages/ray/tune/integration/pytorch_lightning.py:198: ray.tune.integration.pytorch_lightning.TuneReportCallback is deprecated. Use ray.tune.integration.pytorch_lightning.TuneReportCheckpointCallback instead.
(_train_tune pid=3163) Seed set to 1
(_train_tune pid=3163) GPU available: True (cuda), used: True
(_train_tune pid=3163) TPU available: False, using: 0 TPU cores
(_train_tune pid=3163) IPU available: False, using: 0 IPUs
(_train_tune pid=3163) HPU available: False, using: 0 HPUs
(_train_tune pid=3163) Trainer(val_check_interval=1) was configured so validation will run after every batch.
(_train_tune pid=3163) Missing logger folder: /tmp/ray/session_2024-05-15_18-47-11_316201_352/artifacts/2024-05-15_18-47-15/_train_tune_2024-05-15_18-47-11/working_dirs/_train_tune_4003e_00013_13_2024-05-15_18-47-16/lightning_logs
(_train_tune pid=3163) 2024-05-15 18:50:05.258245: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
(_train_tune pid=3163) 2024-05-15 18:50:05.258300: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
(_train_tune pid=3163) 2024-05-15 18:50:05.259671: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
(_train_tune pid=3163) 2024-05-15 18:50:06.660238: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
(_train_tune pid=3163) LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
(_train_tune pid=3163)
(_train_tune pid=3163)   | Name            | Type          | Params
(_train_tune pid=3163) --------------------------------------------------
(_train_tune pid=3163) 0 | loss            | MAE           | 0
(_train_tune pid=3163) 1 | padder          | ConstantPad1d | 0
(_train_tune pid=3163) 2 | scaler          | TemporalNorm  | 0
(_train_tune pid=3163) 3 | hist_encoder    | LSTM          | 484 K
(_train_tune pid=3163) 4 | context_adapter | Linear        | 733 K
(_train_tune pid=3163) 5 | mlp_decoder     | MLP           | 2.4 K
(_train_tune pid=3163) --------------------------------------------------
(_train_tune pid=3163) 1.2 M     Trainable params
(_train_tune pid=3163) 0         Non-trainable params
(_train_tune pid=3163) 1.2 M     Total params
(_train_tune pid=3163) 4.880     Total estimated model params size (MB)
Sanity Checking DataLoader 0:   0%|          | 0/2 [00:00<?, ?it/s]
2024-05-15 18:50:08,865	ERROR tune_controller.py:1331 -- Trial task failed for trial _train_tune_4003e_00013
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/ray/air/execution/_internal/event_manager.py", line 110, in resolve_future
result = ray.get(future)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 2623, in get
values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 861, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(OutOfMemoryError): ray::ImplicitFunc.train() (pid=3163, ip=<IP_ADDRESS>, actor_id=52d743851cafdc7d4d83cebf01000000, repr=_train_tune)
File "/usr/local/lib/python3.10/dist-packages/ray/tune/trainable/trainable.py", line 331, in train
raise skipped from exception_cause(skipped)
File "/usr/local/lib/python3.10/dist-packages/ray/air/_internal/util.py", line 98, in run
self._ret = self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/tune/trainable/function_trainable.py", line 45, in
training_func=lambda: self._trainable_func(self.config),
File "/usr/local/lib/python3.10/dist-packages/ray/tune/trainable/function_trainable.py", line 248, in _trainable_func
output = fn()
File "/usr/local/lib/python3.10/dist-packages/ray/tune/trainable/util.py", line 130, in inner
return trainable(config, **fn_kwargs)
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_auto.py", line 209, in _train_tune
_ = self._fit_model(
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_auto.py", line 357, in _fit_model
model = model.fit(
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_recurrent.py", line 530, in fit
return self._fit(
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_model.py", line 219, in _fit
trainer.fit(model, datamodule=datamodule)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 544, in fit
call._call_and_handle_interrupt(
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/call.py", line 44, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 580, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 987, in _run
results = self._run_stage()
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1031, in _run_stage
self._run_sanity_check()
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1060, in _run_sanity_check
val_loop.run()
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/utilities.py", line 182, in _decorator
return loop_run(self, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/evaluation_loop.py", line 135, in run
self._evaluation_step(batch, batch_idx, dataloader_idx, dataloader_iter)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/evaluation_loop.py", line 396, in _evaluation_step
output = call._call_strategy_hook(trainer, hook_name, *step_args)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/call.py", line 309, in _call_strategy_hook
output = fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/strategies/strategy.py", line 412, in validation_step
return self.lightning_module.validation_step(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_recurrent.py", line 392, in validation_step
output = self(windows_batch)  # tuple([B, seq_len, H, output])
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/models/lstm.py", line 210, in forward
output = self.mlp_decoder(context)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_modules.py", line 60, in forward
return self.layers(x)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/container.py", line 217, in forward
input = module(input)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/activation.py", line 101, in forward
return F.relu(input, inplace=self.inplace)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py", line 1473, in relu
result = torch.relu(input)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 12.72 GiB. GPU 0 has a total capacity of 14.75 GiB of which 1.07 GiB is free. Process 34467 has 13.68 GiB memory in use. Of the allocated memory 13.54 GiB is allocated by PyTorch, and 10.24 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Trial _train_tune_4003e_00013 errored after 0 iterations at 2024-05-15 18:50:08. Total running time: 2min 52s
Error file: /tmp/ray/session_2024-05-15_18-47-11_316201_352/artifacts/2024-05-15_18-47-15/_train_tune_2024-05-15_18-47-11/driver_artifacts/_train_tune_4003e_00013_13_2024-05-15_18-47-16/error.txt
(_train_tune pid=3254) Seed set to 1
(_train_tune pid=3254) /usr/local/lib/python3.10/dist-packages/ray/tune/integration/pytorch_lightning.py:198: ray.tune.integration.pytorch_lightning.TuneReportCallback is deprecated. Use ray.tune.integration.pytorch_lightning.TuneReportCheckpointCallback instead.
(_train_tune pid=3254) GPU available: True (cuda), used: True
(_train_tune pid=3254) TPU available: False, using: 0 TPU cores
(_train_tune pid=3254) IPU available: False, using: 0 IPUs
(_train_tune pid=3254) HPU available: False, using: 0 HPUs
(_train_tune pid=3254) Trainer(val_check_interval=1) was configured so validation will run after every batch.
(_train_tune pid=3254) Missing logger folder: /tmp/ray/session_2024-05-15_18-47-11_316201_352/artifacts/2024-05-15_18-47-15/_train_tune_2024-05-15_18-47-11/working_dirs/_train_tune_4003e_00014_14_2024-05-15_18-47-16/lightning_logs
(_train_tune pid=3254) 2024-05-15 18:50:19.166225: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
(_train_tune pid=3254) 2024-05-15 18:50:19.166276: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
(_train_tune pid=3254) 2024-05-15 18:50:19.167749: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
(_train_tune pid=3254) 2024-05-15 18:50:20.505432: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
(_train_tune pid=3254) LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
(_train_tune pid=3254)
(_train_tune pid=3254)   | Name            | Type          | Params
(_train_tune pid=3254) --------------------------------------------------
(_train_tune pid=3254) 0 | loss            | MAE           | 0
(_train_tune pid=3254) 1 | padder          | ConstantPad1d | 0
(_train_tune pid=3254) 2 | scaler          | TemporalNorm  | 0
(_train_tune pid=3254) 3 | hist_encoder    | LSTM          | 484 K
(_train_tune pid=3254) 4 | context_adapter | Linear        | 733 K
(_train_tune pid=3254) 5 | mlp_decoder     | MLP           | 2.4 K
(_train_tune pid=3254) --------------------------------------------------
(_train_tune pid=3254) 1.2 M     Trainable params
(_train_tune pid=3254) 0         Non-trainable params
(_train_tune pid=3254) 1.2 M     Total params
(_train_tune pid=3254) 4.880     Total estimated model params size (MB)
Sanity Checking DataLoader 0:   0%|          | 0/2 [00:00<?, ?it/s]
2024-05-15 18:50:22,545	ERROR tune_controller.py:1331 -- Trial task failed for trial _train_tune_4003e_00014
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/ray/air/execution/_internal/event_manager.py", line 110, in resolve_future
result = ray.get(future)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 2623, in get
values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 861, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(OutOfMemoryError): ray::ImplicitFunc.train() (pid=3254, ip=<IP_ADDRESS>, actor_id=ecac20e4935be6c35f7019b801000000, repr=_train_tune)
File "/usr/local/lib/python3.10/dist-packages/ray/tune/trainable/trainable.py", line 331, in train
raise skipped from exception_cause(skipped)
File "/usr/local/lib/python3.10/dist-packages/ray/air/_internal/util.py", line 98, in run
self._ret = self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/tune/trainable/function_trainable.py", line 45, in
training_func=lambda: self._trainable_func(self.config),
File "/usr/local/lib/python3.10/dist-packages/ray/tune/trainable/function_trainable.py", line 248, in _trainable_func
output = fn()
File "/usr/local/lib/python3.10/dist-packages/ray/tune/trainable/util.py", line 130, in inner
return trainable(config, **fn_kwargs)
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_auto.py", line 209, in _train_tune
_ = self._fit_model(
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_auto.py", line 357, in _fit_model
model = model.fit(
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_recurrent.py", line 530, in fit
return self._fit(
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_model.py", line 219, in _fit
trainer.fit(model, datamodule=datamodule)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 544, in fit
call._call_and_handle_interrupt(
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/call.py", line 44, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 580, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 987, in _run
results = self._run_stage()
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1031, in _run_stage
self._run_sanity_check()
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1060, in _run_sanity_check
val_loop.run()
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/utilities.py", line 182, in _decorator
return loop_run(self, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/evaluation_loop.py", line 135, in run
self._evaluation_step(batch, batch_idx, dataloader_idx, dataloader_iter)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/evaluation_loop.py", line 396, in _evaluation_step
output = call._call_strategy_hook(trainer, hook_name, *step_args)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/call.py", line 309, in _call_strategy_hook
output = fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/strategies/strategy.py", line 412, in validation_step
return self.lightning_module.validation_step(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_recurrent.py", line 392, in validation_step
output = self(windows_batch)  # tuple([B, seq_len, H, output])
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/models/lstm.py", line 210, in forward
output = self.mlp_decoder(context)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_modules.py", line 60, in forward
return self.layers(x)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/container.py", line 217, in forward
input = module(input)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/activation.py", line 101, in forward
return F.relu(input, inplace=self.inplace)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py", line 1473, in relu
result = torch.relu(input)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 12.72 GiB. GPU 0 has a total capacity of 14.75 GiB of which 1.07 GiB is free. Process 35435 has 13.68 GiB memory in use. Of the allocated memory 13.54 GiB is allocated by PyTorch, and 10.24 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Trial _train_tune_4003e_00014 errored after 0 iterations at 2024-05-15 18:50:22. Total running time: 3min 6s
Error file: /tmp/ray/session_2024-05-15_18-47-11_316201_352/artifacts/2024-05-15_18-47-15/_train_tune_2024-05-15_18-47-11/driver_artifacts/_train_tune_4003e_00014_14_2024-05-15_18-47-16/error.txt
(_train_tune pid=3350) /usr/local/lib/python3.10/dist-packages/ray/tune/integration/pytorch_lightning.py:198: ray.tune.integration.pytorch_lightning.TuneReportCallback is deprecated. Use ray.tune.integration.pytorch_lightning.TuneReportCheckpointCallback instead.
(_train_tune pid=3350) Seed set to 1
(_train_tune pid=3350) GPU available: True (cuda), used: True
(_train_tune pid=3350) TPU available: False, using: 0 TPU cores
(_train_tune pid=3350) IPU available: False, using: 0 IPUs
(_train_tune pid=3350) HPU available: False, using: 0 HPUs
(_train_tune pid=3350) Trainer(val_check_interval=1) was configured so validation will run after every batch.
(_train_tune pid=3350) Missing logger folder: /tmp/ray/session_2024-05-15_18-47-11_316201_352/artifacts/2024-05-15_18-47-15/_train_tune_2024-05-15_18-47-11/working_dirs/_train_tune_4003e_00015_15_2024-05-15_18-47-16/lightning_logs
(_train_tune pid=3350) 2024-05-15 18:50:32.416163: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
(_train_tune pid=3350) 2024-05-15 18:50:32.416219: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
(_train_tune pid=3350) 2024-05-15 18:50:32.417601: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
(_train_tune pid=3350) 2024-05-15 18:50:33.770497: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
(_train_tune pid=3350) LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
(_train_tune pid=3350)
(_train_tune pid=3350)   | Name            | Type          | Params
(_train_tune pid=3350) --------------------------------------------------
(_train_tune pid=3350) 0 | loss            | MAE           | 0
(_train_tune pid=3350) 1 | padder          | ConstantPad1d | 0
(_train_tune pid=3350) 2 | scaler          | TemporalNorm  | 0
(_train_tune pid=3350) 3 | hist_encoder    | LSTM          | 484 K
(_train_tune pid=3350) 4 | context_adapter | Linear        | 733 K
(_train_tune pid=3350) 5 | mlp_decoder     | MLP           | 2.4 K
(_train_tune pid=3350) --------------------------------------------------
(_train_tune pid=3350) 1.2 M     Trainable params
(_train_tune pid=3350) 0         Non-trainable params
(_train_tune pid=3350) 1.2 M     Total params
(_train_tune pid=3350) 4.880     Total estimated model params size (MB)
Sanity Checking DataLoader 0:   0%|          | 0/2 [00:00<?, ?it/s]
2024-05-15 18:50:35,801	ERROR tune_controller.py:1331 -- Trial task failed for trial _train_tune_4003e_00015
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/ray/air/execution/_internal/event_manager.py", line 110, in resolve_future
result = ray.get(future)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 2623, in get
values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 861, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(OutOfMemoryError): ray::ImplicitFunc.train() (pid=3350, ip=<IP_ADDRESS>, actor_id=6069a3e61b3cec32c2edcb3901000000, repr=_train_tune)
File "/usr/local/lib/python3.10/dist-packages/ray/tune/trainable/trainable.py", line 331, in train
raise skipped from exception_cause(skipped)
File "/usr/local/lib/python3.10/dist-packages/ray/air/_internal/util.py", line 98, in run
self._ret = self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/tune/trainable/function_trainable.py", line 45, in
training_func=lambda: self._trainable_func(self.config),
File "/usr/local/lib/python3.10/dist-packages/ray/tune/trainable/function_trainable.py", line 248, in _trainable_func
output = fn()
File "/usr/local/lib/python3.10/dist-packages/ray/tune/trainable/util.py", line 130, in inner
return trainable(config, **fn_kwargs)
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_auto.py", line 209, in _train_tune
_ = self._fit_model(
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_auto.py", line 357, in _fit_model
model = model.fit(
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_recurrent.py", line 530, in fit
return self._fit(
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_model.py", line 219, in _fit
trainer.fit(model, datamodule=datamodule)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 544, in fit
call._call_and_handle_interrupt(
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/call.py", line 44, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 580, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 987, in _run
results = self._run_stage()
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1031, in _run_stage
self._run_sanity_check()
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1060, in _run_sanity_check
val_loop.run()
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/utilities.py", line 182, in _decorator
return loop_run(self, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/evaluation_loop.py", line 135, in run
self._evaluation_step(batch, batch_idx, dataloader_idx, dataloader_iter)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/evaluation_loop.py", line 396, in _evaluation_step
output = call._call_strategy_hook(trainer, hook_name, *step_args)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/call.py", line 309, in _call_strategy_hook
output = fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/strategies/strategy.py", line 412, in validation_step
return self.lightning_module.validation_step(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_recurrent.py", line 392, in validation_step
output = self(windows_batch)  # tuple([B, seq_len, H, output])
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/models/lstm.py", line 210, in forward
output = self.mlp_decoder(context)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_modules.py", line 60, in forward
return self.layers(x)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/container.py", line 217, in forward
input = module(input)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/activation.py", line 101, in forward
return F.relu(input, inplace=self.inplace)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py", line 1473, in relu
result = torch.relu(input)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 12.72 GiB. GPU 0 has a total capacity of 14.75 GiB of which 1.07 GiB is free. Process 36671 has 13.68 GiB memory in use. Of the allocated memory 13.54 GiB is allocated by PyTorch, and 10.24 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Trial _train_tune_4003e_00015 errored after 0 iterations at 2024-05-15 18:50:35. Total running time: 3min 19s
Error file: /tmp/ray/session_2024-05-15_18-47-11_316201_352/artifacts/2024-05-15_18-47-15/_train_tune_2024-05-15_18-47-11/driver_artifacts/_train_tune_4003e_00015_15_2024-05-15_18-47-16/error.txt
(_train_tune pid=3440) Seed set to 1
(_train_tune pid=3440) /usr/local/lib/python3.10/dist-packages/ray/tune/integration/pytorch_lightning.py:198: ray.tune.integration.pytorch_lightning.TuneReportCallback is deprecated. Use ray.tune.integration.pytorch_lightning.TuneReportCheckpointCallback instead.
(_train_tune pid=3440) GPU available: True (cuda), used: True
(_train_tune pid=3440) TPU available: False, using: 0 TPU cores
(_train_tune pid=3440) IPU available: False, using: 0 IPUs
(_train_tune pid=3440) HPU available: False, using: 0 HPUs
(_train_tune pid=3440) Trainer(val_check_interval=1) was configured so validation will run after every batch.
(_train_tune pid=3440) Missing logger folder: /tmp/ray/session_2024-05-15_18-47-11_316201_352/artifacts/2024-05-15_18-47-15/_train_tune_2024-05-15_18-47-11/working_dirs/_train_tune_4003e_00016_16_2024-05-15_18-47-16/lightning_logs
(_train_tune pid=3440) 2024-05-15 18:50:45.394308: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
(_train_tune pid=3440) 2024-05-15 18:50:45.394386: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
(_train_tune pid=3440) 2024-05-15 18:50:45.396492: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
(_train_tune pid=3440) 2024-05-15 18:50:47.074955: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
(_train_tune pid=3440) LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
(_train_tune pid=3440)
(_train_tune pid=3440)   | Name            | Type          | Params
(_train_tune pid=3440) --------------------------------------------------
(_train_tune pid=3440) 0 | loss            | MAE           | 0
(_train_tune pid=3440) 1 | padder          | ConstantPad1d | 0
(_train_tune pid=3440) 2 | scaler          | TemporalNorm  | 0
(_train_tune pid=3440) 3 | hist_encoder    | LSTM          | 484 K
(_train_tune pid=3440) 4 | context_adapter | Linear        | 733 K
(_train_tune pid=3440) 5 | mlp_decoder     | MLP           | 2.4 K
(_train_tune pid=3440) --------------------------------------------------
(_train_tune pid=3440) 1.2 M     Trainable params
(_train_tune pid=3440) 0         Non-trainable params
(_train_tune pid=3440) 1.2 M     Total params
(_train_tune pid=3440) 4.880     Total estimated model params size (MB)
Sanity Checking DataLoader 0:   0%|          | 0/2 [00:00<?, ?it/s]
2024-05-15 18:50:49,131	ERROR tune_controller.py:1331 -- Trial task failed for trial _train_tune_4003e_00016
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/ray/air/execution/_internal/event_manager.py", line 110, in resolve_future
result = ray.get(future)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 2623, in get
values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 861, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(OutOfMemoryError): ray::ImplicitFunc.train() (pid=3440, ip=<IP_ADDRESS>, actor_id=c305d2cbb98392178ed4779b01000000, repr=_train_tune)
File "/usr/local/lib/python3.10/dist-packages/ray/tune/trainable/trainable.py", line 331, in train
raise skipped from exception_cause(skipped)
File "/usr/local/lib/python3.10/dist-packages/ray/air/_internal/util.py", line 98, in run
self._ret = self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/tune/trainable/function_trainable.py", line 45, in
training_func=lambda: self._trainable_func(self.config),
File "/usr/local/lib/python3.10/dist-packages/ray/tune/trainable/function_trainable.py", line 248, in _trainable_func
output = fn()
File "/usr/local/lib/python3.10/dist-packages/ray/tune/trainable/util.py", line 130, in inner
return trainable(config, **fn_kwargs)
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_auto.py", line 209, in _train_tune
_ = self._fit_model(
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_auto.py", line 357, in _fit_model
model = model.fit(
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_recurrent.py", line 530, in fit
return self._fit(
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_model.py", line 219, in _fit
trainer.fit(model, datamodule=datamodule)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 544, in fit
call._call_and_handle_interrupt(
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/call.py", line 44, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 580, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 987, in _run
results = self._run_stage()
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1031, in _run_stage
self._run_sanity_check()
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1060, in _run_sanity_check
val_loop.run()
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/utilities.py", line 182, in _decorator
return loop_run(self, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/evaluation_loop.py", line 135, in run
self._evaluation_step(batch, batch_idx, dataloader_idx, dataloader_iter)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/evaluation_loop.py", line 396, in _evaluation_step
output = call._call_strategy_hook(trainer, hook_name, *step_args)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/call.py", line 309, in _call_strategy_hook
output = fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/strategies/strategy.py", line 412, in validation_step
return self.lightning_module.validation_step(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_recurrent.py", line 392, in validation_step
output = self(windows_batch)  # tuple([B, seq_len, H, output])
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/models/lstm.py", line 210, in forward
output = self.mlp_decoder(context)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_modules.py", line 60, in forward
return self.layers(x)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/container.py", line 217, in forward
input = module(input)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/activation.py", line 101, in forward
return F.relu(input, inplace=self.inplace)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py", line 1473, in relu
result = torch.relu(input)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 12.72 GiB. GPU 0 has a total capacity of 14.75 GiB of which 1.07 GiB is free. Process 37881 has 13.68 GiB memory in use. Of the allocated memory 13.54 GiB is allocated by PyTorch, and 10.24 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Trial _train_tune_4003e_00016 errored after 0 iterations at 2024-05-15 18:50:49. Total running time: 3min 33s
Error file: /tmp/ray/session_2024-05-15_18-47-11_316201_352/artifacts/2024-05-15_18-47-15/_train_tune_2024-05-15_18-47-11/driver_artifacts/_train_tune_4003e_00016_16_2024-05-15_18-47-16/error.txt
(_train_tune pid=3535) /usr/local/lib/python3.10/dist-packages/ray/tune/integration/pytorch_lightning.py:198: ray.tune.integration.pytorch_lightning.TuneReportCallback is deprecated. Use ray.tune.integration.pytorch_lightning.TuneReportCheckpointCallback instead.
(_train_tune pid=3535) Seed set to 1
(_train_tune pid=3535) GPU available: True (cuda), used: True
(_train_tune pid=3535) TPU available: False, using: 0 TPU cores
(_train_tune pid=3535) IPU available: False, using: 0 IPUs
(_train_tune pid=3535) HPU available: False, using: 0 HPUs
(_train_tune pid=3535) Trainer(val_check_interval=1) was configured so validation will run after every batch.
(_train_tune pid=3535) Missing logger folder: /tmp/ray/session_2024-05-15_18-47-11_316201_352/artifacts/2024-05-15_18-47-15/_train_tune_2024-05-15_18-47-11/working_dirs/_train_tune_4003e_00017_17_2024-05-15_18-47-16/lightning_logs
(_train_tune pid=3535) 2024-05-15 18:50:57.128093: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
(_train_tune pid=3535) 2024-05-15 18:50:57.128330: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
(_train_tune pid=3535) 2024-05-15 18:50:57.130380: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
(_train_tune pid=3535) 2024-05-15 18:50:59.109092: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
(_train_tune pid=3535) LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
(_train_tune pid=3535)
(_train_tune pid=3535)   | Name            | Type          | Params
(_train_tune pid=3535) --------------------------------------------------
(_train_tune pid=3535) 0 | loss            | MAE           | 0
(_train_tune pid=3535) 1 | padder          | ConstantPad1d | 0
(_train_tune pid=3535) 2 | scaler          | TemporalNorm  | 0
(_train_tune pid=3535) 3 | hist_encoder    | LSTM          | 484 K
(_train_tune pid=3535) 4 | context_adapter | Linear        | 733 K
(_train_tune pid=3535) 5 | mlp_decoder     | MLP           | 2.4 K
(_train_tune pid=3535) --------------------------------------------------
(_train_tune pid=3535) 1.2 M     Trainable params
(_train_tune pid=3535) 0         Non-trainable params
(_train_tune pid=3535) 1.2 M     Total params
(_train_tune pid=3535) 4.880     Total estimated model params size (MB)
Sanity Checking: |          | 0/? [00:00<?, ?it/s]
Sanity Checking DataLoader 0:   0%|          | 0/2 [00:00<?, ?it/s]
2024-05-15 18:51:01,882	ERROR tune_controller.py:1331 -- Trial task failed for trial _train_tune_4003e_00017
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/ray/air/execution/_internal/event_manager.py", line 110, in resolve_future
result = ray.get(future)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 2623, in get
values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 861, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(OutOfMemoryError): ray::ImplicitFunc.train() (pid=3535, ip=<IP_ADDRESS>, actor_id=e8d645cb71673bfc6acc62ba01000000, repr=_train_tune)
File "/usr/local/lib/python3.10/dist-packages/ray/tune/trainable/trainable.py", line 331, in train
raise skipped from exception_cause(skipped)
File "/usr/local/lib/python3.10/dist-packages/ray/air/_internal/util.py", line 98, in run
self._ret = self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/tune/trainable/function_trainable.py", line 45, in
training_func=lambda: self._trainable_func(self.config),
File "/usr/local/lib/python3.10/dist-packages/ray/tune/trainable/function_trainable.py", line 248, in _trainable_func
output = fn()
File "/usr/local/lib/python3.10/dist-packages/ray/tune/trainable/util.py", line 130, in inner
return trainable(config, **fn_kwargs)
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_auto.py", line 209, in _train_tune
_ = self._fit_model(
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_auto.py", line 357, in _fit_model
model = model.fit(
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_recurrent.py", line 530, in fit
return self._fit(
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_model.py", line 219, in _fit
trainer.fit(model, datamodule=datamodule)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 544, in fit
call._call_and_handle_interrupt(
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/call.py", line 44, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 580, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 987, in _run
results = self._run_stage()
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1031, in _run_stage
self._run_sanity_check()
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1060, in _run_sanity_check
val_loop.run()
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/utilities.py", line 182, in _decorator
return loop_run(self, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/evaluation_loop.py", line 135, in run
self._evaluation_step(batch, batch_idx, dataloader_idx, dataloader_iter)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/evaluation_loop.py", line 396, in _evaluation_step
output = call._call_strategy_hook(trainer, hook_name, *step_args)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/call.py", line 309, in _call_strategy_hook
output = fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/strategies/strategy.py", line 412, in validation_step
return self.lightning_module.validation_step(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_recurrent.py", line 392, in validation_step
output = self(windows_batch)  # tuple([B, seq_len, H, output])
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/models/lstm.py", line 210, in forward
output = self.mlp_decoder(context)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_modules.py", line 60, in forward
return self.layers(x)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/container.py", line 217, in forward
input = module(input)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/activation.py", line 101, in forward
return F.relu(input, inplace=self.inplace)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py", line 1473, in relu
result = torch.relu(input)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 12.72 GiB. GPU 0 has a total capacity of 14.75 GiB of which 1.07 GiB is free. Process 39098 has 13.68 GiB memory in use. Of the allocated memory 13.54 GiB is allocated by PyTorch, and 10.24 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Trial _train_tune_4003e_00017 errored after 0 iterations at 2024-05-15 18:51:01. Total running time: 3min 45s
Error file: /tmp/ray/session_2024-05-15_18-47-11_316201_352/artifacts/2024-05-15_18-47-15/_train_tune_2024-05-15_18-47-11/driver_artifacts/_train_tune_4003e_00017_17_2024-05-15_18-47-16/error.txt
(_train_tune pid=3625) Seed set to 1
(_train_tune pid=3625) /usr/local/lib/python3.10/dist-packages/ray/tune/integration/pytorch_lightning.py:198: ray.tune.integration.pytorch_lightning.TuneReportCallback is deprecated. Use ray.tune.integration.pytorch_lightning.TuneReportCheckpointCallback instead.
(_train_tune pid=3625) GPU available: True (cuda), used: True
(_train_tune pid=3625) TPU available: False, using: 0 TPU cores
(_train_tune pid=3625) IPU available: False, using: 0 IPUs
(_train_tune pid=3625) HPU available: False, using: 0 HPUs
(_train_tune pid=3625) Trainer(val_check_interval=1) was configured so validation will run after every batch.
(_train_tune pid=3625) Missing logger folder: /tmp/ray/session_2024-05-15_18-47-11_316201_352/artifacts/2024-05-15_18-47-15/_train_tune_2024-05-15_18-47-11/working_dirs/_train_tune_4003e_00018_18_2024-05-15_18-47-16/lightning_logs
(_train_tune pid=3625) 2024-05-15 18:51:09.586569: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
(_train_tune pid=3625) 2024-05-15 18:51:09.586624: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
(_train_tune pid=3625) 2024-05-15 18:51:09.588053: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
(_train_tune pid=3625) 2024-05-15 18:51:10.939656: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
(_train_tune pid=3625) LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
(_train_tune pid=3625)
(_train_tune pid=3625)   | Name            | Type          | Params
(_train_tune pid=3625) --------------------------------------------------
(_train_tune pid=3625) 0 | loss            | MAE           | 0
(_train_tune pid=3625) 1 | padder          | ConstantPad1d | 0
(_train_tune pid=3625) 2 | scaler          | TemporalNorm  | 0
(_train_tune pid=3625) 3 | hist_encoder    | LSTM          | 484 K
(_train_tune pid=3625) 4 | context_adapter | Linear        | 733 K
(_train_tune pid=3625) 5 | mlp_decoder     | MLP           | 2.4 K
(_train_tune pid=3625) --------------------------------------------------
(_train_tune pid=3625) 1.2 M     Trainable params
(_train_tune pid=3625) 0         Non-trainable params
(_train_tune pid=3625) 1.2 M     Total params
(_train_tune pid=3625) 4.880     Total estimated model params size (MB)
Sanity Checking DataLoader 0:   0%|          | 0/2 [00:00<?, ?it/s]
2024-05-15 18:51:13,382	ERROR tune_controller.py:1331 -- Trial task failed for trial _train_tune_4003e_00018
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/ray/air/execution/_internal/event_manager.py", line 110, in resolve_future
result = ray.get(future)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 2623, in get
values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 861, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(OutOfMemoryError): ray::ImplicitFunc.train() (pid=3625, ip=<IP_ADDRESS>, actor_id=b85224b75f519021d2f8974001000000, repr=_train_tune)
File "/usr/local/lib/python3.10/dist-packages/ray/tune/trainable/trainable.py", line 331, in train
raise skipped from exception_cause(skipped)
File "/usr/local/lib/python3.10/dist-packages/ray/air/_internal/util.py", line 98, in run
self._ret = self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/tune/trainable/function_trainable.py", line 45, in
training_func=lambda: self._trainable_func(self.config),
File "/usr/local/lib/python3.10/dist-packages/ray/tune/trainable/function_trainable.py", line 248, in _trainable_func
output = fn()
File "/usr/local/lib/python3.10/dist-packages/ray/tune/trainable/util.py", line 130, in inner
return trainable(config, **fn_kwargs)
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_auto.py", line 209, in _train_tune
_ = self._fit_model(
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_auto.py", line 357, in _fit_model
model = model.fit(
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_recurrent.py", line 530, in fit
return self._fit(
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_model.py", line 219, in _fit
trainer.fit(model, datamodule=datamodule)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 544, in fit
call._call_and_handle_interrupt(
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/call.py", line 44, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 580, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 987, in _run
results = self._run_stage()
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1031, in _run_stage
self._run_sanity_check()
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1060, in _run_sanity_check
val_loop.run()
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/utilities.py", line 182, in _decorator
return loop_run(self, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/evaluation_loop.py", line 135, in run
self._evaluation_step(batch, batch_idx, dataloader_idx, dataloader_iter)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/evaluation_loop.py", line 396, in _evaluation_step
output = call._call_strategy_hook(trainer, hook_name, *step_args)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/call.py", line 309, in _call_strategy_hook
output = fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/strategies/strategy.py", line 412, in validation_step
return self.lightning_module.validation_step(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_recurrent.py", line 392, in validation_step
output = self(windows_batch)  # tuple([B, seq_len, H, output])
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/models/lstm.py", line 210, in forward
output = self.mlp_decoder(context)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_modules.py", line 60, in forward
return self.layers(x)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/container.py", line 217, in forward
input = module(input)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/activation.py", line 101, in forward
return F.relu(input, inplace=self.inplace)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py", line 1473, in relu
result = torch.relu(input)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 12.72 GiB. GPU 0 has a total capacity of 14.75 GiB of which 1.07 GiB is free. Process 40264 has 13.68 GiB memory in use. Of the allocated memory 13.54 GiB is allocated by PyTorch, and 10.24 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Trial _train_tune_4003e_00018 errored after 0 iterations at 2024-05-15 18:51:13. Total running time: 3min 57s
Error file: /tmp/ray/session_2024-05-15_18-47-11_316201_352/artifacts/2024-05-15_18-47-15/_train_tune_2024-05-15_18-47-11/driver_artifacts/_train_tune_4003e_00018_18_2024-05-15_18-47-16/error.txt
(_train_tune pid=3707) /usr/local/lib/python3.10/dist-packages/ray/tune/integration/pytorch_lightning.py:198: ray.tune.integration.pytorch_lightning.TuneReportCallback is deprecated. Use ray.tune.integration.pytorch_lightning.TuneReportCheckpointCallback instead.
(_train_tune pid=3707) Seed set to 1
(_train_tune pid=3707) GPU available: True (cuda), used: True
(_train_tune pid=3707) TPU available: False, using: 0 TPU cores
(_train_tune pid=3707) IPU available: False, using: 0 IPUs
(_train_tune pid=3707) HPU available: False, using: 0 HPUs
(_train_tune pid=3707) Trainer(val_check_interval=1) was configured so validation will run after every batch.
(_train_tune pid=3707) Missing logger folder: /tmp/ray/session_2024-05-15_18-47-11_316201_352/artifacts/2024-05-15_18-47-15/_train_tune_2024-05-15_18-47-11/working_dirs/_train_tune_4003e_00019_19_2024-05-15_18-47-16/lightning_logs
(_train_tune pid=3707) 2024-05-15 18:51:22.313082: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
(_train_tune pid=3707) 2024-05-15 18:51:22.313161: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
(_train_tune pid=3707) 2024-05-15 18:51:22.314817: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
(_train_tune pid=3707) 2024-05-15 18:51:23.665602: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
(_train_tune pid=3707) LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
(_train_tune pid=3707)
(_train_tune pid=3707)   | Name            | Type          | Params
(_train_tune pid=3707) --------------------------------------------------
(_train_tune pid=3707) 0 | loss            | MAE           | 0
(_train_tune pid=3707) 1 | padder          | ConstantPad1d | 0
(_train_tune pid=3707) 2 | scaler          | TemporalNorm  | 0
(_train_tune pid=3707) 3 | hist_encoder    | LSTM          | 484 K
(_train_tune pid=3707) 4 | context_adapter | Linear        | 733 K
(_train_tune pid=3707) 5 | mlp_decoder     | MLP           | 2.4 K
(_train_tune pid=3707) --------------------------------------------------
(_train_tune pid=3707) 1.2 M     Trainable params
(_train_tune pid=3707) 0         Non-trainable params
(_train_tune pid=3707) 1.2 M     Total params
(_train_tune pid=3707) 4.880     Total estimated model params size (MB)
Sanity Checking DataLoader 0:   0%|          | 0/2 [00:00<?, ?it/s]
2024-05-15 18:51:25,685	ERROR tune_controller.py:1331 -- Trial task failed for trial _train_tune_4003e_00019
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/ray/air/execution/_internal/event_manager.py", line 110, in resolve_future
result = ray.get(future)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 2623, in get
values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 861, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(OutOfMemoryError): ray::ImplicitFunc.train() (pid=3707, ip=<IP_ADDRESS>, actor_id=a454d5dc5c3d4c81658a621601000000, repr=_train_tune)
File "/usr/local/lib/python3.10/dist-packages/ray/tune/trainable/trainable.py", line 331, in train
raise skipped from exception_cause(skipped)
File "/usr/local/lib/python3.10/dist-packages/ray/air/_internal/util.py", line 98, in run
self._ret = self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/tune/trainable/function_trainable.py", line 45, in
training_func=lambda: self._trainable_func(self.config),
File "/usr/local/lib/python3.10/dist-packages/ray/tune/trainable/function_trainable.py", line 248, in _trainable_func
output = fn()
File "/usr/local/lib/python3.10/dist-packages/ray/tune/trainable/util.py", line 130, in inner
return trainable(config, **fn_kwargs)
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_auto.py", line 209, in _train_tune
_ = self._fit_model(
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_auto.py", line 357, in _fit_model
model = model.fit(
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_recurrent.py", line 530, in fit
return self._fit(
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_model.py", line 219, in _fit
trainer.fit(model, datamodule=datamodule)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 544, in fit
call._call_and_handle_interrupt(
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/call.py", line 44, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 580, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 987, in _run
results = self._run_stage()
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1031, in _run_stage
self._run_sanity_check()
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1060, in _run_sanity_check
val_loop.run()
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/utilities.py", line 182, in _decorator
return loop_run(self, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/evaluation_loop.py", line 135, in run
self._evaluation_step(batch, batch_idx, dataloader_idx, dataloader_iter)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/evaluation_loop.py", line 396, in _evaluation_step
output = call._call_strategy_hook(trainer, hook_name, *step_args)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/call.py", line 309, in _call_strategy_hook
output = fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/strategies/strategy.py", line 412, in validation_step
return self.lightning_module.validation_step(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_base_recurrent.py", line 392, in validation_step
output = self(windows_batch)  # tuple([B, seq_len, H, output])
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/models/lstm.py", line 210, in forward
output = self.mlp_decoder(context)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/neuralforecast/common/_modules.py", line 60, in forward
return self.layers(x)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/container.py", line 217, in forward
input = module(input)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/activation.py", line 101, in forward
return F.relu(input, inplace=self.inplace)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py", line 1473, in relu
result = torch.relu(input)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 12.72 GiB. GPU 0 has a total capacity of 14.75 GiB of which 1.07 GiB is free. Process 40709 has 13.68 GiB memory in use. Of the allocated memory 13.54 GiB is allocated by PyTorch, and 10.24 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-05-15 18:51:25,714	INFO tune.py:1007 -- Wrote the latest version of all result files and experiment state to '/root/ray_results/_train_tune_2024-05-15_18-47-11' in 0.0219s.
2024-05-15 18:51:25,720	ERROR tune.py:1035 -- Trials did not complete: [_train_tune_4003e_00000, _train_tune_4003e_00001, _train_tune_4003e_00002, _train_tune_4003e_00003, _train_tune_4003e_00004, _train_tune_4003e_00005, _train_tune_4003e_00006, _train_tune_4003e_00007, _train_tune_4003e_00008, _train_tune_4003e_00009, _train_tune_4003e_00010, _train_tune_4003e_00011, _train_tune_4003e_00012, _train_tune_4003e_00013, _train_tune_4003e_00014, _train_tune_4003e_00015, _train_tune_4003e_00016, _train_tune_4003e_00017, _train_tune_4003e_00018, _train_tune_4003e_00019]
2024-05-15 18:51:25,750	WARNING experiment_analysis.py:558 -- Could not find best trial. Did you pass the correct metric parameter?
Trial _train_tune_4003e_00019 errored after 0 iterations at 2024-05-15 18:51:25. Total running time: 4min 9s
Error file: /tmp/ray/session_2024-05-15_18-47-11_316201_352/artifacts/2024-05-15_18-47-15/_train_tune_2024-05-15_18-47-11/driver_artifacts/_train_tune_4003e_00019_19_2024-05-15_18-47-16/error.txt

RuntimeError                              Traceback (most recent call last)
 in <cell line: 16>()
14
15 # Fit model
---> 16 nf.fit(df=df_daily, val_size=12, sort_df=True, verbose=True)
17 Y_hat_df = nf.predict()
18 Y_hat_df = Y_hat_df.reset_index()
2 frames
/usr/local/lib/python3.10/dist-packages/ray/tune/result_grid.py in get_best_result(self, metric, mode, scope, filter_nan_and_inf)
159                 else "."
160             )
--> 161             raise RuntimeError(error_msg)
162
163         return self._trial_to_result(best_trial)
RuntimeError: No best trial found for the given metric: loss. This means that no trial has reported this metric, or all values reported for this metric are NaN. To not ignore NaN values, you can set the filter_nan_and_inf arg to False.

It seems to give a GPU OOM error. So your GPU doesn't have enough RAM to run this task. This would also explain why it does run on weekly/monthly, as those will have less data than daily.
The solution is to buy a GPU with more RAM, or run a less compute-intensive experiment (e.g. use less data, or less frequent)

It seems to give a GPU OOM error. So your GPU doesn't have enough RAM to run this task. This would also explain why it does run on weekly/monthly, as those will have less data than daily.
The solution is to buy a GPU with more RAM, or run a less compute-intensive experiment (e.g. use less data, or less frequent)

But I am running it in MS Azure with a RAM of 112GB

It seems to give a GPU OOM error. So your GPU doesn't have enough RAM to run this task. This would also explain why it does run on weekly/monthly, as those will have less data than daily.
The solution is to buy a GPU with more RAM, or run a less compute-intensive experiment (e.g. use less data, or less frequent)

But I am running it in MS Azure on a compute with GPU and with a RAM of 112GB

GPU RAM, not normal RAM.

Funny it worked after I ran a loop for each unique_id with the same resources like:
unique_ids = df_daily['unique_id'].unique()
for unique_id in unique_ids:
df_sub = df_daily[df_daily['unique_id'] == unique_id]

nf.fit(df=df_sub, val_size=365, sort_df=True, verbose=True)

Y_hat_df = nf.predict()
Y_hat_df['unique_id'] = unique_id  # Add the unique_id to the predictions

all_predictions.append(Y_hat_df.reset_index())


Add STASH build requirements for clue scroll overlay

Changes implemented for issue: Clue stash build helper #14744
Fixes: #14744

Where is this information actually being displayed to the user? It doesn't seem like anything is actually using either of the helper strings currently

It should be displayed in the EmoteClue.java. I am trying to determine where to implement it.

Where is this information actually being displayed to the user? It doesn't seem like anything is actually using either of the helper strings currently

It should be displayed in the EmoteClue.java, within the constructor's STASHUnit parameter


You can see them, but the label won't show as it's near to impossible pinpointing the second where the marker is on a 1h+ video.
Make them show the label way earlier than of now you can't use labels on long videos.
Also, please add more styling possibilities.

Open to CRs for this 👍🏼


App dose not install

Hi, I have added the library but while installing the app getting below error, kindly help
More than one file was found with OS independent path 'lib/armeabi-v7a/libc++_shared.so'.

packagingOptions {
pickFirst 'lib/x86/libc++_shared.so'
pickFirst 'lib/x86_64/libc++_shared.so'
pickFirst 'lib/armeabi-v7a/libc++_shared.so'
pickFirst 'lib/arm64-v8a/libc++_shared.so'
}

include this in your gradle file above compiled sdk version in android tags

Thanks a lot...it worked.


How can I access the previous version of the documentation?

Hello,
I saw that the documentation is currently being rewritten.
I'm using Bunit <IP_ADDRESS> preview 6 and I could use the documentation for it .
Is there a way I could access it?
Regards, Hajni

Hi Hajni,
Sorry, I didn't think to preserve the older versions of the docs somehow on the website. However, GIT has us covered.
This commit contains all the docs from right before beta-7 was released: https://github.com/egil/bunit/tree/e741bba57db0776da225b2b24255f4a0a9b8e7a3/docs/docs
Alternatively, you can go back to the docs at the initial release of beta-6: https://github.com/egil/bunit/tree/v1.0.0-beta-6/bunit.docs/docs


Comments are not read only and groups comments can be made

I could only test this with one group, because more do not exist in the current state of the project.
In firebase I have changed the title to another title, and then the comment was not displayed.
Run together with preview-code/backend#5
Fixes #91

This needs a rebase, what's the status Eva?

Still have the duplicated response bug from backend-ajax, can't seem to figure it out.

We need to find a solution for the per-page on get-comments.
If the amount of comments goes beyond only the oldest comments are shown.

LGTM


multicompiler build with webpack serve

Bug report
Multicompiler build with dependencies fails with webpack serve and success with common webpack build

What is the current behavior?
build with webpack serve fails without building dependency compiler first.
If the current behavior is a bug, please provide the steps to reproduce.
https://github.com/vankop/webpack-serve-bug

run npm run build. it should successfully build
remove src2/build1.js
run npm run serve. it should fail

What is the expected behavior?
build succeeded

Other relevant information:
webpack version:  latest
Node.js version:
Operating System:
Additional tools:

bump

Yeah, it is intresting, by default when you run serve, we run dev server on both compiation (but we keep files in memory), to solve it we need:

Allow to disable dev server for required compilation, here is issue https://github.com/webpack/webpack-cli/issues/2408

As workaround now you can do it:
const path = require("path");

module.exports = [
    {
        mode: "production",
        name: "build1",
        entry: "./src1",
        output: { filename: "build1.js", path: path.resolve(__dirname, "src2") },
        devServer: {
            devMiddleware: {
                writeToDisk: true,
            }
        }
    },
    {
        mode: "production",
        name: "app",
        dependencies: ["build1"],
        entry: "./src2",
    }
];

I want to close the issue here (in favor https://github.com/webpack/webpack-cli/issues/2408), because we should fix bug/implement enchantment in webpack-cli.
Feel free to feedback


Voice Access commang like "Delete that" is not working wit PasswordBox

Describe the bug
I test the voice command with TextBox control and works as expected. But not for PasswordBox when the password is not reveal. If
PasswordRevealMode is true the "delete that" command works.
Steps to reproduce the bug
Using a PasswordBox control, try to delete the content using voice access command "Delete that"
Expected behavior
No response
Screenshots
delete that.webm

NuGet package version
WinUI 3 - Windows App SDK 1.5.3: 1.5.240428000
Windows version
Windows 11 (22H2): Build 22621
Additional context
No response

Thanks for reporting this issue. I've moved the internal version of this bug to the Voice Access team.


Lazy Indexing

Here's the idea. Instead of doing mass reindex in project root there could be something like a lazy index. On the first completion, we could collect files and classes only (which should be much faster than a full index), and then when doing completion inside the file index missing related files. So if class A uses object of class B then index class B etc. Hopefully, it's clear what the idea is.
With that feature, we could have full completion in any project without needs for full reindex.
Just to give you some figures, reindex in one of my projects:
$ \time -f %es padawan generate
...
74.82s

In the same project:
$ \time -f %es ag --php '^class.*$'
...
0.98s

It would be probably slower in pure PHP, but still should be faster than full reindex.
How feasible you think it is?

@pbogut there is a new feature in padawan "Index update", it will update index only for changed files. For now it's only available via http interface


Sector access/provisioning in files

Maybe this is a feature request, maybe this already exists (hallelujah).
The use case: I receive data in sequential order, but some chunks get corrupted, so when they are resent I need to go back and patch the file (space was inserted instead). In my case all chunks are 1kB.
For this to work I need just 2 functions:

SPIFFS_allocate(file, size) - sets the file-size and reserves the necessary amount of (wiped) sectors
SPIFFS_write_sector(file, sector, contents) - write a sector (once suffices)

SPIFFS_allocate() probably exists, as it is similar to ftruncate() under POSIX.
SPIFFS_write_sector() may also already exists in some form, because it means finding the sector in a dictionary and performing a write to it (I hope so because I painted myself in a corner without this functionality)
Note that using the C API would be possible, but extremely cumbersome, slow, destructive to the flash, and energy inefficient to the point of not being viable. So I think this is important functionality, specially in an embedded (IOT) environment. Note: I use an ESP32.
Many thanks for your efforts on SPIFFS.

Really, if ftruncate() provisions wiped space (0xFF), a specialized SPIFFS_write_once(range, data) could simply write the wiped range. It probably needs to fail if the range doesn't hold 0xFF to begin with.

I've been looking into the docs, and it seems sectors (pages) sacrifice a few bytes for the "page header". Which means the user-visible sector-size is less than the hardware sector-size (256B) in the case of the ESP32. So I guess that rules out any exposure.


xseed.sh working but not finding results question.

Greetings!
Thanks for sharing such an awesome set of scripts.
I've setup the xseed.sh script and everything seems to be working as intended however I have yet to see any matches detected.
Wondering if this is due to the files having been renamed by Sonarr and when searched no long match what the sites have provided.
Wondering if there is something I should be doing different/configure different to allow these searches to match properly with torrents. Thanks!

for xseed support see the xseed support methods; unrelated to the script

Got it, thanks for pointing in the right direction!


Cart: Bundle options are not displayed in the Minicart

By some reason the configured bundle options are not displayed in the cart:

It works pretty OK with 1.10.6 (demo.storefrontcloud.io):

This is probably related to https://github.com/DivanteLtd/vue-storefront/issues/4104 - which has been confirmed to work properly on 1.11.2 (so we probably just need a merge of capybara instance to 1.11.2) - the problem was with totals (this product options infromations came from Totals)


Use github.com/go-gl/gl instead of go-gl/glow

I was just about to open a new issue about this when I noticed this one.
I currently have to locally change it to github.com/go-gl/gl/v3.2-core/gl in order to build my code.
(Also the go-gl/glow import doesn’t work on Travis-CI)

Thank you for noticing me. I'll be working on this asap.

Now it seems that Ebiten can be built on Travis CI.


Unit tests

Updated the unittest: Made them compatible with Python 3 (tested agains @tcalmant's port). Also made some minor improvements to the tests. Of course there are tests failing due to bugs in the Python 3 port that have already been fixed in origninell's version.

thanks for the great job!


PacReq: apx

Types
-git
URL
https://github.com/Vanilla-OS/apx
Building guide
We have to add distrobox as a pacdep and change the distrobox path line in config/config.json
Then I think this should do:
go build
go install

I think apx will be a brilliant idea to implement into Pacstall


Using k8s SD for alertmanager: doesn't cache address when API server goes away

We've had this a couple of times now: when the API servers go away or Prometheus has problems speaking to them, Prometheus "forgets" where the alertmanagers are and we no longer get alerts.
I'm investigating.
This is on 2.2.1

That sounds like a general SD issue. All SDs are meant to be resilient to this.

We're using k8s SD for the targets, and they didn't disappear.

Hmm, it should all be the same code.

They are two separate instances of the same code, so its conceivable one got into a bad state and the other didn't.

Heres the goroutines: https://gist.github.com/tomwilkie/054376c3559dd24281a333ab2ec82d01

Yeah, looks like an SD issue (rather that a alerts issue) - here are the events, and they all just stopped at about 14:11, which is about when the master went down:

Another interesting aside; the config was reloaded at 14:46, and a bunch of the k8s client goroutines are stuck since then (waiting for HasSynced).
So a new theory: API server went away, but everything was fine.  Then out config got reloaded, and our alertmanagers go forgotten about - our alerts started clearing around 14:51PM, indicating the alert manager was forgotten at ~14:46PM.

We've experienced multiple issues with the K8s discovery stopping to work as well in 2.x releases. See the original description in #3810. It can be unbblocked by sending a SIGHUP. The latest occurance was today on a v2.2.1 server.

Can you try the patch in #4013? It could explain some of this.

Is 2.3.1 better?

@tomwilkie are you still seeing this issue? v2.4.x included lots of improvements fixes on the SD side and in particular, the annoying https://github.com/prometheus/prometheus/issues/4124 should :tm: be fixed.