Issue executing PyTorch training on agent remote compute resource

Description of issue
I am having an issue getting a PyTorch model to train on a remote compute server using clearml and I am wondering if it is something to do with the virtual environment. I have tried all three methods pip venv, conda venv and docker. The closest have something working is using the default pip package venv method. The model runs for an iteration and then crashes with cudnn error. I can’t help thinking this is something to do with the pip installation of PyTorch as they recommended using the conda channel to install it. When I setup my local virtual environment I use a combination of Conda and pip. I use conda as my environment manager, and then use pip for packages that are not in the conda repositories.
I am running this on a bespoke azure vm image that I created, with 10.1, 10.2, 11.1 and 11.2 CUDA versions installed, and the correct CUDNN libraries supported by the latest nvidia driver. I have verified that the CUDA drivers are working as I have been able to train models in conda environments directly on the machine, but not using clearml. I use the same trainer classes that I have written using Ignite, that I have used with the clearml experiments.
I have made sure that the CUDA version is correctly specified for the version being used by the PyTorch installation, and I have been running the clearml-agent in its own conda environment, with the environment variables pointing towards the correct version CUDA, in this case 11.1. I have verified that the different versions of CUDA work properly by setting up conda environments with the different versions of PyTorch and successfully trained models using 10.2, 11.1 and 11.2, outside of clearml
However I run into the same cudnn error on the forward calculation of first iteration when I run it through clearml.
I have also tried training a variety of network architectures from a number of libraries (Torchvision, pytorchcv, TIMM), as well  as a simple VGG implementation from scratch, and come across the same issues.
Is this potentially an issue with having multiple CUDA versions installed on the server?
The CUDNN error on execution by clearml-agent
Following execution of the experiment on the remote compute resource, the model trains for an iteration, and then fails with the following error.
Current run is terminating due to exception: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
You can try to repro this exception using the following code snippet. If that doesn't trigger the error, please include your original repro script when reporting this issue.

import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.allow_tf32 = True
data = torch.randn([16, 128, 28, 28], dtype=torch.float, device='cuda', requires_grad=True)
net = torch.nn.Conv2d(128, 128, kernel_size=[3, 3], padding=[1, 1], stride=[1, 1], dilation=[1, 1], groups=1)
net = net.cuda().float()
out = net(data)
out.backward(torch.randn_like(out))
torch.cuda.synchronize()

ConvolutionParams
    data_type = CUDNN_DATA_FLOAT
    padding = [1, 1, 0]
    stride = [1, 1, 0]
    dilation = [1, 1, 0]
    groups = 1
    deterministic = false
    allow_tf32 = true
input: TensorDescriptor 0x564675040c30
    type = CUDNN_DATA_FLOAT
    nbDims = 4
    dimA = 16, 128, 28, 28,
    strideA = 100352, 784, 28, 1,
output: TensorDescriptor 0x564674fa4210
    type = CUDNN_DATA_FLOAT
    nbDims = 4
    dimA = 16, 128, 28, 28,
    strideA = 100352, 784, 28, 1,
weight: FilterDescriptor 0x564674fa1b60
    type = CUDNN_DATA_FLOAT
    tensor_format = CUDNN_TENSOR_NCHW
    nbDims = 4
    dimA = 128, 128, 3, 3,
Pointer addresses:
    input: 0x7f151ec40000
    output: 0x7f1518000000
    weight: 0x7f154cd2e400
Forward algorithm: 7

Engine run is terminating due to exception: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
You can try to repro this exception using the following code snippet. If that doesn't trigger the error, please include your original repro script when reporting this issue.

import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.allow_tf32 = True
data = torch.randn([16, 128, 28, 28], dtype=torch.float, device='cuda', requires_grad=True)
net = torch.nn.Conv2d(128, 128, kernel_size=[3, 3], padding=[1, 1], stride=[1, 1], dilation=[1, 1], groups=1)
net = net.cuda().float()
out = net(data)
out.backward(torch.randn_like(out))
torch.cuda.synchronize()

ConvolutionParams
    data_type = CUDNN_DATA_FLOAT
    padding = [1, 1, 0]
    stride = [1, 1, 0]
    dilation = [1, 1, 0]
    groups = 1
    deterministic = false
    allow_tf32 = true
input: TensorDescriptor 0x564675040c30
    type = CUDNN_DATA_FLOAT
    nbDims = 4
    dimA = 16, 128, 28, 28,
    strideA = 100352, 784, 28, 1,
output: TensorDescriptor 0x564674fa4210
    type = CUDNN_DATA_FLOAT
    nbDims = 4
    dimA = 16, 128, 28, 28,
    strideA = 100352, 784, 28, 1,
weight: FilterDescriptor 0x564674fa1b60
    type = CUDNN_DATA_FLOAT
    tensor_format = CUDNN_TENSOR_NCHW
    nbDims = 4
    dimA = 128, 128, 3, 3,
Pointer addresses:
    input: 0x7f151ec40000
    output: 0x7f1518000000
    weight: 0x7f154cd2e400
Forward algorithm: 7

Traceback (most recent call last):
  File "train_clearml_pytorch_ignite_caltech_birds.py", line 104, in <module>
    trainer.run()
  File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/cub_tools/trainer.py", line 640, in run
    self.train_engine.run(self.train_loader, max_epochs=self.config.TRAIN.NUM_EPOCHS)
  File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/ignite/engine/engine.py", line 702, in run
    return self._internal_run()
  File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/ignite/engine/engine.py", line 775, in _internal_run
    self._handle_exception(e)
  File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/ignite/engine/engine.py", line 469, in _handle_exception
    raise e
  File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/ignite/engine/engine.py", line 745, in _internal_run
    time_taken = self._run_once_on_dataset()
  File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/ignite/engine/engine.py", line 850, in _run_once_on_dataset
    self._handle_exception(e)
  File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/ignite/engine/engine.py", line 469, in _handle_exception
    raise e
  File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/ignite/engine/engine.py", line 833, in _run_once_on_dataset
    self.state.output = self._process_function(self, self.state.batch)
  File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/cub_tools/trainer.py", line 448, in train_step
    y_pred = self.model(x)
  File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/torchvision/models/resnet.py", line 249, in forward
    return self._forward_impl(x)
  File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/torchvision/models/resnet.py", line 238, in _forward_impl
    x = self.layer2(x)
  File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/torch/nn/modules/container.py", line 119, in forward
    input = module(input)
  File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/torchvision/models/resnet.py", line 74, in forward
    out = self.conv2(out)
  File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 399, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 395, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
You can try to repro this exception using the following code snippet. If that doesn't trigger the error, please include your original repro script when reporting this issue.

import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.allow_tf32 = True
data = torch.randn([16, 128, 28, 28], dtype=torch.float, device='cuda', requires_grad=True)
net = torch.nn.Conv2d(128, 128, kernel_size=[3, 3], padding=[1, 1], stride=[1, 1], dilation=[1, 1], groups=1)
net = net.cuda().float()
out = net(data)
out.backward(torch.randn_like(out))
torch.cuda.synchronize()

ConvolutionParams
    data_type = CUDNN_DATA_FLOAT
    padding = [1, 1, 0]
    stride = [1, 1, 0]
    dilation = [1, 1, 0]
    groups = 1
    deterministic = false
    allow_tf32 = true
input: TensorDescriptor 0x564675040c30
    type = CUDNN_DATA_FLOAT
    nbDims = 4
    dimA = 16, 128, 28, 28,
    strideA = 100352, 784, 28, 1,
output: TensorDescriptor 0x564674fa4210
    type = CUDNN_DATA_FLOAT
    nbDims = 4
    dimA = 16, 128, 28, 28,
    strideA = 100352, 784, 28, 1,
weight: FilterDescriptor 0x564674fa1b60
    type = CUDNN_DATA_FLOAT
    tensor_format = CUDNN_TENSOR_NCHW
    nbDims = 4
    dimA = 128, 128, 3, 3,
Pointer addresses:
    input: 0x7f151ec40000
    output: 0x7f1518000000
    weight: 0x7f154cd2e400
Forward algorithm: 7

1621437626467 ecm-clearml-compute-gpu-001:0 DEBUG Process failed, exit code 1

clearml-agent environment setup logs
The following below shows the terminal logging of the environment setup on execution of the experiment by the clearml-server.
1621437371563 ecm-clearml-compute-gpu-001:0 INFO task 1fc27ce7118542ea8845d8963a71bff7 pulled from b6da68d36d9944d89d05bc778bfcb196 by worker ecm-clearml-compute-gpu-001:0

1621437376755 ecm-clearml-compute-gpu-001:0 DEBUG Current configuration (clearml_agent v1.0.0, location: /tmp/.clearml_agent.oieq9_34.cfg):
----------------------
api.version = 1.5
api.verify_certificate = true
api.default_version = 1.5
api.http.max_req_size = 15728640
api.http.retries.total = 240
api.http.retries.connect = 240
api.http.retries.read = 240
api.http.retries.redirect = 240
api.http.retries.status = 240
api.http.retries.backoff_factor = 1.0
api.http.retries.backoff_max = 120.0
api.http.wait_on_maintenance_forever = true
api.http.pool_maxsize = 512
api.http.pool_connections = 512
api.api_server = http://ecm-clearml-server-001.westeurope.cloudapp.azure.com:8008
api.web_server = http://ecm-clearml-server-001.westeurope.cloudapp.azure.com:8080
api.files_server = http://ecm-clearml-server-001.westeurope.cloudapp.azure.com:8081
api.credentials.access_key = A6NUAZ2MQ87B25LYQT89
api.host = http://ecm-clearml-server-001.westeurope.cloudapp.azure.com:8008
agent.worker_id = ecm-clearml-compute-gpu-001:0
agent.worker_name = ecm-clearml-compute-gpu-001
agent.force_git_ssh_protocol = false
agent.python_binary =
agent.package_manager.type = pip
agent.package_manager.pip_version = <20.2
agent.package_manager.system_site_packages = false
agent.package_manager.force_upgrade = false
agent.package_manager.conda_channels.0 = pytorch
agent.package_manager.conda_channels.1 = conda-forge
agent.package_manager.conda_channels.2 = defaults
agent.package_manager.torch_nightly = false
agent.venvs_dir = /home/edmorris/.clearml/venvs-builds
agent.venvs_cache.max_entries = 10
agent.venvs_cache.free_space_threshold_gb = 2.0
agent.vcs_cache.enabled = true
agent.vcs_cache.path = /home/edmorris/.clearml/vcs-cache
agent.venv_update.enabled = false
agent.pip_download_cache.enabled = true
agent.pip_download_cache.path = /home/edmorris/.clearml/pip-download-cache
agent.translate_ssh = true
agent.reload_config = false
agent.docker_pip_cache = /home/edmorris/.clearml/pip-cache
agent.docker_apt_cache = /home/edmorris/.clearml/apt-cache
agent.docker_force_pull = false
agent.default_docker.image = nvidia/cuda:11.1.1-cudnn8-runtime-ubuntu18.04
agent.enable_task_env = false
agent.git_user =<EMAIL_ADDRESS>agent.cuda_version = 111
agent.cudnn_version = 81
agent.default_python = 3.8
sdk.storage.cache.default_base_dir = ~/.clearml/cache
sdk.storage.cache.size.min_free_bytes = 10GB
sdk.storage.direct_access.0.url = file://*
sdk.metrics.file_history_size = 100
sdk.metrics.matplotlib_untitled_history_size = 100
sdk.metrics.images.format = JPEG
sdk.metrics.images.quality = 87
sdk.metrics.images.subsampling = 0
sdk.metrics.tensorboard_single_series_per_graph = false
sdk.network.metrics.file_upload_threads = 4
sdk.network.metrics.file_upload_starvation_warning_sec = 120
sdk.network.iteration.max_retries_on_server_error = 5
sdk.network.iteration.retry_backoff_factor_sec = 10
sdk.aws.s3.key =
sdk.aws.s3.region =
sdk.aws.boto3.pool_connections = 512
sdk.aws.boto3.max_multipart_concurrency = 16
sdk.log.null_log_propagate = false
sdk.log.task_log_buffer_capacity = 66
sdk.log.disable_urllib3_info = true
sdk.development.task_reuse_time_window_in_hours = 72.0
sdk.development.vcs_repo_detect_async = true
sdk.development.store_uncommitted_code_diff = true
sdk.development.support_stopping = true
sdk.development.default_output_uri =
sdk.development.force_analyze_entire_repo = false
sdk.development.suppress_update_message = false
sdk.development.detect_with_pip_freeze = false
sdk.development.worker.report_period_sec = 2
sdk.development.worker.ping_period_sec = 30
sdk.development.worker.log_stdout = true
sdk.development.worker.report_global_mem_used = false

Executing task id [1fc27ce7118542ea8845d8963a71bff7]:
repository = https://github.com/ecm200/caltech_birds.git
branch = clearml_integrations
version_num = b2e6741c9cc3c869fe36490ec39d165c5fba8c6c
tag =
docker_cmd = None
entry_point = train_clearml_pytorch_ignite_caltech_birds.py
working_dir = scripts

created virtual environment CPython3.8.8.final.0-64 in 136ms
  creator CPython3Posix(dest=/home/edmorris/.clearml/venvs-builds/3.8, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/edmorris/.local/share/virtualenv)
    added seed packages: pip==21.1.1, setuptools==56.0.0, wheel==0.36.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

Using cached repository in "/home/edmorris/.clearml/vcs-cache/caltech_birds.git.c0f811d2350a4cd7faa8a3d3d8453cca/caltech_birds.git"
Note: checking out 'b2e6741c9cc3c869fe36490ec39d165c5fba8c6c'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at b2e6741 More updates for ClearML execution
type: git
url: https://github.com/ecm200/caltech_birds.git
branch: HEAD
commit: b2e6741c9cc3c869fe36490ec39d165c5fba8c6c
root: /home/edmorris/.clearml/venvs-builds/3.8/task_repository/caltech_birds.git
Applying uncommitted changes
Executing: ('git', 'apply', '--unidiff-zero'): b'<stdin>:60: trailing whitespace.\n    \nwarning: 1 line adds whitespace errors.\n'

1621437381702 ecm-clearml-compute-gpu-001:0 DEBUG Collecting pip<20.2
  Using cached pip-20.1.1-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.1.1
    Uninstalling pip-21.1.1:
      Successfully uninstalled pip-21.1.1
Successfully installed pip-20.1.1
Collecting Cython
  Using cached Cython-0.29.23-cp38-cp38-manylinux1_x86_64.whl (1.9 MB)
Installing collected packages: Cython
Successfully installed Cython-0.29.23
Collecting numpy==1.20.2
  Using cached numpy-1.20.2-cp38-cp38-manylinux2010_x86_64.whl (15.4 MB)
Installing collected packages: numpy

1621437386811 ecm-clearml-compute-gpu-001:0 DEBUG Successfully installed numpy-1.20.2
Torch CUDA 111 download page found
Found PyTorch version torch==1.8.1 matching CUDA version 111
Found PyTorch version torchvision==0.9.1 matching CUDA version 111
Collecting torch==1.8.1+cu111
  File was already downloaded /home/edmorris/.clearml/pip-download-cache/cu111/torch-1.8.1+cu111-cp38-cp38-linux_x86_64.whl
Successfully downloaded torch
Collecting torchvision==0.9.1+cu111
  File was already downloaded /home/edmorris/.clearml/pip-download-cache/cu111/torchvision-0.9.1+cu111-cp38-cp38-linux_x86_64.whl
Successfully downloaded torchvision
Collecting Pillow==8.2.0
  Using cached Pillow-8.2.0-cp38-cp38-manylinux1_x86_64.whl (3.0 MB)
Collecting captum==0.3.1
  Using cached captum-0.3.1-py3-none-any.whl (4.4 MB)
Processing /home/edmorris/.cache/pip/wheels/59/1b/52/0dea905f8278d5514dc4d0be5e251967f8681670cadd3dca89/imutils-0.5.4-py3-none-any.whl

1621437391868 ecm-clearml-compute-gpu-001:0 DEBUG Collecting matplotlib==3.3.4
  Using cached matplotlib-3.3.4-cp38-cp38-manylinux1_x86_64.whl (11.6 MB)
Requirement already satisfied: numpy==1.20.2 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from -r /tmp/cached-reqsypd2ylb1.txt (line 11)) (1.20.2)
Collecting pandas==1.2.4
  Using cached pandas-1.2.4-cp38-cp38-manylinux1_x86_64.whl (9.7 MB)
Collecting pytorch_ignite==0.4.4
  Using cached pytorch_ignite-0.4.4-py3-none-any.whl (200 kB)
Collecting pytorchcv==0.0.65
  Using cached pytorchcv-0.0.65-py2.py3-none-any.whl (527 kB)
Collecting scikit_image==0.18.1
  Using cached scikit_image-0.18.1-cp38-cp38-manylinux1_x86_64.whl (30.2 MB)
Collecting tensorboard==2.4.1
  Using cached tensorboard-2.4.1-py3-none-any.whl (10.6 MB)
Processing /home/edmorris/.clearml/pip-download-cache/cu111/torch-1.8.1+cu111-cp38-cp38-linux_x86_64.whl

1621437422073 ecm-clearml-compute-gpu-001:0 DEBUG Collecting torch_lucent==0.1.7
  Using cached torch_lucent-0.1.7-py3-none-any.whl (46 kB)
Processing /home/edmorris/.clearml/pip-download-cache/cu111/torchvision-0.9.1+cu111-cp38-cp38-linux_x86_64.whl
Collecting yacs==0.1.8
  Using cached yacs-0.1.8-py3-none-any.whl (14 kB)
Collecting clearml==1.0.2
  Using cached clearml-1.0.2-py2.py3-none-any.whl (990 kB)
Collecting cycler>=0.10
  Using cached cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)
Collecting kiwisolver>=1.0.1
  Using cached kiwisolver-1.3.1-cp38-cp38-manylinux1_x86_64.whl (1.2 MB)
Collecting pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting python-dateutil>=2.1
  Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting pytz>=2017.3
  Using cached pytz-2021.1-py2.py3-none-any.whl (510 kB)
Collecting requests
  Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting imageio>=2.3.0
  Using cached imageio-2.9.0-py3-none-any.whl (3.3 MB)
Collecting tifffile>=2019.7.26
  Using cached tifffile-2021.4.8-py3-none-any.whl (165 kB)
Collecting scipy>=1.0.1
  Using cached scipy-1.6.3-cp38-cp38-manylinux1_x86_64.whl (27.2 MB)
Collecting networkx>=2.0
  Using cached networkx-2.5.1-py3-none-any.whl (1.6 MB)
Collecting PyWavelets>=1.1.1
  Using cached PyWavelets-1.1.1-cp38-cp38-manylinux1_x86_64.whl (4.4 MB)
Collecting google-auth<2,>=1.6.3
  Using cached google_auth-1.30.0-py2.py3-none-any.whl (146 kB)

1621437427113 ecm-clearml-compute-gpu-001:0 DEBUG Collecting google-auth-oauthlib<0.5,>=0.4.1
  Using cached google_auth_oauthlib-0.4.4-py2.py3-none-any.whl (18 kB)

1621437437254 ecm-clearml-compute-gpu-001:0 DEBUG Collecting protobuf>=3.6.0
  Using cached protobuf-3.17.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB)

1621437442312 ecm-clearml-compute-gpu-001:0 DEBUG Collecting six>=1.10.0
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)

1621437457446 ecm-clearml-compute-gpu-001:0 DEBUG Collecting grpcio>=1.24.3
  Using cached grpcio-1.37.1-cp38-cp38-manylinux2014_x86_64.whl (4.2 MB)
Collecting tensorboard-plugin-wit>=1.6.0
  Using cached tensorboard_plugin_wit-1.8.0-py3-none-any.whl (781 kB)

1621437462533 ecm-clearml-compute-gpu-001:0 DEBUG Collecting werkzeug>=0.11.15
  Using cached Werkzeug-2.0.1-py3-none-any.whl (288 kB)
Requirement already satisfied: wheel>=0.26; python_version >= "3" in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from tensorboard==2.4.1->-r /tmp/cached-reqsypd2ylb1.txt (line 16)) (0.36.2)
Requirement already satisfied: setuptools>=41.0.0 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from tensorboard==2.4.1->-r /tmp/cached-reqsypd2ylb1.txt (line 16)) (56.0.0)
Collecting markdown>=2.6.8
  Using cached Markdown-3.3.4-py3-none-any.whl (97 kB)
Collecting absl-py>=0.4
  Using cached absl_py-0.12.0-py3-none-any.whl (129 kB)
Collecting typing-extensions
  Using cached typing_extensions-<IP_ADDRESS>-py3-none-any.whl (26 kB)
Processing /home/edmorris/.cache/pip/wheels/8e/70/28/3d6ccd6e315f65f245da085482a2e1c7d14b90b30f239e2cf4/future-0.18.2-py3-none-any.whl
Collecting pytest
  Using cached pytest-6.2.4-py3-none-any.whl (280 kB)
Collecting scikit-learn
  Using cached scikit_learn-0.24.2-cp38-cp38-manylinux2010_x86_64.whl (24.9 MB)
Collecting coveralls
  Using cached coveralls-3.0.1-py2.py3-none-any.whl (13 kB)
Collecting decorator
  Using cached decorator-5.0.9-py3-none-any.whl (8.9 kB)
Collecting pytest-mock
  Using cached pytest_mock-3.6.1-py3-none-any.whl (12 kB)
Collecting coverage
  Using cached coverage-5.5-cp38-cp38-manylinux2010_x86_64.whl (245 kB)
Collecting kornia<=0.4.1
  Using cached kornia-0.4.1-py2.py3-none-any.whl (225 kB)
Collecting tqdm
  Using cached tqdm-4.60.0-py2.py3-none-any.whl (75 kB)
Collecting ipython
  Using cached ipython-7.23.1-py3-none-any.whl (785 kB)
Collecting PyYAML
  Using cached PyYAML-5.4.1-cp38-cp38-manylinux1_x86_64.whl (662 kB)
Collecting humanfriendly>=2.1
  Using cached humanfriendly-9.1-py2.py3-none-any.whl (86 kB)

1621437467650 ecm-clearml-compute-gpu-001:0 DEBUG Collecting psutil>=3.4.2
  Using cached psutil-5.8.0-cp38-cp38-manylinux2010_x86_64.whl (296 kB)
Collecting urllib3>=1.21.1
  Using cached urllib3-1.26.4-py2.py3-none-any.whl (153 kB)
Collecting furl>=2.0.0
  Using cached furl-2.1.2-py2.py3-none-any.whl (20 kB)
Collecting pathlib2>=2.3.0
  Using cached pathlib2-2.3.5-py2.py3-none-any.whl (18 kB)
Collecting jsonschema>=2.6.0
  Using cached jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
Collecting attrs>=18.0
  Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting pyjwt<3.0.0,>=1.6.4
  Using cached PyJWT-2.1.0-py3-none-any.whl (16 kB)
Collecting chardet<5,>=3.0.2
  Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
Collecting idna<3,>=2.5
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting rsa<5,>=3.1.4; python_version >= "3.6"
  Using cached rsa-4.7.2-py3-none-any.whl (34 kB)
Collecting pyasn1-modules>=0.2.1
  Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting cachetools<5.0,>=2.0.0
  Using cached cachetools-4.2.2-py3-none-any.whl (11 kB)
Collecting requests-oauthlib>=0.7.0
  Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting iniconfig
  Using cached iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
Collecting pluggy<1.0.0a1,>=0.12
  Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting py>=1.8.2
  Using cached py-1.10.0-py2.py3-none-any.whl (97 kB)
Collecting toml
  Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting packaging
  Using cached packaging-20.9-py2.py3-none-any.whl (40 kB)
Collecting joblib>=0.11
  Using cached joblib-1.0.1-py3-none-any.whl (303 kB)
Collecting threadpoolctl>=2.0.0
  Using cached threadpoolctl-2.1.0-py3-none-any.whl (12 kB)
Processing /home/edmorris/.cache/pip/wheels/56/ea/58/ead137b087d9e326852a851351d1debf4ada529b6ac0ec4e8c/docopt-0.6.2-py2.py3-none-any.whl
Collecting pygments
  Using cached Pygments-2.9.0-py3-none-any.whl (1.0 MB)
Collecting backcall
  Using cached backcall-0.2.0-py2.py3-none-any.whl (11 kB)
Collecting pickleshare
  Using cached pickleshare-0.7.5-py2.py3-none-any.whl (6.9 kB)
Collecting matplotlib-inline
  Using cached matplotlib_inline-0.1.2-py3-none-any.whl (8.2 kB)
Collecting jedi>=0.16
  Using cached jedi-0.18.0-py2.py3-none-any.whl (1.4 MB)
Collecting pexpect>4.3; sys_platform != "win32"
  Using cached pexpect-4.8.0-py2.py3-none-any.whl (59 kB)
Collecting prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0
  Using cached prompt_toolkit-3.0.18-py3-none-any.whl (367 kB)
Collecting traitlets>=4.2
  Using cached traitlets-5.0.5-py3-none-any.whl (100 kB)
Collecting orderedmultidict>=1.0.1
  Using cached orderedmultidict-1.0.1-py2.py3-none-any.whl (11 kB)
Processing /home/edmorris/.cache/pip/wheels/3d/22/08/7042eb6309c650c7b53615d5df5cc61f1ea9680e7edd3a08d2/pyrsistent-0.17.3-cp38-cp38-linux_x86_64.whl
Collecting pyasn1>=0.1.3
  Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting oauthlib>=3.0.0
  Using cached oauthlib-3.1.0-py2.py3-none-any.whl (147 kB)
Collecting parso<0.9.0,>=0.8.0
  Using cached parso-0.8.2-py2.py3-none-any.whl (94 kB)
Collecting ptyprocess>=0.5
  Using cached ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)
Collecting wcwidth
  Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting ipython-genutils
  Using cached ipython_genutils-0.2.0-py2.py3-none-any.whl (26 kB)
ERROR: networkx 2.5.1 has requirement decorator<5,>=4.3, but you'll have decorator 5.0.9 which is incompatible.
Installing collected packages: Pillow, six, cycler, kiwisolver, pyparsing, python-dateutil, matplotlib, typing-extensions, torch, captum, imutils, pytz, pandas, pytorch-ignite, urllib3, chardet, certifi, idna, requests, pytorchcv, imageio, tifffile, scipy, decorator, networkx, PyWavelets, scikit-image, pyasn1, rsa, pyasn1-modules, cachetools, google-auth, oauthlib, requests-oauthlib, google-auth-oauthlib, protobuf, grpcio, tensorboard-plugin-wit, werkzeug, markdown, absl-py, tensorboard, future, iniconfig, pluggy, py, toml, attrs, packaging, pytest, joblib, threadpoolctl, scikit-learn, docopt, coverage, coveralls, pytest-mock, kornia, tqdm, torchvision, pygments, backcall, pickleshare, ipython-genutils, traitlets, matplotlib-inline, parso, jedi, ptyprocess, pexpect, wcwidth, prompt-toolkit, ipython, torch-lucent, PyYAML, yacs, humanfriendly, psutil, orderedmultidict, furl, pathlib2, pyrsistent, jsonschema, pyjwt, clearml

1621437522917 ecm-clearml-compute-gpu-001:0 DEBUG Successfully installed Pillow-8.2.0 PyWavelets-1.1.1 PyYAML-5.4.1 absl-py-0.12.0 attrs-21.2.0 backcall-0.2.0 cachetools-4.2.2 captum-0.3.1 certifi-2020.12.5 chardet-4.0.0 clearml-1.0.2 coverage-5.5 coveralls-3.0.1 cycler-0.10.0 decorator-5.0.9 docopt-0.6.2 furl-2.1.2 future-0.18.2 google-auth-1.30.0 google-auth-oauthlib-0.4.4 grpcio-1.37.1 humanfriendly-9.1 idna-2.10 imageio-2.9.0 imutils-0.5.4 iniconfig-1.1.1 ipython-7.23.1 ipython-genutils-0.2.0 jedi-0.18.0 joblib-1.0.1 jsonschema-3.2.0 kiwisolver-1.3.1 kornia-0.4.1 markdown-3.3.4 matplotlib-3.3.4 matplotlib-inline-0.1.2 networkx-2.5.1 oauthlib-3.1.0 orderedmultidict-1.0.1 packaging-20.9 pandas-1.2.4 parso-0.8.2 pathlib2-2.3.5 pexpect-4.8.0 pickleshare-0.7.5 pluggy-0.13.1 prompt-toolkit-3.0.18 protobuf-3.17.0 psutil-5.8.0 ptyprocess-0.7.0 py-1.10.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pygments-2.9.0 pyjwt-2.1.0 pyparsing-2.4.7 pyrsistent-0.17.3 pytest-6.2.4 pytest-mock-3.6.1 python-dateutil-2.8.1 pytorch-ignite-0.4.4 pytorchcv-0.0.65 pytz-2021.1 requests-2.25.1 requests-oauthlib-1.3.0 rsa-4.7.2 scikit-image-0.18.1 scikit-learn-0.24.2 scipy-1.6.3 six-1.16.0 tensorboard-2.4.1 tensorboard-plugin-wit-1.8.0 threadpoolctl-2.1.0 tifffile-2021.4.8 toml-0.10.2 torch-1.8.1+cu111 torch-lucent-0.1.7 torchvision-0.9.1+cu111 tqdm-4.60.0 traitlets-5.0.5 typing-extensions-<IP_ADDRESS> urllib3-1.26.4 wcwidth-0.2.5 werkzeug-2.0.1 yacs-0.1.8
Processing ./cub_tools
Requirement already satisfied: imutils in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from cub-tools==1.0.0) (0.5.4)
Requirement already satisfied: pandas in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from cub-tools==1.0.0) (1.2.4)
Requirement already satisfied: matplotlib in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from cub-tools==1.0.0) (3.3.4)
Requirement already satisfied: numpy in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from cub-tools==1.0.0) (1.20.2)
Requirement already satisfied: torch-lucent in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from cub-tools==1.0.0) (0.1.7)
Requirement already satisfied: pytorchcv in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from cub-tools==1.0.0) (0.0.65)
Requirement already satisfied: scikit-image in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from cub-tools==1.0.0) (0.18.1)
Requirement already satisfied: Pillow in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from cub-tools==1.0.0) (8.2.0)
Requirement already satisfied: pytz>=2017.3 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from pandas->cub-tools==1.0.0) (2021.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from pandas->cub-tools==1.0.0) (2.8.1)
Requirement already satisfied: cycler>=0.10 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from matplotlib->cub-tools==1.0.0) (0.10.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from matplotlib->cub-tools==1.0.0) (2.4.7)
Requirement already satisfied: kiwisolver>=1.0.1 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from matplotlib->cub-tools==1.0.0) (1.3.1)
Requirement already satisfied: future in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from torch-lucent->cub-tools==1.0.0) (0.18.2)
Requirement already satisfied: coveralls in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from torch-lucent->cub-tools==1.0.0) (3.0.1)
Requirement already satisfied: coverage in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from torch-lucent->cub-tools==1.0.0) (5.5)
Requirement already satisfied: kornia<=0.4.1 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from torch-lucent->cub-tools==1.0.0) (0.4.1)
Requirement already satisfied: pytest in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from torch-lucent->cub-tools==1.0.0) (6.2.4)

1621437528026 ecm-clearml-compute-gpu-001:0 DEBUG Requirement already satisfied: pytest-mock in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from torch-lucent->cub-tools==1.0.0) (3.6.1)
Requirement already satisfied: scikit-learn in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from torch-lucent->cub-tools==1.0.0) (0.24.2)
Requirement already satisfied: torchvision in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from torch-lucent->cub-tools==1.0.0) (0.9.1+cu111)
Requirement already satisfied: decorator in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from torch-lucent->cub-tools==1.0.0) (5.0.9)
Requirement already satisfied: tqdm in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from torch-lucent->cub-tools==1.0.0) (4.60.0)
Requirement already satisfied: ipython in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from torch-lucent->cub-tools==1.0.0) (7.23.1)
Requirement already satisfied: torch>=1.5.0 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from torch-lucent->cub-tools==1.0.0) (1.8.1+cu111)
Requirement already satisfied: requests in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from pytorchcv->cub-tools==1.0.0) (2.25.1)
Requirement already satisfied: imageio>=2.3.0 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from scikit-image->cub-tools==1.0.0) (2.9.0)
Requirement already satisfied: PyWavelets>=1.1.1 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from scikit-image->cub-tools==1.0.0) (1.1.1)
Requirement already satisfied: tifffile>=2019.7.26 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from scikit-image->cub-tools==1.0.0) (2021.4.8)
Requirement already satisfied: networkx>=2.0 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from scikit-image->cub-tools==1.0.0) (2.5.1)
Requirement already satisfied: scipy>=1.0.1 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from scikit-image->cub-tools==1.0.0) (1.6.3)
Requirement already satisfied: six>=1.5 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas->cub-tools==1.0.0) (1.16.0)
Requirement already satisfied: docopt>=0.6.1 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from coveralls->torch-lucent->cub-tools==1.0.0) (0.6.2)
Requirement already satisfied: attrs>=19.2.0 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from pytest->torch-lucent->cub-tools==1.0.0) (21.2.0)
Requirement already satisfied: toml in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from pytest->torch-lucent->cub-tools==1.0.0) (0.10.2)
Requirement already satisfied: pluggy<1.0.0a1,>=0.12 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from pytest->torch-lucent->cub-tools==1.0.0) (0.13.1)
Requirement already satisfied: iniconfig in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from pytest->torch-lucent->cub-tools==1.0.0) (1.1.1)
Requirement already satisfied: py>=1.8.2 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from pytest->torch-lucent->cub-tools==1.0.0) (1.10.0)
Requirement already satisfied: packaging in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from pytest->torch-lucent->cub-tools==1.0.0) (20.9)
Requirement already satisfied: joblib>=0.11 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from scikit-learn->torch-lucent->cub-tools==1.0.0) (1.0.1)
Requirement already satisfied: threadpoolctl>=2.0.0 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from scikit-learn->torch-lucent->cub-tools==1.0.0) (2.1.0)
Requirement already satisfied: pexpect>4.3; sys_platform != "win32" in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from ipython->torch-lucent->cub-tools==1.0.0) (4.8.0)
Requirement already satisfied: pickleshare in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from ipython->torch-lucent->cub-tools==1.0.0) (0.7.5)
Requirement already satisfied: matplotlib-inline in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from ipython->torch-lucent->cub-tools==1.0.0) (0.1.2)
Requirement already satisfied: pygments in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from ipython->torch-lucent->cub-tools==1.0.0) (2.9.0)
Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from ipython->torch-lucent->cub-tools==1.0.0) (3.0.18)
Requirement already satisfied: backcall in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from ipython->torch-lucent->cub-tools==1.0.0) (0.2.0)
Requirement already satisfied: traitlets>=4.2 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from ipython->torch-lucent->cub-tools==1.0.0) (5.0.5)
Requirement already satisfied: setuptools>=18.5 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from ipython->torch-lucent->cub-tools==1.0.0) (56.0.0)
Requirement already satisfied: jedi>=0.16 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from ipython->torch-lucent->cub-tools==1.0.0) (0.18.0)
Requirement already satisfied: typing-extensions in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from torch>=1.5.0->torch-lucent->cub-tools==1.0.0) (<IP_ADDRESS>)
Requirement already satisfied: certifi>=2017.4.17 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from requests->pytorchcv->cub-tools==1.0.0) (2020.12.5)
Requirement already satisfied: chardet<5,>=3.0.2 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from requests->pytorchcv->cub-tools==1.0.0) (4.0.0)
Requirement already satisfied: idna<3,>=2.5 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from requests->pytorchcv->cub-tools==1.0.0) (2.10)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from requests->pytorchcv->cub-tools==1.0.0) (1.26.4)
Requirement already satisfied: ptyprocess>=0.5 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from pexpect>4.3; sys_platform != "win32"->ipython->torch-lucent->cub-tools==1.0.0) (0.7.0)
Requirement already satisfied: wcwidth in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython->torch-lucent->cub-tools==1.0.0) (0.2.5)
Requirement already satisfied: ipython-genutils in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from traitlets>=4.2->ipython->torch-lucent->cub-tools==1.0.0) (0.2.0)
Requirement already satisfied: parso<0.9.0,>=0.8.0 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from jedi>=0.16->ipython->torch-lucent->cub-tools==1.0.0) (0.8.2)
Building wheels for collected packages: cub-tools
  Building wheel for cub-tools (setup.py): started
  Building wheel for cub-tools (setup.py): finished with status 'done'
  Created wheel for cub-tools: filename=cub_tools-1.0.0-py3-none-any.whl size=23095 sha256=6b03842a7c470cab541b2ce3fba09ac1e45d0caf8c626e27ecc467c261809152
  Stored in directory: /tmp/pip-ephem-wheel-cache-jrjdo6km/wheels/57/1d/f4/bf063c8201c0f9be0ad6c491d4653eec58dbde40f0961fc339
Successfully built cub-tools
Installing collected packages: cub-tools
Successfully installed cub-tools-1.0.0
Replacing original pip vcs 'git+https://github.com/rwightman/pytorch-image-models.git' with<EMAIL_ADDRESS>Collecting<EMAIL_ADDRESS>  Cloning<EMAIL_ADDRESS>to /tmp/pip-req-build-jcs909cs
  Running command git clone -q<EMAIL_ADDRESS>/tmp/pip-req-build-jcs909cs
Requirement already satisfied: torch>=1.4 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from timm==0.4.9) (1.8.1+cu111)
Requirement already satisfied: torchvision in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from timm==0.4.9) (0.9.1+cu111)
Requirement already satisfied: typing-extensions in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from torch>=1.4->timm==0.4.9) (<IP_ADDRESS>)
Requirement already satisfied: numpy in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from torch>=1.4->timm==0.4.9) (1.20.2)
Requirement already satisfied: pillow>=4.1.1 in /home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from torchvision->timm==0.4.9) (8.2.0)
Building wheels for collected packages: timm
  Building wheel for timm (setup.py): started

1621437535859 ecm-clearml-compute-gpu-001:0 DEBUG   Building wheel for timm (setup.py): finished with status 'done'
  Created wheel for timm: filename=timm-0.4.9-py3-none-any.whl size=346137 sha256=f487f72a8c6bcee4a149cfd73ed2ade3fd1b326d78d0b6874d867575c223fee9
  Stored in directory: /tmp/pip-ephem-wheel-cache-ev14df7u/wheels/91/92/6f/483882dce58f372566b25305bc184b4e8235968437833cff0d
Successfully built timm
Installing collected packages: timm

1621437546070 ecm-clearml-compute-gpu-001:0 DEBUG Successfully installed timm-0.4.9
Adding venv into cache: /home/edmorris/.clearml/venvs-builds/3.8
Running task id [1fc27ce7118542ea8845d8963a71bff7]:
[scripts]$ /home/edmorris/.clearml/venvs-builds/3.8/bin/python -u train_clearml_pytorch_ignite_caltech_birds.py
Summary - installed python packages:
pip:
- absl-py==0.12.0
- attrs==21.2.0
- backcall==0.2.0
- cachetools==4.2.2
- captum==0.3.1
- certifi==2020.12.5
- chardet==4.0.0
- clearml==1.0.2
- coverage==5.5
- coveralls==3.0.1
- cycler==0.10.0
- Cython==0.29.23
- decorator==5.0.9
- docopt==0.6.2
- furl==2.1.2
- future==0.18.2
- google-auth==1.30.0
- google-auth-oauthlib==0.4.4
- grpcio==1.37.1
- humanfriendly==9.1
- idna==2.10
- imageio==2.9.0
- imutils==0.5.4
- iniconfig==1.1.1
- ipython==7.23.1
- ipython-genutils==0.2.0
- jedi==0.18.0
- joblib==1.0.1
- jsonschema==3.2.0
- kiwisolver==1.3.1
- kornia==0.4.1
- Markdown==3.3.4
- matplotlib==3.3.4
- matplotlib-inline==0.1.2
- networkx==2.5.1
- numpy==1.20.2
- oauthlib==3.1.0
- orderedmultidict==1.0.1
- packaging==20.9
- pandas==1.2.4
- parso==0.8.2
- pathlib2==2.3.5
- pexpect==4.8.0
- pickleshare==0.7.5
- Pillow==8.2.0
- pluggy==0.13.1
- prompt-toolkit==3.0.18
- protobuf==3.17.0
- psutil==5.8.0
- ptyprocess==0.7.0
- py==1.10.0
- pyasn1==0.4.8
- pyasn1-modules==0.2.8
- Pygments==2.9.0
- PyJWT==2.1.0
- pyparsing==2.4.7
- pyrsistent==0.17.3
- pytest==6.2.4
- pytest-mock==3.6.1
- python-dateutil==2.8.1
- pytorch-ignite==0.4.4
- pytorchcv==0.0.65
- pytz==2021.1
- PyWavelets==1.1.1
- PyYAML==5.4.1
- requests==2.25.1
- requests-oauthlib==1.3.0
- rsa==4.7.2
- scikit-image==0.18.1
- scikit-learn==0.24.2
- scipy==1.6.3
- six==1.16.0
- tensorboard==2.4.1
- tensorboard-plugin-wit==1.8.0
- threadpoolctl==2.1.0
- tifffile==2021.4.8
- timm @ git+https://github.com/rwightman/pytorch-image-models.git@e7f0db866412b9ae61332c205270c9fc0ef5083c
- toml==0.10.2
- 'torch==1.8.1 # https://download.pytorch.org/whl/cu111/torch-1.8.1%2Bcu111-cp38-cp38-linux_x86_64.whl'
- torch-lucent==0.1.7
- 'torchvision==0.9.1 # https://download.pytorch.org/whl/cu111/torchvision-0.9.1%2Bcu111-cp38-cp38-linux_x86_64.whl'
- tqdm==4.60.0
- traitlets==5.0.5
- typing-extensions==<IP_ADDRESS>
- urllib3==1.26.4
- wcwidth==0.2.5
- Werkzeug==2.0.1
- yacs==0.1.8
- ./cub_tools

Environment setup completed successfully

Starting Task Execution:

usage: train_clearml_pytorch_ignite_caltech_birds.py [-h] [--config FILE]
                                                     [--opts ...]

PyTorch Image Classification Trainer - Ed Morris (c) 2021

optional arguments:
  -h, --help     show this help message and exit
  --config FILE  Path and name of configuration file for training. Should be a
                 .yaml file.
  --opts ...     Modify config options using the command-line 'KEY VALUE'
                 pairs

1621437551101 ecm-clearml-compute-gpu-001:0 DEBUG ClearML results page: http://ecm-clearml-server-001.westeurope.cloudapp.azure.com:8080/projects/30034b3199e24123896c8eff9bf16d29/experiments/1fc27ce7118542ea8845d8963a71bff7/output/log
{'MODEL.MODEL_LIBRARY': 'torchvision', 'MODEL.MODEL_NAME': 'resnet34', 'MODEL.PRETRAINED': True, 'MODEL.WITH_AMP': False, 'MODEL.WITH_GRAD_SCALE': False, 'TRAIN.BATCH_SIZE': 16, 'TRAIN.NUM_WORKERS': 4, 'TRAIN.NUM_EPOCHS': 40, 'TRAIN.LOSS.CRITERION': 'CrossEntropy', 'TRAIN.OPTIMIZER.TYPE': 'SGD', 'TRAIN.OPTIMIZER.PARAMS.lr': 0.001, 'TRAIN.OPTIMIZER.PARAMS.momentum': 0.9, 'TRAIN.OPTIMIZER.PARAMS.nesterov': True, 'TRAIN.SCHEDULER.TYPE': 'StepLR', 'TRAIN.SCHEDULER.PARAMS.step_size': 7, 'TRAIN.SCHEDULER.PARAMS.gamma': 0.1, 'EARLY_STOPPING_PATIENCE': 5, 'DIRS.ROOT_DIR': '/home/edmorris/projects/image_classification/caltech_birds', 'DIRS.WORKING_DIR': 'models/classification', 'DIRS.CLEAN_UP': True, 'DATA.DATA_DIR': 'data/images', 'DATA.TRAIN_DIR': 'train', 'DATA.TEST_DIR': 'test', 'DATA.NUM_CLASSES': 200, 'DATA.TRANSFORMS.TYPE': 'default', 'DATA.TRANSFORMS.PARAMS.DEFAULT.img_crop_size': 224, 'DATA.TRANSFORMS.PARAMS.DEFAULT.img_resize': 256, 'DATA.TRANSFORMS.PARAMS.AGGRESIVE.type': 'all', 'DATA.TRANSFORMS.PARAMS.AGGRESIVE.persp_distortion_scale': 0.25, 'DATA.TRANSFORMS.PARAMS.AGGRESIVE.rotation_range': (-10.0, 10.0), 'SYSTEM.LOG_HISTORY': True}
[INFO] Getting a local copy of the CUB200 birds datasets
[INFO] Default location of training dataset:: http://ecm-clearml-server-001.westeurope.cloudapp.azure.com:8081
[INFO] Default location of training dataset:: /home/edmorris/.clearml/cache/storage_manager/datasets/ds_0ccff21334e84b3d8e0618c5f1734cc8
[INFO] Default location of testing dataset:: http://ecm-clearml-server-001.westeurope.cloudapp.azure.com:8081
[INFO] Default location of testing dataset:: /home/edmorris/.clearml/cache/storage_manager/datasets/ds_b435c4ffda374bca83d9a746137dc3ca
[INFO] Task output destination:: None
[INFO] Final parameter list passed to Trainer object:: ['MODEL.MODEL_LIBRARY', 'torchvision', 'MODEL.MODEL_NAME', 'resnet34', 'MODEL.PRETRAINED', True, 'MODEL.WITH_AMP', False, 'MODEL.WITH_GRAD_SCALE', False, 'TRAIN.BATCH_SIZE', 16, 'TRAIN.NUM_WORKERS', 4, 'TRAIN.NUM_EPOCHS', 40, 'TRAIN.LOSS.CRITERION', 'CrossEntropy', 'TRAIN.OPTIMIZER.TYPE', 'SGD', 'TRAIN.OPTIMIZER.PARAMS.lr', 0.001, 'TRAIN.OPTIMIZER.PARAMS.momentum', 0.9, 'TRAIN.OPTIMIZER.PARAMS.nesterov', True, 'TRAIN.SCHEDULER.TYPE', 'StepLR', 'TRAIN.SCHEDULER.PARAMS.step_size', 7, 'TRAIN.SCHEDULER.PARAMS.gamma', 0.1, 'EARLY_STOPPING_PATIENCE', 5, 'DIRS.ROOT_DIR', '/home/edmorris/projects/image_classification/caltech_birds', 'DIRS.WORKING_DIR', 'models/classification', 'DIRS.CLEAN_UP', True, 'DATA.DATA_DIR', 'data/images', 'DATA.TRAIN_DIR', 'train', 'DATA.TEST_DIR', 'test', 'DATA.NUM_CLASSES', 200, 'DATA.TRANSFORMS.TYPE', 'default', 'DATA.TRANSFORMS.PARAMS.DEFAULT.img_crop_size', 224, 'DATA.TRANSFORMS.PARAMS.DEFAULT.img_resize', 256, 'DATA.TRANSFORMS.PARAMS.AGGRESIVE.type', 'all', 'DATA.TRANSFORMS.PARAMS.AGGRESIVE.persp_distortion_scale', 0.25, 'DATA.TRANSFORMS.PARAMS.AGGRESIVE.rotation_range', (-10.0, 10.0), 'SYSTEM.LOG_HISTORY', True, 'DIRS.ROOT_DIR', '', 'DATA.DATA_DIR', '/home/edmorris/.clearml/cache/storage_manager/datasets', 'DATA.TRAIN_DIR', 'ds_0ccff21334e84b3d8e0618c5f1734cc8', 'DATA.TEST_DIR', 'ds_b435c4ffda374bca83d9a746137dc3ca', 'DIRS.WORKING_DIR', '/home/edmorris/.clearml/cache/1fc27ce7118542ea8845d8963a71bff7']
[INFO] Parameters Override:: ['MODEL.MODEL_LIBRARY', 'torchvision', 'MODEL.MODEL_NAME', 'resnet34', 'MODEL.PRETRAINED', True, 'MODEL.WITH_AMP', False, 'MODEL.WITH_GRAD_SCALE', False, 'TRAIN.BATCH_SIZE', 16, 'TRAIN.NUM_WORKERS', 4, 'TRAIN.NUM_EPOCHS', 40, 'TRAIN.LOSS.CRITERION', 'CrossEntropy', 'TRAIN.OPTIMIZER.TYPE', 'SGD', 'TRAIN.OPTIMIZER.PARAMS.lr', 0.001, 'TRAIN.OPTIMIZER.PARAMS.momentum', 0.9, 'TRAIN.OPTIMIZER.PARAMS.nesterov', True, 'TRAIN.SCHEDULER.TYPE', 'StepLR', 'TRAIN.SCHEDULER.PARAMS.step_size', 7, 'TRAIN.SCHEDULER.PARAMS.gamma', 0.1, 'EARLY_STOPPING_PATIENCE', 5, 'DIRS.ROOT_DIR', '/home/edmorris/projects/image_classification/caltech_birds', 'DIRS.WORKING_DIR', 'models/classification', 'DIRS.CLEAN_UP', True, 'DATA.DATA_DIR', 'data/images', 'DATA.TRAIN_DIR', 'train', 'DATA.TEST_DIR', 'test', 'DATA.NUM_CLASSES', 200, 'DATA.TRANSFORMS.TYPE', 'default', 'DATA.TRANSFORMS.PARAMS.DEFAULT.img_crop_size', 224, 'DATA.TRANSFORMS.PARAMS.DEFAULT.img_resize', 256, 'DATA.TRANSFORMS.PARAMS.AGGRESIVE.type', 'all', 'DATA.TRANSFORMS.PARAMS.AGGRESIVE.persp_distortion_scale', 0.25, 'DATA.TRANSFORMS.PARAMS.AGGRESIVE.rotation_range', (-10.0, 10.0), 'SYSTEM.LOG_HISTORY', True, 'DIRS.ROOT_DIR', '', 'DATA.DATA_DIR', '/home/edmorris/.clearml/cache/storage_manager/datasets', 'DATA.TRAIN_DIR', 'ds_0ccff21334e84b3d8e0618c5f1734cc8', 'DATA.TEST_DIR', 'ds_b435c4ffda374bca83d9a746137dc3ca', 'DIRS.WORKING_DIR', '/home/edmorris/.clearml/cache/1fc27ce7118542ea8845d8963a71bff7']
DATA:
  DATA_DIR: /home/edmorris/.clearml/cache/storage_manager/datasets
  NUM_CLASSES: 200
  TEST_DIR: ds_b435c4ffda374bca83d9a746137dc3ca
  TRAIN_DIR: ds_0ccff21334e84b3d8e0618c5f1734cc8
  TRANSFORMS:
    PARAMS:
      AGGRESIVE:
        persp_distortion_scale: 0.25
        rotation_range: (-10.0, 10.0)
        type: all
      DEFAULT:
        img_crop_size: 224
        img_resize: 256
    TYPE: default
DIRS:
  CLEAN_UP: True
  ROOT_DIR:
  WORKING_DIR: /home/edmorris/.clearml/cache/1fc27ce7118542ea8845d8963a71bff7/ignite_resnet34
EARLY_STOPPING_PATIENCE: 5
MODEL:
  MODEL_LIBRARY: torchvision
  MODEL_NAME: resnet34
  PRETRAINED: True
  WITH_AMP: False
  WITH_GRAD_SCALE: False
SYSTEM:
  LOG_HISTORY: True
TRAIN:
  BATCH_SIZE: 16
  LOSS:
    CRITERION: CrossEntropy
  NUM_EPOCHS: 40
  NUM_WORKERS: 4
  OPTIMIZER:
    PARAMS:
      lr: 0.001
      momentum: 0.9
      nesterov: True
    TYPE: SGD
  SCHEDULER:
    PARAMS:
      gamma: 0.1
      step_size: 7
    TYPE: StepLR
[INFO] Creating data transforms...
[INFO] Creating data loaders...
***********************************************
**            DATASET SUMMARY                **
***********************************************
train  size::  5994  images
test  size::  5794  images
Number of classes::  200
***********************************************
[INFO] Created data loaders.
[INFO] Creating the model...
2021-05-19 15:19:07,332 - clearml.model - INFO - Selected model id: 8df52efca2684e5f8b727fa928623a82
[INFO] Successfully created model and pushed it to the device cuda:0
[INFO] Creating optimizer...
[INFO] Successfully created optimizer object.
[INFO] Successfully created learning rate scheduler object.
[INFO] Trainer pass OK for training.
Tensorboard Logging...done
[INFO] Creating callback functions for training loop...Early Stopping (5 epochs)...Model Checkpointing...Done
[INFO] Executing model training...

1621437621593 ecm-clearml-compute-gpu-001:0 DEBUG Epoch: 0001  TrAcc: 0.296 ValAcc: 0.005 TrPrec: 0.393 ValPrec: 0.000 TrRec: 0.296 ValRec: 0.005 TrF1: 0.262 ValF1: 0.000 TrTopK: 0.613 ValTopK: 0.026 TrLoss: 3.506 ValLoss: 5.299
Current run is terminating due to exception: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
You can try to repro this exception using the following code snippet. If that doesn't trigger the error, please include your original repro script when reporting this issue.

PyTorch Ignite training script
Below is the trainer script, which has been modified to run with clearml. A version of this script without the clearml interface has successfully trained these models on the compute server in a conda environment.
from __future__ import print_function, division
import os, pathlib

# Clear ML experiment
from clearml import Task, StorageManager, Dataset

# Local modules
from cub_tools.trainer import Ignite_Trainer
from cub_tools.args import get_parser
from cub_tools.config import get_cfg_defaults

# Get the arguments from the command line, including configuration file and any overrides.
parser = get_parser()
parser.print_help()
args = parser.parse_args()

#print('[INFO] Optional Arguments from CLI:: {}'.format(args.opts))
#if args.opts == '[]':
#    args.opts = list()
#    print('[INFO] Setting empty CLI args to an explicit empty list')

## CLEAR ML
# Tmp config load for network name
cfg = get_cfg_defaults()
cfg.merge_from_file(args.config)
# Connecting with the ClearML process
task = Task.init(project_name='Caltech Birds', task_name='Train PyTorch CNN on CUB200 using Ignite [Library: '+cfg.MODEL.MODEL_LIBRARY+', Network: '+cfg.MODEL.MODEL_NAME+']', task_type=Task.TaskTypes.training)
# Add the local python package as a requirement
task.add_requirements('./cub_tools')
task.add_requirements('git+https://github.com/rwightman/pytorch-image-models.git')
# Setup ability to add configuration parameters control.
params = {'TRAIN.NUM_EPOCHS': 20, 'TRAIN.BATCH_SIZE': 32, 'TRAIN.OPTIMIZER.PARAMS.lr': 0.001, 'TRAIN.OPTIMIZER.PARAMS.momentum': 0.9}
params = task.connect(params)  # enabling configuration override by clearml
print(params)  # printing actual configuration (after override in remote mode)
# Convert Params dictionary into a set of key value pairs in a list
params_list = []
for key in params:
    params_list.extend([key,params[key]])

# Execute task remotely
task.execute_remotely()

# Get the dataset from the clearml-server and cache locally.
print('[INFO] Getting a local copy of the CUB200 birds datasets')
# Train
train_dataset = Dataset.get(dataset_project='Caltech Birds', dataset_name='cub200_2011_train_dataset')
#train_dataset.get_mutable_local_copy(target_folder='./data/images/train')
print('[INFO] Default location of training dataset:: {}'.format(train_dataset.get_default_storage()))
train_dataset_base = train_dataset.get_local_copy()
print('[INFO] Default location of training dataset:: {}'.format(train_dataset_base))

# Test
test_dataset = Dataset.get(dataset_project='Caltech Birds', dataset_name='cub200_2011_test_dataset')
#train_dataset.get_mutable_local_copy(target_folder='./data/images/train')
print('[INFO] Default location of testing dataset:: {}'.format(test_dataset.get_default_storage()))
test_dataset_base = test_dataset.get_local_copy()
print('[INFO] Default location of testing dataset:: {}'.format(test_dataset_base))

# Amend the input data directories and output directories for remote execution
# Modify experiment root dir
params_list = params_list + ['DIRS.ROOT_DIR', '']
# Add data root dir
params_list = params_list + ['DATA.DATA_DIR', str(pathlib.PurePath(train_dataset_base).parent)]
# Add data train dir
params_list = params_list + ['DATA.TRAIN_DIR', str(pathlib.PurePath(train_dataset_base).name)]
# Add data test dir
params_list = params_list + ['DATA.TEST_DIR', str(pathlib.PurePath(test_dataset_base).name)]
# Add working dir
params_list = params_list + ['DIRS.WORKING_DIR', str(task.cache_dir)]
print('[INFO] Task output destination:: {}'.format(task.get_output_destination()))

print('[INFO] Final parameter list passed to Trainer object:: {}'.format(params_list))

# Create the trainer object
trainer = Ignite_Trainer(config=args.config, cmd_args=params_list) # NOTE: disabled cmd line argument passing but using it to pass ClearML configs.

# Setup the data transformers
print('[INFO] Creating data transforms...')
trainer.create_datatransforms()

# Setup the dataloaders
print('[INFO] Creating data loaders...')
trainer.create_dataloaders()

# Setup the model
print('[INFO] Creating the model...')
trainer.create_model()

# Setup the optimizer
print('[INFO] Creating optimizer...')
trainer.create_optimizer()

# Setup the scheduler
trainer.create_scheduler()

# Train the model
trainer.run()

Hi @ecm200
This cuDNN error: CUDNN_STATUS_EXECUTION_FAILED seems like a CUDA / CUDNN mismatch issue.
It seems from the log clearml-agent installed the correct pytorch version (based on the auto detected CUDA 11.1 version).
Is this the same setup that worked on your development machine ?
(Basically I suspect this is not a direct issue of clearml but a cuda/pytorch thing)
BTW: Running the clearml-agent would solve such issues, as you will have the ability to launch the code inside a container with the correct CUDA support.

Hi @bmartinn,
I think you're right, however I think this boils down to how you build environments on the remote machine.
When I set up my development environments on remote machines to work directly on them (i.e. not with ClearML) I tend to default to Conda for most packages and then use PIP when packages are not available on CondaCloud.
Local execution of venv created by clearml-agent
I have gone onto the remote compute resource, and executed the code locally, using the virtual environment created by the clearml-agent, and I get the same error.
Local execution in conda environment
I have also created a new conda environment on the same machine, using mostly conda to install the package dependencies as described above, and executed the exact same code as I did before, but it is now computing fine and iterating and logging as expected into clearml-server.
(py38_pytorch18) edmorris@ecm-clearml-compute-gpu-001:~/.clearml/venvs-builds/3.8/task_repository/caltech_birds.git/scripts$ python local_train_clearml_pytorch_ignite_caltech_birds.py --config configs/torchvision/resnet34_config.yaml
usage: local_train_clearml_pytorch_ignite_caltech_birds.py [-h] [--config FILE] [--opts ...]

PyTorch Image Classification Trainer - Ed Morris (c) 2021

optional arguments:
  -h, --help     show this help message and exit
  --config FILE  Path and name of configuration file for training. Should be a .yaml file.
  --opts ...     Modify config options using the command-line 'KEY VALUE' pairs
ClearML Task: overwriting (reusing) task id=ea8903a29bf443d5ab469f9c56c2a8b5
ClearML results page: http://ecm-clearml-server-001.westeurope.cloudapp.azure.com:8080/projects/30034b3199e24123896c8eff9bf16d29/experiments/ea8903a29bf443d5ab469f9c56c2a8b5/output/log
2021-05-20 09:29:55,856 - clearml.task - WARNING - Requirement ignored, Task.add_requirements() must be called before Task.init()
2021-05-20 09:29:55,861 - clearml.task - WARNING - Requirement ignored, Task.add_requirements() must be called before Task.init()
{'MODEL.MODEL_LIBRARY': 'torchvision', 'MODEL.MODEL_NAME': 'resnet34', 'MODEL.PRETRAINED': True, 'MODEL.WITH_AMP': False, 'MODEL.WITH_GRAD_SCALE': False, 'TRAIN.BATCH_SIZE': 16, 'TRAIN.NUM_WORKERS': 4, 'TRAIN.NUM_EPOCHS': 40, 'TRAIN.LOSS.CRITERION': 'CrossEntropy', 'TRAIN.OPTIMIZER.TYPE': 'SGD', 'TRAIN.OPTIMIZER.PARAMS.lr': 0.001, 'TRAIN.OPTIMIZER.PARAMS.momentum': 0.9, 'TRAIN.OPTIMIZER.PARAMS.nesterov': True, 'TRAIN.SCHEDULER.TYPE': 'StepLR', 'TRAIN.SCHEDULER.PARAMS.step_size': 7, 'TRAIN.SCHEDULER.PARAMS.gamma': 0.1, 'EARLY_STOPPING_PATIENCE': 5, 'DIRS.ROOT_DIR': '/home/edmorris/projects/image_classification/caltech_birds', 'DIRS.WORKING_DIR': 'models/classification', 'DIRS.CLEAN_UP': True, 'DATA.DATA_DIR': 'data/images', 'DATA.TRAIN_DIR': 'train', 'DATA.TEST_DIR': 'test', 'DATA.NUM_CLASSES': 200, 'DATA.TRANSFORMS.TYPE': 'default', 'DATA.TRANSFORMS.PARAMS.DEFAULT.img_crop_size': 224, 'DATA.TRANSFORMS.PARAMS.DEFAULT.img_resize': 256, 'DATA.TRANSFORMS.PARAMS.AGGRESIVE.type': 'all', 'DATA.TRANSFORMS.PARAMS.AGGRESIVE.persp_distortion_scale': 0.25, 'DATA.TRANSFORMS.PARAMS.AGGRESIVE.rotation_range': (-10.0, 10.0), 'SYSTEM.LOG_HISTORY': True}
[INFO] Getting a local copy of the CUB200 birds datasets
[INFO] Default location of training dataset:: http://ecm-clearml-server-001.westeurope.cloudapp.azure.com:8081
[INFO] Default location of training dataset:: /home/edmorris/.clearml/cache/storage_manager/datasets/ds_0ccff21334e84b3d8e0618c5f1734cc8
[INFO] Default location of testing dataset:: http://ecm-clearml-server-001.westeurope.cloudapp.azure.com:8081
[INFO] Default location of testing dataset:: /home/edmorris/.clearml/cache/storage_manager/datasets/ds_b435c4ffda374bca83d9a746137dc3ca
[INFO] Task output destination::
[INFO] Final parameter list passed to Trainer object:: ['MODEL.MODEL_LIBRARY', 'torchvision', 'MODEL.MODEL_NAME', 'resnet34', 'MODEL.PRETRAINED', True, 'MODEL.WITH_AMP', False, 'MODEL.WITH_GRAD_SCALE', False, 'TRAIN.BATCH_SIZE', 16, 'TRAIN.NUM_WORKERS', 4, 'TRAIN.NUM_EPOCHS', 40, 'TRAIN.LOSS.CRITERION', 'CrossEntropy', 'TRAIN.OPTIMIZER.TYPE', 'SGD', 'TRAIN.OPTIMIZER.PARAMS.lr', 0.001, 'TRAIN.OPTIMIZER.PARAMS.momentum', 0.9, 'TRAIN.OPTIMIZER.PARAMS.nesterov', True, 'TRAIN.SCHEDULER.TYPE', 'StepLR', 'TRAIN.SCHEDULER.PARAMS.step_size', 7, 'TRAIN.SCHEDULER.PARAMS.gamma', 0.1, 'EARLY_STOPPING_PATIENCE', 5, 'DIRS.ROOT_DIR', '/home/edmorris/projects/image_classification/caltech_birds', 'DIRS.WORKING_DIR', 'models/classification', 'DIRS.CLEAN_UP', True, 'DATA.DATA_DIR', 'data/images', 'DATA.TRAIN_DIR', 'train', 'DATA.TEST_DIR', 'test', 'DATA.NUM_CLASSES', 200, 'DATA.TRANSFORMS.TYPE', 'default', 'DATA.TRANSFORMS.PARAMS.DEFAULT.img_crop_size', 224, 'DATA.TRANSFORMS.PARAMS.DEFAULT.img_resize', 256, 'DATA.TRANSFORMS.PARAMS.AGGRESIVE.type', 'all', 'DATA.TRANSFORMS.PARAMS.AGGRESIVE.persp_distortion_scale', 0.25, 'DATA.TRANSFORMS.PARAMS.AGGRESIVE.rotation_range', (-10.0, 10.0), 'SYSTEM.LOG_HISTORY', True, 'DIRS.ROOT_DIR', '', 'DATA.DATA_DIR', '/home/edmorris/.clearml/cache/storage_manager/datasets', 'DATA.TRAIN_DIR', 'ds_0ccff21334e84b3d8e0618c5f1734cc8', 'DATA.TEST_DIR', 'ds_b435c4ffda374bca83d9a746137dc3ca', 'DIRS.WORKING_DIR', '/home/edmorris/.clearml/cache/ea8903a29bf443d5ab469f9c56c2a8b5']
[INFO] Parameters Override:: ['MODEL.MODEL_LIBRARY', 'torchvision', 'MODEL.MODEL_NAME', 'resnet34', 'MODEL.PRETRAINED', True, 'MODEL.WITH_AMP', False, 'MODEL.WITH_GRAD_SCALE', False, 'TRAIN.BATCH_SIZE', 16, 'TRAIN.NUM_WORKERS', 4, 'TRAIN.NUM_EPOCHS', 40, 'TRAIN.LOSS.CRITERION', 'CrossEntropy', 'TRAIN.OPTIMIZER.TYPE', 'SGD', 'TRAIN.OPTIMIZER.PARAMS.lr', 0.001, 'TRAIN.OPTIMIZER.PARAMS.momentum', 0.9, 'TRAIN.OPTIMIZER.PARAMS.nesterov', True, 'TRAIN.SCHEDULER.TYPE', 'StepLR', 'TRAIN.SCHEDULER.PARAMS.step_size', 7, 'TRAIN.SCHEDULER.PARAMS.gamma', 0.1, 'EARLY_STOPPING_PATIENCE', 5, 'DIRS.ROOT_DIR', '/home/edmorris/projects/image_classification/caltech_birds', 'DIRS.WORKING_DIR', 'models/classification', 'DIRS.CLEAN_UP', True, 'DATA.DATA_DIR', 'data/images', 'DATA.TRAIN_DIR', 'train', 'DATA.TEST_DIR', 'test', 'DATA.NUM_CLASSES', 200, 'DATA.TRANSFORMS.TYPE', 'default', 'DATA.TRANSFORMS.PARAMS.DEFAULT.img_crop_size', 224, 'DATA.TRANSFORMS.PARAMS.DEFAULT.img_resize', 256, 'DATA.TRANSFORMS.PARAMS.AGGRESIVE.type', 'all', 'DATA.TRANSFORMS.PARAMS.AGGRESIVE.persp_distortion_scale', 0.25, 'DATA.TRANSFORMS.PARAMS.AGGRESIVE.rotation_range', (-10.0, 10.0), 'SYSTEM.LOG_HISTORY', True, 'DIRS.ROOT_DIR', '', 'DATA.DATA_DIR', '/home/edmorris/.clearml/cache/storage_manager/datasets', 'DATA.TRAIN_DIR', 'ds_0ccff21334e84b3d8e0618c5f1734cc8', 'DATA.TEST_DIR', 'ds_b435c4ffda374bca83d9a746137dc3ca', 'DIRS.WORKING_DIR', '/home/edmorris/.clearml/cache/ea8903a29bf443d5ab469f9c56c2a8b5']
DATA:
  DATA_DIR: /home/edmorris/.clearml/cache/storage_manager/datasets
  NUM_CLASSES: 200
  TEST_DIR: ds_b435c4ffda374bca83d9a746137dc3ca
  TRAIN_DIR: ds_0ccff21334e84b3d8e0618c5f1734cc8
  TRANSFORMS:
    PARAMS:
      AGGRESIVE:
        persp_distortion_scale: 0.25
        rotation_range: (-10.0, 10.0)
        type: all
      DEFAULT:
        img_crop_size: 224
        img_resize: 256
    TYPE: default
DIRS:
  CLEAN_UP: True
  ROOT_DIR:
  WORKING_DIR: /home/edmorris/.clearml/cache/ea8903a29bf443d5ab469f9c56c2a8b5/ignite_resnet34
EARLY_STOPPING_PATIENCE: 5
MODEL:
  MODEL_LIBRARY: torchvision
  MODEL_NAME: resnet34
  PRETRAINED: True
  WITH_AMP: False
  WITH_GRAD_SCALE: False
SYSTEM:
  LOG_HISTORY: True
TRAIN:
  BATCH_SIZE: 16
  LOSS:
    CRITERION: CrossEntropy
  NUM_EPOCHS: 40
  NUM_WORKERS: 4
  OPTIMIZER:
    PARAMS:
      lr: 0.001
      momentum: 0.9
      nesterov: True
    TYPE: SGD
  SCHEDULER:
    PARAMS:
      gamma: 0.1
      step_size: 7
    TYPE: StepLR
[INFO] Creating data transforms...
[INFO] Creating data loaders...
***********************************************
**            DATASET SUMMARY                **
***********************************************
train  size::  5994  images
test  size::  5794  images
Number of classes::  200
***********************************************
[INFO] Created data loaders.
[INFO] Creating the model...
2021-05-20 09:29:57,271 - clearml.model - INFO - Selected model id: 8df52efca2684e5f8b727fa928623a82
[INFO] Successfully created model and pushed it to the device cuda:0
[INFO] Creating optimizer...
[INFO] Successfully created optimizer object.
[INFO] Successfully created learning rate scheduler object.
[INFO] Trainer pass OK for training.
Tensorboard Logging...done
[INFO] Creating callback functions for training loop...Early Stopping (5 epochs)...Model Checkpointing...Done
[INFO] Executing model training...

Questions about environment creation
So I am wondering, if you have an issue like this, where you have a package selection that is mainly in say Conda, but there are a few packages in PIP, then how can this be handled?
I could create a YAML file of the Conda environment that is now successfully running the code, and that could be used to create a conda environment, but how can this be used in conjunction with clearml to do that automatically when an experiment is cloned and executed?

@bmartinn
Update, I created another environment manually on the compute server, using CONDA to create the environment object, but then I installed all packages, including PyTorch using PIP and NOT CONDA. I made sure the versions matched those picked up by the dependency map created by clearml.
Executing the code in this environment caused the same issue as the clearml-agent created environment, as these both installed PyTorch using PIP.
So is there anyway to use both CONDA and PIP, like I do when creating environments manually, so to install most from CONDA and what isn't available using PIP?
The YAML file created by a CONDA environment creates a package list that differentiates between package sources, either CONDA or PIP. The call is as follows, and should be run inside the environment you want to get the details of:
conda env export > environment_specs.yml

This results in a YAML file as follows:
name: py38_pytorch18
channels:
  - pytorch
  - nvidia
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - absl-py=0.12.0=py38h06a4308_0
  - aiohttp=3.6.3=py38h7b6447c_0
  - async-timeout=3.0.1=py38h06a4308_0
  - attrs=21.2.0=pyhd3eb1b0_0
  - blas=1.0=mkl
  - blinker=1.4=py38h06a4308_0
  - blosc=1.19.0=hd408876_0
  - brotli=1.0.9=he6710b0_2
  - brotlipy=0.7.0=py38h27cfd23_1003
  - bzip2=1.0.8=h7b6447c_0
  - c-ares=1.17.1=h27cfd23_0
  - ca-certificates=2021.4.13=h06a4308_1
  - cachetools=4.2.2=pyhd3eb1b0_0
  - certifi=2020.12.5=py38h06a4308_0
  - cffi=1.14.5=py38h261ae71_0
  - chardet=3.0.4=py38h06a4308_1003
  - charls=2.1.0=he6710b0_2
  - click=8.0.0=pyhd3eb1b0_0
  - cloudpickle=1.6.0=py_0
  - coverage=5.5=py38h27cfd23_2
  - cryptography=3.4.7=py38hd23ed53_0
  - cudatoolkit=11.1.74=h6bb024c_0
  - cycler=0.10.0=py38_0
  - cython=0.29.23=py38h2531618_0
  - cytoolz=0.11.0=py38h7b6447c_0
  - dask-core=2021.5.0=pyhd3eb1b0_0
  - dbus=1.13.18=hb2f20db_0
  - decorator=5.0.9=pyhd3eb1b0_0
  - expat=2.3.0=h2531618_2
  - ffmpeg=4.3=hf484d3e_0
  - fontconfig=2.13.1=h6c09931_0
  - freetype=2.10.4=h5ab3b9f_0
  - fsspec=0.9.0=pyhd3eb1b0_0
  - giflib=5.1.4=h14c3975_1
  - glib=2.68.2=h36276a3_0
  - gmp=6.2.1=h2531618_2
  - gnutls=3.6.15=he1e5248_0
  - google-auth=1.30.0=pyhd3eb1b0_0
  - google-auth-oauthlib=0.4.4=pyhd3eb1b0_0
  - grpcio=1.36.1=py38h2157cd5_1
  - gst-plugins-base=1.14.0=h8213a91_2
  - gstreamer=1.14.0=h28cd5cc_2
  - icu=58.2=he6710b0_3
  - idna=2.10=pyhd3eb1b0_0
  - ignite=0.4.4=py_0
  - imagecodecs=2020.5.30=py38h567f118_1
  - imageio=2.9.0=pyhd3eb1b0_0
  - importlib-metadata=3.10.0=py38h06a4308_0
  - intel-openmp=2021.2.0=h06a4308_610
  - joblib=1.0.1=pyhd3eb1b0_0
  - jpeg=9b=h024ee3a_2
  - jxrlib=1.1=h7b6447c_2
  - kiwisolver=1.3.1=py38h2531618_0
  - lame=3.100=h7b6447c_0
  - lcms2=2.12=h3be6417_0
  - ld_impl_linux-64=2.33.1=h53a641e_7
  - libaec=1.0.4=he6710b0_1
  - libffi=3.3=he6710b0_2
  - libgcc-ng=9.1.0=hdf63c60_0
  - libgfortran-ng=7.3.0=hdf63c60_0
  - libiconv=1.15=h63c8f33_5
  - libidn2=2.3.1=h27cfd23_0
  - libpng=1.6.37=hbc83047_0
  - libprotobuf=3.14.0=h8c45485_0
  - libstdcxx-ng=9.1.0=hdf63c60_0
  - libtasn1=4.16.0=h27cfd23_0
  - libtiff=4.1.0=h2733197_1
  - libunistring=0.9.10=h27cfd23_0
  - libuuid=1.0.3=h1bed415_2
  - libuv=1.40.0=h7b6447c_0
  - libwebp=1.0.1=h8e7db2f_0
  - libxcb=1.14=h7b6447c_0
  - libxml2=2.9.10=hb55368b_3
  - libzopfli=1.0.3=he6710b0_0
  - locket=0.2.1=py38h06a4308_1
  - lz4-c=1.9.3=h2531618_0
  - markdown=3.3.4=py38h06a4308_0
  - matplotlib=3.3.4=py38h06a4308_0
  - matplotlib-base=3.3.4=py38h62a2d02_0
  - mkl=2021.2.0=h06a4308_296
  - mkl-service=2.3.0=py38h27cfd23_1
  - mkl_fft=1.3.0=py38h42c9631_2
  - mkl_random=1.2.1=py38ha9443f7_2
  - multidict=4.7.6=py38h7b6447c_1
  - ncurses=6.2=he6710b0_1
  - nettle=3.7.2=hbbd107a_1
  - networkx=2.5=py_0
  - ninja=1.10.2=hff7bd54_1
  - numpy=1.20.2=py38h2d18471_0
  - numpy-base=1.20.2=py38hfae3a4d_0
  - oauthlib=3.1.0=py_0
  - olefile=0.46=py_0
  - openh264=2.1.0=hd408876_0
  - openjpeg=2.3.0=h05c96fa_1
  - openssl=1.1.1k=h27cfd23_0
  - pandas=1.2.4=py38h2531618_0
  - partd=1.2.0=pyhd3eb1b0_0
  - pcre=8.44=he6710b0_0
  - pillow=8.2.0=py38he98fc37_0
  - pip=21.0.1=py38h06a4308_0
  - protobuf=3.14.0=py38h2531618_1
  - pyasn1=0.4.8=py_0
  - pyasn1-modules=0.2.8=py_0
  - pycparser=2.20=py_2
  - pyjwt=1.7.1=py38_0
  - pyopenssl=20.0.1=pyhd3eb1b0_1
  - pyparsing=2.4.7=pyhd3eb1b0_0
  - pyqt=5.9.2=py38h05f1152_4
  - pysocks=1.7.1=py38h06a4308_0
  - python=3.8.10=hdb3f193_7
  - python-dateutil=2.8.1=pyhd3eb1b0_0
  - pytorch=1.8.1=py3.8_cuda11.1_cudnn8.0.5_0
  - pytz=2021.1=pyhd3eb1b0_0
  - pywavelets=1.1.1=py38h7b6447c_2
  - pyyaml=5.4.1=py38h27cfd23_1
  - qt=5.9.7=h5867ecd_1
  - readline=8.1=h27cfd23_0
  - requests=2.25.1=pyhd3eb1b0_0
  - requests-oauthlib=1.3.0=py_0
  - rsa=4.7.2=pyhd3eb1b0_1
  - scikit-image=0.18.1=py38ha9443f7_0
  - scikit-learn=0.24.2=py38ha9443f7_0
  - scipy=1.6.2=py38had2a1c9_1
  - setuptools=52.0.0=py38h06a4308_0
  - sip=4.19.13=py38he6710b0_0
  - six=1.15.0=py38h06a4308_0
  - snappy=1.1.8=he6710b0_0
  - sqlite=3.35.4=hdfb4753_0
  - tensorboard=2.4.0=pyhc547734_0
  - tensorboard-plugin-wit=1.6.0=py_0
  - threadpoolctl=2.1.0=pyh5ca1d4c_0
  - tifffile=2021.3.31=pyhd3eb1b0_1
  - tk=8.6.10=hbc83047_0
  - toolz=0.11.1=pyhd3eb1b0_0
  - torchaudio=0.8.1=py38
  - torchvision=0.9.1=py38_cu111
  - tornado=6.1=py38h27cfd23_0
  - typing_extensions=<IP_ADDRESS>=pyha847dfd_0
  - urllib3=1.26.4=pyhd3eb1b0_0
  - werkzeug=1.0.1=pyhd3eb1b0_0
  - wheel=0.36.2=pyhd3eb1b0_0
  - xz=5.2.5=h7b6447c_0
  - yaml=0.2.5=h7b6447c_0
  - yarl=1.6.3=py38h27cfd23_0
  - zipp=3.4.1=pyhd3eb1b0_0
  - zlib=1.2.11=h7b6447c_3
  - zstd=1.4.9=haebb681_0
  - pip:
    - backcall==0.2.0
    - clearml==1.0.2
    - coveralls==3.0.1
    - cub-tools==1.0.0
    - docopt==0.6.2
    - furl==2.1.2
    - humanfriendly==9.1
    - imutils==0.5.4
    - iniconfig==1.1.1
    - ipython==7.23.1
    - ipython-genutils==0.2.0
    - jedi==0.18.0
    - jsonschema==3.2.0
    - kornia==0.4.1
    - matplotlib-inline==0.1.2
    - orderedmultidict==1.0.1
    - packaging==20.9
    - parso==0.8.2
    - pathlib2==2.3.5
    - pexpect==4.8.0
    - pickleshare==0.7.5
    - pluggy==0.13.1
    - prompt-toolkit==3.0.18
    - ptyprocess==0.7.0
    - py==1.10.0
    - pygments==2.9.0
    - pyrsistent==0.17.3
    - pytest==6.2.4
    - pytest-mock==3.6.1
    - pytorchcv==0.0.65
    - timm==0.4.9
    - toml==0.10.2
    - torch-lucent==0.1.8
    - tqdm==4.60.0
    - traitlets==5.0.5
    - wcwidth==0.2.5
    - yacs==0.1.8
prefix: /home/edmorris/.conda/envs/py38_pytorch18

@bmartinn
Thanks for all your tips and help.
After a lot experimenting with various clearml-agent options, including running in docker mode, I started again using the conda package manager as the virtual environment creator on the remote compute node. This time, I was able to see that the clearml-agent was able to use PIP to install additional packages, if they could not be resolved using CONDA. This means that PyTorch was installed using the recommended CONDA method and therefore circumented the issues found with using PIP to install PyTorch, as detailed above.
This has led to a successful creation of a training environment on the remote compute and a successful training of PyTorch models.

Great work!
BTW: If you are running the  local code with conda, you can set the agent to use conda as well [see here] .
Notice that if you are running locally with pip, the agent's conda environement will use pip to install the packages to avoid version mismatch.


The all_dthreads[i] maybe nullptr

The DEDICATED EThread can be created during ATS running.
In the eventProcessor.spawn_thread(), the n_dthreads is incremented first
then create the EThread object and put it into the array all_dthreads[].
Therefore, the all_dthreads[eventProcessor.n_dthreads] maybe nullptr.

Try to fix the autest after the PR #1969 merged

Seems a bit of overkill, but let's go with it. I'll try to work on the larger thread start up overhaul (TS-4265) once this is landed.

close this since the PR #2010 merged.


Feature Request: 希望能在网页的链接中支持obsidian url

Feature Requested
如果网页中本身含有obsidian链接类似obsidian://open?file=
希望可以跳到对应的笔记
Relevant Screenshot
No response
Checklist

[X] The feature would be useful to more users than just me.

存在风险。


Add context to partials in hugo conversion article

Thanks for this article. I was needing to do this recently and it took me a while to figure out on my own; this approach seems easier.
I would suggest that you include the . context variable when calling partials. It doesn't matter for the given example but I can see someone running into a bug down the line when they want to use site variables in their partial and don't understand why they're not available.

Aaaah, yes. Thank you!


split the wasm Context implementation to different class

Title: split the wasm Context implementation to different class
In the WASM implementation, the root context, the http filter context, the network filter context share same class Context. It make the code hard to read or to maintain.
It would be better to split the Context to different class like RootContext, HttpContext, NetworkContext, etc.

WDYT cc @mpwarres cc @kyessenov

Makes sense to me to have a base and subclasses if that is cleaner. I think it was done this way to reduce allocations so we shouldn't lose that.

I think it was done this way to reduce allocations so we shouldn't lose that.

I didn't get this. What you mean by reduce allocations? I think it's weird for a HttpContext to contains network filter's callback or state. It acutally bring additional unnecessary memory allocations.

I wasn't the author - but this shared context is effectively a global, and global help with reducing temporaries - that's just my best guess why it's done this way.
IMO the ABI should have really focused on HTTP calls. Network filter and bootstrap extensions should have been an Envoy specific ABI, but I guess it was slapped on later over HTTP context.

Yes, this makes sense to me, and in fact is what we have done elsewhere in some non-Envoy code built on top of proxy-wasm-cpp-host. The class hierarchy we ended up using there was:
// Base class for all Foo*Context classes, each of which implements
// ContextBase. In proxy_wasm, all three Context classes (VM, Root, Stream)
// are different use-cases of the same underlying class (ContextBase), which
// reduces readability and makes it messier to separate out logic between the
// different streams, so the purpose of this is to hold all common logic between
// the three contexts so that each one can painlessly subclass it. Each
// Context class will appear distinct but actually separately implement
// different methods in the underlying ContextBase. Each one will ultimately
// call the ContextBase constructor corresponding to its context type.
//
// This class overrides all on* callbacks from proxy_wasm::ContextBase and
// implements them to log an error containing details about which function was
// invoked and which kind of context invoked it. Subclasses must override
// callbacks they expect to implement.
class FooContextBase : public proxy_wasm::ContextBase { ... };

// Also known as the "Plugin" Context. Created in proxy_wasm::WasmBase::start.
// Provides the host implementation for root context Proxy-WASM APIs.
// A root context is created for every plugin and persists across the creation
// of multiple stream contexts.
//
// Implements the following proxy_wasm::ContextBase on* callbacks:
// - onStart
// - onConfigure
// - onDone
// - onDelete
class FooRootContext : public FooContextBase { ... };

// Provides the host implementation for stream context
// Proxy-Wasm APIs. Manages per-stream state, so a separate instance
// should be created for each stream (where a stream is a single
// request/response pair). This class is thread-hostile, and must be used from a
// single thread, as it is associated with a V8 VM that has thread-local state.
//
// Implements the following proxy_wasm::ContextBase on* callbacks:
// - onCreate
// - onDone
// - onDelete
// - onRequestHeaders
// - onResponseHeaders
class FooStreamContext : public FooContextBase { ... };

(We didn't differentiate HTTP vs. network streams since this other implementation didn't support network stream callbacks.)


chore: starter CI setup (build & lint actions, fix tests, removes prettier)

Now that this repository is getting serious (see #2), wanted to set up some typical CI stuff before really moving along.
Here's the cliff's notes:

adds GitHub Action for linting (ESLint + types), and one for build + test
fixes smoke/snapshot test (that was broken), removes other template test
uninstalls prettier and related deps (not needed for now)
migrates husky to v6 (a large breaking change); removes the push hook for type-checks (the CI will catch this now)
updates the README :)

Note: I'll slow-drip in stricter ESLint configs in another PR!

Not sure how the repo was created, but if it was through a template, maybe we'd want to do the same changes to the template repo?

Not sure how the repo was created, but if it was through a template, maybe we'd want to do the same changes to the template repo?

Actually, that's a great idea - totally slipped my mind! If I have some free time this weekend, I'll try to knock it out 😄


Allow users to delete backend environments on console without requesting cli

Is your feature request related to a problem? Please describe.
There're many backend environments I don't use now.

I'd like to delete these environments, however when I choose Delete from Actions drop menu, I see

Those entries are already dropped from my team-provider-info.json. So I can't delete them.
Describe the solution you'd like
Provide independent way to delete backend environments.

@hirochachacha are you saying that you already deleted these envs from the CLI but they are still showing up in the console?

@swaminator I'm not sure about that. I don't always use CLI for deleting resources. I often use CloudFormation console to delete them. In any case, I have no way to delete them now.

Hello,
I have the same issue athough I used the CLI to remove the environment. An error happened during the environment removal causing an infinite waiting loop in the CLI command.
I can remove manually the environment in team-provider-info.json and manually delete all allocated ressources in the cloud but the backend entry still remain in the Amplify console without any way to force remove it. How come that the CloudFormation stack doesn't clean up as well the Amplify console?
Thanks!

@dletozeun currently you can use the AWS CLI as a workaround to delete the backend environment after you have deleted the stack manually via the Console - https://docs.aws.amazon.com/cli/latest/reference/amplify/delete-backend-environment.html

I just ran into this issue, we had several phantom backend environments that couldn't be deleted from the console UI.  @swaminator 's workaround did the trick, but it took a while to figure out.  Given that the original issue (can't delete phantom backend envs from the console) still persists, can this be reopened?


How do I request enablement of a Privileged Admin Group assigned to a user?

I need a PowerShell way to find the/all Privileged Admin Group assigned to a specific user.
I have the Enablement Command, but fail to determine the Group RoledefinitionId
There is currently no page describing PowerShell for Admin Groups.
Thank you,
David

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: b427c59d-d87e-7b8d-f197-fe5ed529f74c
Version Independent ID: 57b69578-ed34-4906-65ca-4178f2befb5a
Content: Managing privileged Azure AD groups in Privileged Identity Management (PIM)
Content Source: articles/active-directory/privileged-identity-management/groups-features.md
Service: active-directory
Sub-service: pim
GitHub Login: @curtand
Microsoft Alias: curtand

@DEberhardt
Thanks for your feedback! We will investigate and update as appropriate.

@DEberhardt
Thanks for your feedback! I have assigned this issue to the author who will investigate and update as appropriate.

#reassign:@amsliu

@DEberhardt Thanks for your dedication to our documentation and our Product Group has now acknowledged your request in the backlog. Unfortunately, we have been unable to review your issue in a timely manner and we sincerely apologize for the delayed response. The requested updates on PowerShell for Admin Groups have not been made since the creation of this issue, and the timeline for resolution may vary based on resourcing, so we've created an internal work item to incorporate your suggestions. We are closing this issue for now, but feel free to comment here as necessary. #please-close


lines should persist for all ccgs

I added lines with gui.views[4].view.lines = [-3, 0, 3] (great!) but when two clusters are selected, only the first panel shows the lines. They should all show them.

Fixed in last PR (yet to be pushed).

Oh and also, in the future PR, you'll be able to put default CCG lines in your user_settings.

Great!
On Tue, Jun 16, 2015 at 11:27 PM, Cyrille Rossant<EMAIL_ADDRESS>wrote:

Oh and also, in the future PR, you'll be able to put default CCG lines in
your user_settings.
—
Reply to this email directly or view it on GitHub
https://github.com/kwikteam/phy/issues/347#issuecomment-112588786.


Backup single save file?

What's your question?
Hi! I have a PopCap game called Big Money Deluxe. It does not have a dedicated save game directory. Instead, there's a "savegame.dat" and a "hiscores.dat", next to the executeable (BigMoney.exe).
I have added Big Money Deluxe as a custom game. Is it possible to not save the whole game all the time but instead, only savegame.dat and hiscores.dat? I have only seen the option to specify a path, which of course means backing up the whole game.
Thanks!

Hi! Although the path selector dialog only lets you choose a folder, you can type in a file name (or glob pattern) afterwards :) There's some documentation about this here: https://github.com/mtkennerly/ludusavi/blob/master/docs/help/custom-games.md

Works perfect, thanks!


PowerShell Core crashs when app is closed

Environment
Windows build number:  10.0.18362.207
Windows Terminal version (if applicable): 0.3.2171.0
PowerShell Core: 6.2.2, 7.0preview2

Steps to reproduce
Open terminal with PowerShell Core, close app (click "x" top right)
Expected behavior
Application closes without any (obvious) issue....
Actual behavior
Visual Studio as JIT debugger is triggered and reports this unhandled exception of pwsh.exe:
System.InvalidOperationException: "Cannot read keys when either application does not have a console or when console input has been redirected. Try Console.Read."

This was fixed with #2198, which has not yet been released.


We should put the progress ring, zoom indicator, etc, in the tab switcher

In the tab running bx, it's got a progress ring (and the icon is hidden to show the ring). We should make those other indicators also show up in the tab switcher, so you can view them at a glance.
This might be extra hard, because right now the CommandPalette is just a set of {label, icon}-like objects. All those other icons and indicators are a LOT more content. That would make it something more like a set of {Content, Icon}-like objects.

@zadjii-msft - this can be much simpler once filtered list view is extracted. Until then we can select item template dynamically.
If this is not urgent, I can absolutely work on this

@zadjii-msft - this can be much simpler once filtered list view is extracted. Until then we can select item template dynamically.
If this is not urgent, I can absolutely work on this

😍


Register allocator checker

This PR adds a register allocator checker: an analysis that does a dataflow analysis of the register allocator's output, including spills, reloads, moves, and mapped real registers, to ensure that the resulting program has the same semantics as the virtual-register-based input.
The checker is wired into the backtracking allocator (via the BacktrackingChecked alternative algorithm choice, because we don't have a separate way to specify flags). Integration into linear-scan should be pretty straightforward too; I just haven't done it yet because the top-level is a bit different (doesn't use edit_inst_stream).

This looks awesome (and very quickly put together!).  I look forward to trying it on our current non-working inputs.  I have no particular comments on the low level details/coding, and would be in favour of landing it.  I do have three general points though:

I see places where instructions created by func.gen_{spill,reload,move} are now replaced by an InstToInsert, which looks like some abstractified version of s/r/m insns that presumably is later transformed into a target-specific version somehow.  (Right?)  The concern is whether this might slow down the allocator in as much as it constitutes extra copying of information.  But I think I can live with it .. if it is an inefficiency, there are much bigger ones elsewhere in the allocator as a whole, that first need to be dealt with.  Plus, in most cases, there won't be much, if any, spilling.

The current BT algorith, and I suspect LSRA too, only uses each spill slot for one virtual live range.  It never recycles them.  For a production implementation that's obviously unacceptable -- I plan to add a simple spill slot occupancy tracker/allocator at some point.  Q: does your analysis actually depend on the current each-spilled-VLR-has-its-own-spill-slot behaviour?  It would be a shame if so, since that'll mean it will break when that is fixed.

I see you added new RegAllocAlgorithm allocator kinds.  I have been contemplating a more general scheme, at least for this early stage development, where we provide a general and easy to use way to enable cross-checking functionality.  I say this because the current sources already provide a cross-check mechanism for BT's core CommitmentMap machinery, and I am about to land a second mechanism for cross-checking of dominator-set computation.  So your checker could be a third case for this.
The scheme I have been contemplating simply involves an env var that is set at run time:

  REGALLOC_CHECK=   some subset of {btcm,dom,overall,timing}
  where
        btcm     is BT's CommitmentMap
        dom      is dominator-set computation
        overall  is this analysis
        timing   is kind of anomalous, but means: print summaries of RA pass
                 timings without all the debug spew.  I think this will be
                 important.

(this should be indented; oh well)
Then we could use cross checking in whatever embedding (test framework, wasmtime, SpiderMonkey) uniformly and without the hassle of having to try and wire up control settings separately in each embedding.  For this reason I'm not entirely enthusiastic about the change to RegAllocAlgorithm.

I haven't looked at the code itself, but great news this is working!
Regarding selection of features: env variables have their limits, and it would be nice to allow users of this crate to enable the supplementary checks, if needed. So we could have a real way to give configuration options, and in the minira binary, these could be enabled through env variables and/or binary flags. If that makes sense, I'm happy to implement this.

Just rebased on top of latest linear-scan changes. Thanks for feedback! A few answers:

I see places where instructions created by func.gen_{spill,reload,move} are now replaced by an InstToInsert, which looks like some abstractified version of s/r/m insns that presumably is later transformed into a target-specific version somehow. (Right?)

Indeed, and hopefully this isn't too much overhead -- in fact the list of ops to insert will likely be more compact in this form than as the full arm64::Inst struct. In any case the alternative was to add more API surface to Inst to query whether something is a spill/reload; the current approach feels cleaner.

Q: does your analysis actually depend on the current each-spilled-VLR-has-its-own-spill-slot behaviour? It would be a shame if so, since that'll mean it will break when that is fixed.

No, it should be fully flexible w.r.t. spill-slot sharing. The content of each spill-slot is tracked at each program-point (i.e., it's a flow-sensitive analysis).
Regarding configuration options: I generally agree with Ben that it'd be nice to have a central configuration infrastructure, with different ways of setting flags. I wouldn't necessarily want to insert surprising dependencies on environment variables into a program that uses this library, at least not without explicit opt-in to that.
If you both don't mind, though, perhaps we could land this as-is first, and then work out the config stuff as a separate issue? Mostly I'm motivated by wanting to get this going in the Cranelift branch without local tweaks to build with this PR :-) Alternatively, if we don't want to clutter the algorithm-choice enum, we could add an environment-variable check for now.

Fine by me to land as-is.

Sounds good! I'll create an issue to track the configuration-infra / flags / env vars question.


Upgrade Storm Dependency to 1.1.1

Upgrade Storm Dependency to 1.1.1.
This dependency is scoped at provided, and Storm 1.1.x <--> Storm 1.0.x appear to be compatible, so we should be able to continue supporting both versions.

@Crim do we want to bump the library to 0.9 when we do this? We might want to use milestones and aggregate some 0.9 changes together.

Sounds like a plan


logging: Fix silent logs dropping when LOG_MODE_OVERFLOW disabled

There was no notification about dropped logs When logger operated
in the mode where message is dropped when logger has no space to
store the message. Notification was printed only if logger operated
in the mode which overwrites oldest log.
Signed-off-by: Krzysztof Chruscinski<EMAIL_ADDRESS>

Codecov Report

Merging #13785 into master will decrease coverage by <.01%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master   #13785      +/-   ##
==========================================
- Coverage   52.26%   52.25%   -0.01%
==========================================
  Files         307      307
  Lines       45404    45408       +4
  Branches    10504    10504
==========================================
  Hits        23729    23729
- Misses      16886    16890       +4
  Partials     4789     4789

Impacted Files
Coverage Δ

include/logging/log_core.h
100% <ø> (ø)
:arrow_up:

subsys/logging/log_core.c
37.34% <0%> (-0.31%)
:arrow_down:

subsys/logging/log_msg.c
71.75% <0%> (-0.83%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 51a00cf...6217096. Read the comment docs.


Best practices for monitoring?

Hi,
This article gives you a general introduction to monitoring.
I am looking for more specific best practices. Set up alarms that fires when these metric reaches these thresholds. Is there such best practices somewhere?
Regards peter

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 7aa5ecfe-e37e-999a-df1d-76ed909e6c09
Version Independent ID: 4cfce2e0-ad27-dde2-ac1f-fef92dcb9d01
Content: Monitoring and performance tuning - Azure SQL Database
Content Source: articles/sql-database/sql-database-monitor-tune-overview.md
Service: sql-database
Sub-service: performance
GitHub Login: @jovanpop-msft
Microsoft Alias: jovanpop

@theheatDK Thanks for the feedback! I have assigned the issue to the content author to evaluate and update as appropriate.

@danimir could you help with this?

@AdamS-MSFT - @danimir is new PM contact for monitoring and alerting so could you please reassign it?

@AdamS-MSFT Before reassigning and escalating to PG please check for existence of already available documentation.

@theheatDK For alerting, see this: https://docs.microsoft.com/bs-latn-ba/azure/sql-database/sql-database-insights-alerts-portal
In terms of best practices and monitoring no recommendations can be given at this time because it all depends. This is because every database, environment, workload and application are different, therefore there exists no one fits all solution applicable equally to all customers.
Thank you on your feedback.

@theheatDK The document that @danimir has provided does explain the metrics available for which alerts could be configured for and possible actions to take, and for configuring webhooks in alerts.
We will now proceed to close this thread. If there are further questions regarding this matter, please comment and we will gladly continue the discussion.


Fix the naming difference for mapAutoZoom in n2.mapUtilities.js

Before this commit the map-utility mapAutoZoom is named
mapAlwaysIncludesExtentsZoom in 2.2.9 and mapAutoZoom in 2.3.0.
The name should be changed to the same "mapAutoZoom"
Get fixed in commit ae9089379996a734231518649f98232ed4fd943e

Get fixed in commit ae90893


It's a bit strange to have an uppercase letter (await.allSettled) in an operator

It's a metaproperty - await is the operator. You can already have super.SCREAMING_METHOD_NAME(), for example.

Meta property is a property. But this is a unary operator 🤔

new is a unary operator and new.target is a metaproperty that we already have for years

I guess consistency with method names on Promise have the high priority 🤓

Given that consistency is the priority here, and that "operators are all lowercase" is not an intentional axiom, just a convention, I think this can be closed.


Option to download one after another

and not all at the same time.
useful for a slow internet connection.

You refer to the number of parallel downloads?
If so, then currently there is a way to change the number of max parallel downloads by editing the "workers_number" option in your settings.json file.
The settings.json file is located at

Windows:     %APPDATA%\Roaming\youtube-dlg
Linux:    ~/.config/youtube-dlg

You can also read this issue: https://github.com/MrS0m30n3/youtube-dl-gui/issues/20
This options will be available at the new GUI so you won't have to edit your settings file everytime.

Ah, thank you


Foreign key from ActorName to Actor

actor_id is a foreign key to the Actor table.

Whoops! Committed with the wrong bug id


Color progress bar based on delay

I found it hard to understand whether the train was on time or not, just by taking a quick glance at the program output. The red color would make me feel uneasy, as I generally associate it with warning signals; I constantly thought we're late, heh.
To mitigate that, I thought about making the progress bar green when the train is on time and red when it is not.
What do you think about that change?

P.S. testing this live from a train. 😆 There are no tests for the view right now, but I could try to add one if needed.

Hi! Thanks for creating this PR - I'm suprised somebody is actually using this :^)
I chose the red color mostly because I wanted to match the official UI of the iceportal.
But I get that it can be confusing, so maybe it makes sense to add come kind of config option to this? maybe something like a color scheme option in a config file. What do you think of this? If you like it, we can either create a new issue for it or discuss on how it's best implemented.

Makes total sense!
I'm not sure if a single config option is enough to warrant an entire config file. If more settings will be added in the future, it might be reasonable, though.
Alternatively, it could just be a CLI option for now, e.g. --theme portal and --theme delay. It's easy to support config files in the future. Just an idea. 😉

That sounds reasonable to me - do you want to update the PR with this? I guess https://github.com/spf13/pflag would be a good choice :)

Sure, I've updated this PR. Let me know if I should make any adjustments.


Public rooms search: Spinner stays forever

If you make a search in the public room directory and if the result contains a small number of rooms, the spinner stays forever.
This may be related to the testPublicRooms and testSearchPublicRooms tests that currently fails in the SDK(#1797).

I cannot repro the issue a second time :/


Allow polyfills to be organised into subfolders in the source tree

This PR allows polyfills to be organised into subfolders, but also permits the current structure.  This is intended to make it easier to maintain the project after adding large polyfills such as Intl, as discussed in #108.

@triblondon do we need any update on this PR before I try to get the intl polyfill integrated?

Looks like it needs a merge from master. I'll put it on my list, and try to do it this week.

@caridy Branch updated.  You should also have invite to be a collaborator on the repo.  Do you see it?

awesome, I will give it a try today.

Thanks.  Build tests are failing due to a very simple issue I merged without testing properly.  I'll sort that out and merge an update, but you're good to go in the meantime.

Hooray, build is now passing on this branch.

@triblondon Should this be closed?

Yes, looks like it's fully merged into master

thanks a lot @triblondon.
qq, what will be the process to update Intl from now on considering that we only need to update package.json and run the update library task?

Do exactly that.  And then go through the normal RC -> release process, https://github.com/Financial-Times/polyfill-service/issues/328#issue-59561080


Bump up ansible version to fix os-projects. Fixes #50

Which of the issues in #50 does this fix?

Well, when I tested it, neither of the issues reappear.

Okay this seems clearer.


Unable to use GPUs 4,5,6,7 for training

First of all - thanks for all the great work!
My setup is on an H100. I am trying to use GPU 4,5,6,7 but I get the following error. I am able to run successfully with 0,1,2,3,4,5,6,7 GPUs. However, I only want to use half. Not sure how to resolve this.
During pretrain.sh or finetune.sh, I get the following error.
torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp
:1331, internal error - please report this issue to the NCCL developers, NCCL version 2.18.1
ncclInternalError: Internal check failed.
Last error:
Attribute arch of node cpu not found

Hi, please refer to https://github.com/NVIDIA/nccl/issues/833 to see if it helps

I'm afraid that's not the same error. Their issue has something to do with network while my issue is that arch of node cpu not found. I suspect this is because the llava model is initially placed on CPU and then eventually moved to cuda.

BTW able to run this on A100 with GPUs 4,5,6,7 only. So maybe it's an H100 specific issue.

running into same issue on  H100, how did you get around this @shashwat14 ??