Problem running train.py

Hello Professor ！
Thank you for your excellent work! When I was running train.py, the following error occurred
Traceback (most recent call last):
  File "/home/zhaojing/下载/pycharm-community-2022.2.4/plugins/python-ce/helpers/pydev/pydevd.py", line 1496, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/home/zhaojing/下载/pycharm-community-2022.2.4/plugins/python-ce/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/zhaojing/code/superpoint_transformer-master/src/train.py", line 3, in <module>
    root = str(pyrootutils.setup_root(
  File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pyrootutils/pyrootutils.py", line 151, in setup_root
    path = find_root(search_from, indicator)
  File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pyrootutils/pyrootutils.py", line 73, in find_root
    raise FileNotFoundError(f"Project root directory not found. Indicators: {indicator}")
FileNotFoundError: Project root directory not found. Indicators: ['.git', 'pyproject.toml']

Hi, have you followed the instructions here for setting up your own data path ?

The reason for this error is that,  unless specified otherwise, the program will use the data/ in the project root directory.
But, in order to find the absolute path to your project, the pyrootutils library is used. As stated in your error message :
FileNotFoundError: Project root directory not found. Indicators: ['.git', 'pyproject.toml']

the pyrootutils library needs to find of some specific files to identify your root directory:

.git
pyproject.toml

The fact that this failed is probably because you do not have a .git in your repository. I am guessing you did not geet the code using git clone ... ?
I just updated the code for pyrootutils to also identify your project root directory based on the README.md. Can you please let me know if this solves your issue ?

Thank you for your reply!
The previous error was resolved, but I still don't know which directory the dataset should be placed in in your project. I noticed an error in import src. data. Do I want to create a data file in the src directory and then place the dataset in it？
Traceback (most recent call last):
File "src/train.py", line 49, in
from src import utils
File "/home/zhaojing/code/superpoint_transformer-master/src/init.py", line 2, in
import src.data
ModuleNotFoundError: No module named 'src.data'

There was a hotfix an hour ago, pull a fresh version of the repo and this bug should be fixed.

Thank you!
Previous issue resolved，but there is still a lack of necessary module
Traceback (most recent call last):
File "src/train.py", line 49, in
from src import utils
File "/home/zhaojing/code/superpoint_transformer-master/src/init.py", line 2, in
import src.data
File "/home/zhaojing/code/superpoint_transformer-master/src/data/init.py", line 1, in
from .csr import *
File "/home/zhaojing/code/superpoint_transformer-master/src/data/csr.py", line 5, in
from src.utils import tensor_idx, is_sorted, indices_to_pointers,
File "/home/zhaojing/code/superpoint_transformer-master/src/utils/init.py", line 11, in
from .neighbors import *
File "/home/zhaojing/code/superpoint_transformer-master/src/utils/neighbors.py", line 3, in
from src.dependencies.FRNN import frnn
ModuleNotFoundError: No module named 'src.dependencies'

Hi @jing-zhao9 thanks a lot for being our first beta-tester 🙏 !
I just pushed a commit which should fix the src/dependencies error, would you mind pulling the latest version and checking again ?

You did not add any Python files in src/dependencies
I still reported the an error
Traceback (most recent call last):
File "/home/zhaojing/code/superpoint_transformer-master/src/train.py", line 49, in
from src import utils
File "/home/zhaojing/code/superpoint_transformer-master/src/init.py", line 2, in
import src.data
File "/home/zhaojing/code/superpoint_transformer-master/src/data/init.py", line 1, in
from .csr import *
File "/home/zhaojing/code/superpoint_transformer-master/src/data/csr.py", line 5, in
from src.utils import tensor_idx, is_sorted, indices_to_pointers,
File "/home/zhaojing/code/superpoint_transformer-master/src/utils/init.py", line 11, in
from .neighbors import *
File "/home/zhaojing/code/superpoint_transformer-master/src/utils/neighbors.py", line 3, in
from src.dependencies.FRNN import frnn
ModuleNotFoundError: No module named 'src.dependencies.FRNN'

This is the expected behaviour, the src/dependencies is initially empty before you run install.sh. The FRNN library is missing because it has been installed when the src/dependencies directory structure was missing.
Please delete your spt conda env and re-run install.sh.

When I was training, I encountered the following issues
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/multiprocessing/pool.py", line 51, in starmapstar
return list(itertools.starmap(args[0], args[1]))
File "/home/zhaojing/code/superpoint_transformer-master/src/utils/multiprocessing.py", line 41, in apply_args_and_kwargs
return fn(*args, **kwargs)
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/s3dis.py", line 199, in read_s3dis_room
alignments = pd.read_csv(
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pandas/io/parsers/readers.py", line 912, in read_csv
return _read(filepath_or_buffer, kwds)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pandas/io/parsers/readers.py", line 577, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pandas/io/parsers/readers.py", line 1407, in init
self._engine = self._make_engine(f, self.engine)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pandas/io/parsers/readers.py", line 1661, in _make_engine
self.handles = get_handle(
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pandas/io/common.py", line 859, in get_handle
handle = open(
FileNotFoundError: [Errno 2] No such file or directory: '/home/zhaojing/code/superpoint_transformer-master/data/s3dis/raw/Area_1/Area_1_alignmentAngle.txt'
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/zhaojing/code/superpoint_transformer-master/src/utils/utils.py", line 45, in wrap
metric_dict, object_dict = task_func(cfg=cfg)
File "src/train.py", line 114, in train
trainer.fit(model=model, datamodule=datamodule, ckpt_path=cfg.get("ckpt_path"))
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 531, in fit
call._call_and_handle_interrupt(
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 42, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 570, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 927, in _run
self._data_connector.prepare_data()
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py", line 94, in prepare_data
call._call_lightning_datamodule_hook(trainer, "prepare_data")
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 160, in _call_lightning_datamodule_hook
return fn(*args, **kwargs)
File "/home/zhaojing/code/superpoint_transformer-master/src/datamodules/base.py", line 144, in prepare_data
self.dataset_class(
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/s3dis.py", line 255, in init
super().init(*args, val_mixed_in_train=True, **kwargs)
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/base.py", line 193, in init
super().init(root, transform, pre_transform, pre_filter)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/data/in_memory_dataset.py", line 57, in init
super().init(root, transform, pre_transform, pre_filter, log)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/data/dataset.py", line 97, in init
self._process()
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/base.py", line 493, in _process
self.process()
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/base.py", line 528, in process
self._process_single_cloud(p)
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/base.py", line 544, in _process_single_cloud
data = self.read_single_raw_cloud(raw_path)
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/s3dis.py", line 330, in read_single_raw_cloud
return read_s3dis_area(
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/s3dis.py", line 75, in read_s3dis_area
batch = Batch.from_data_list(starmap_with_kwargs(
File "/home/zhaojing/code/superpoint_transformer-master/src/utils/multiprocessing.py", line 36, in starmap_with_kwargs
out = pool.starmap(apply_args_and_kwargs, args_for_starmap)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/multiprocessing/pool.py", line 372, in starmap
return self._map_async(func, iterable, starmapstar, chunksize).get()
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/multiprocessing/pool.py", line 771, in get
raise self._value
FileNotFoundError: [Errno 2] No such file or directory: '/home/zhaojing/code/superpoint_transformer-master/data/s3dis/raw/Area_1/Area_1_alignmentAngle.txt'
[2023-06-17 10:25:52,506][src.utils.utils][INFO] - Closing loggers...
[2023-06-17 10:25:52,506][src.utils.utils][INFO] - Closing wandb!
wandb: Waiting for W&B process to finish... (success).
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/zhaojing/code/superpoint_transformer-master/logs/train/runs/2023-06-17_10-25-24/wandb/offline-run-20230617_102527-lgqrcs8f
wandb: Find logs at: ./logs/train/runs/2023-06-17_10-25-24/wandb/offline-run-20230617_102527-lgqrcs8f/logs
Error executing job with overrides: ['experiment=s3dis', 'datamodule.fold=5']
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/multiprocessing/pool.py", line 51, in starmapstar
return list(itertools.starmap(args[0], args[1]))
File "/home/zhaojing/code/superpoint_transformer-master/src/utils/multiprocessing.py", line 41, in apply_args_and_kwargs
return fn(*args, **kwargs)
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/s3dis.py", line 199, in read_s3dis_room
alignments = pd.read_csv(
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pandas/io/parsers/readers.py", line 912, in read_csv
return _read(filepath_or_buffer, kwds)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pandas/io/parsers/readers.py", line 577, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pandas/io/parsers/readers.py", line 1407, in init
self._engine = self._make_engine(f, self.engine)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pandas/io/parsers/readers.py", line 1661, in _make_engine
self.handles = get_handle(
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pandas/io/common.py", line 859, in get_handle
handle = open(
FileNotFoundError: [Errno 2] No such file or directory: '/home/zhaojing/code/superpoint_transformer-master/data/s3dis/raw/Area_1/Area_1_alignmentAngle.txt'
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "src/train.py", line 139, in main
metric_dict, _ = train(cfg)
File "/home/zhaojing/code/superpoint_transformer-master/src/utils/utils.py", line 48, in wrap
raise ex
File "/home/zhaojing/code/superpoint_transformer-master/src/utils/utils.py", line 45, in wrap
metric_dict, object_dict = task_func(cfg=cfg)
File "src/train.py", line 114, in train
trainer.fit(model=model, datamodule=datamodule, ckpt_path=cfg.get("ckpt_path"))
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 531, in fit
call._call_and_handle_interrupt(
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 42, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 570, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 927, in _run
self._data_connector.prepare_data()
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py", line 94, in prepare_data
call._call_lightning_datamodule_hook(trainer, "prepare_data")
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 160, in _call_lightning_datamodule_hook
return fn(*args, **kwargs)
File "/home/zhaojing/code/superpoint_transformer-master/src/datamodules/base.py", line 144, in prepare_data
self.dataset_class(
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/s3dis.py", line 255, in init
super().init(*args, val_mixed_in_train=True, **kwargs)
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/base.py", line 193, in init
super().init(root, transform, pre_transform, pre_filter)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/data/in_memory_dataset.py", line 57, in init
super().init(root, transform, pre_transform, pre_filter, log)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/data/dataset.py", line 97, in init
self._process()
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/base.py", line 493, in _process
self.process()
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/base.py", line 528, in process
self._process_single_cloud(p)
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/base.py", line 544, in _process_single_cloud
data = self.read_single_raw_cloud(raw_path)
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/s3dis.py", line 330, in read_single_raw_cloud
return read_s3dis_area(
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/s3dis.py", line 75, in read_s3dis_area
batch = Batch.from_data_list(starmap_with_kwargs(
File "/home/zhaojing/code/superpoint_transformer-master/src/utils/multiprocessing.py", line 36, in starmap_with_kwargs
out = pool.starmap(apply_args_and_kwargs, args_for_starmap)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/multiprocessing/pool.py", line 372, in starmap
return self._map_async(func, iterable, starmapstar, chunksize).get()
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/multiprocessing/pool.py", line 771, in get
raise self._value
FileNotFoundError: [Errno 2] No such file or directory: '/home/zhaojing/code/superpoint_transformer-master/data/s3dis/raw/Area_1/Area_1_alignmentAngle.txt'
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

I take it that the previous issue was solved with the last commit ?
The problem you are facing now is that you probably downloaded the S3DIS Aligned version. The S3DIS dataset comes in two flavors: aligned Stanford3dDataset_v1.2_Aligned_Version.zipand non-aligned Stanford3dDataset_v1.2.zip. Our code is expecting the dataset to stem from the non-aligned Stanford3dDataset_v1.2.zip file, which contains these Area_i/Area_i_alignmentAngle.txt files. The reason for this is that SPT operates on the full building floors at once and not on individual rooms like most methods. To reconstruct the whole floors, these room orientation files are needed.
It is my fault for not making this clear enough, I will update the code and documentation to make this more explicit. Thanks for pointing this out !

The previous issue has been resolved, but I encountered a new issue after replacing the dataset
[2023-06-17 20:26:06,316][src.utils.utils][ERROR] -
Traceback (most recent call last):
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/urllib3/connectionpool.py", line 714, in urlopen
httplib_response = self._make_request(
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/urllib3/connectionpool.py", line 403, in _make_request
self._validate_conn(conn)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1053, in validate_conn
conn.connect()
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/urllib3/connection.py", line 419, in connect
self.sock = ssl_wrap_socket(
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/urllib3/util/ssl.py", line 449, in ssl_wrap_socket
ssl_sock = ssl_wrap_socket_impl(
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/urllib3/util/ssl.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/ssl.py", line 1040, in _create
self.do_handshake()
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1131)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/urllib3/connectionpool.py", line 798, in urlopen
retries = retries.increment(
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='drive.google.com', port=443): Max retries exceeded with url: /uc?id=0BweDykwS9vIobkVPN0wzRzFwTDg&export=download (Caused by SSLError(SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1131)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/zhaojing/code/superpoint_transformer-master/src/utils/utils.py", line 45, in wrap
metric_dict, object_dict = task_func(cfg=cfg)
File "src/train.py", line 114, in train
trainer.fit(model=model, datamodule=datamodule, ckpt_path=cfg.get("ckpt_path"))
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 531, in fit
call._call_and_handle_interrupt(
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 42, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 570, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 927, in _run
self._data_connector.prepare_data()
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py", line 94, in prepare_data
call._call_lightning_datamodule_hook(trainer, "prepare_data")
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 160, in _call_lightning_datamodule_hook
return fn(*args, **kwargs)
File "/home/zhaojing/code/superpoint_transformer-master/src/datamodules/base.py", line 144, in prepare_data
self.dataset_class(
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/s3dis.py", line 255, in init
super().init(*args, val_mixed_in_train=True, **kwargs)
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/base.py", line 193, in init
super().init(root, transform, pre_transform, pre_filter)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/data/in_memory_dataset.py", line 57, in init
super().init(root, transform, pre_transform, pre_filter, log)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/data/dataset.py", line 94, in init
self._download()
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/data/dataset.py", line 199, in _download
self.download()
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/base.py", line 447, in download
self.download_dataset()
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/s3dis.py", line 293, in download_dataset
self.download_zip()
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/s3dis.py", line 323, in download_zip
gdown.download(
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/gdown/download.py", line 161, in download
res = sess.get(url, stream=True, verify=verify)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/requests/sessions.py", line 602, in get
return self.request("GET", url, **kwargs)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/requests/adapters.py", line 517, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='drive.google.com', port=443): Max retries exceeded with url: /uc?id=0BweDykwS9vIobkVPN0wzRzFwTDg&export=download (Caused by SSLError(SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1131)')))
[2023-06-17 20:26:06,318][src.utils.utils][INFO] - Closing loggers...
[2023-06-17 20:26:06,319][src.utils.utils][INFO] - Closing wandb!
wandb: Waiting for W&B process to finish... (success).
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/zhaojing/code/superpoint_transformer-master/logs/train/runs/2023-06-17_20-25-51/wandb/offline-run-20230617_202558-4aymg4y0
wandb: Find logs at: ./logs/train/runs/2023-06-17_20-25-51/wandb/offline-run-20230617_202558-4aymg4y0/logs
Error executing job with overrides: ['experiment=s3dis', 'datamodule.fold=5']
Traceback (most recent call last):
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/urllib3/connectionpool.py", line 714, in urlopen
httplib_response = self._make_request(
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/urllib3/connectionpool.py", line 403, in _make_request
self._validate_conn(conn)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1053, in validate_conn
conn.connect()
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/urllib3/connection.py", line 419, in connect
self.sock = ssl_wrap_socket(
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/urllib3/util/ssl.py", line 449, in ssl_wrap_socket
ssl_sock = ssl_wrap_socket_impl(
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/urllib3/util/ssl.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/ssl.py", line 1040, in _create
self.do_handshake()
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1131)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/urllib3/connectionpool.py", line 798, in urlopen
retries = retries.increment(
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='drive.google.com', port=443): Max retries exceeded with url: /uc?id=0BweDykwS9vIobkVPN0wzRzFwTDg&export=download (Caused by SSLError(SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1131)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "src/train.py", line 139, in main
metric_dict, _ = train(cfg)
File "/home/zhaojing/code/superpoint_transformer-master/src/utils/utils.py", line 48, in wrap
raise ex
File "/home/zhaojing/code/superpoint_transformer-master/src/utils/utils.py", line 45, in wrap
metric_dict, object_dict = task_func(cfg=cfg)
File "src/train.py", line 114, in train
trainer.fit(model=model, datamodule=datamodule, ckpt_path=cfg.get("ckpt_path"))
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 531, in fit
call._call_and_handle_interrupt(
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 42, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 570, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 927, in _run
self._data_connector.prepare_data()
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py", line 94, in prepare_data
call._call_lightning_datamodule_hook(trainer, "prepare_data")
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 160, in _call_lightning_datamodule_hook
return fn(*args, **kwargs)
File "/home/zhaojing/code/superpoint_transformer-master/src/datamodules/base.py", line 144, in prepare_data
self.dataset_class(
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/s3dis.py", line 255, in init
super().init(*args, val_mixed_in_train=True, **kwargs)
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/base.py", line 193, in init
super().init(root, transform, pre_transform, pre_filter)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/data/in_memory_dataset.py", line 57, in init
super().init(root, transform, pre_transform, pre_filter, log)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/data/dataset.py", line 94, in init
self._download()
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/data/dataset.py", line 199, in _download
self.download()
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/base.py", line 447, in download
self.download_dataset()
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/s3dis.py", line 293, in download_dataset
self.download_zip()
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/s3dis.py", line 323, in download_zip
gdown.download(
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/gdown/download.py", line 161, in download
res = sess.get(url, stream=True, verify=verify)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/requests/sessions.py", line 602, in get
return self.request("GET", url, **kwargs)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/requests/adapters.py", line 517, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='drive.google.com', port=443): Max retries exceeded with url: /uc?id=0BweDykwS9vIobkVPN0wzRzFwTDg&export=download (Caused by SSLError(SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1131)')))
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

I used your latest commit and delete my spt conda env and re-run install.sh.But the following issues occurred
cc1plus: warning: command-line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
cc1plus: warning: command-line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
cc1plus: warning: command-line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
Traceback (most recent call last):
File "scripts/setup_dependencies.py", line 100, in
os.chdir(osp.join(DEPENDENCIES_DIR, 'parallel_cut_pursuit/python'))
FileNotFoundError: [Errno 2] No such file or directory: '/home/zhaojing/code/superpoint_transformer-master/src/dependencies/parallel_cut_pursuit/python'
🚀 Successfully installed SPT
(spt) zhaojing@zhaojing-System-Product-Name:~/code/superpoint_transformer-master$ # Train SPT on S3DIS Fold 5
(spt) zhaojing@zhaojing-System-Product-Name:~/code/superpoint_transformer-master$ python src/train.py experiment=s3dis datamodule.fold=5
Traceback (most recent call last):
File "src/train.py", line 49, in
from src import utils
File "/home/zhaojing/code/superpoint_transformer-master/src/init.py", line 3, in
import src.datasets
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/init.py", line 2, in
from .base import *
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/base.py", line 17, in
from src.transforms import NAGSelectByKey, NAGRemoveKeys, SampleXYTiling,
File "/home/zhaojing/code/superpoint_transformer-master/src/transforms/init.py", line 10, in
from .partition import *
File "/home/zhaojing/code/superpoint_transformer-master/src/transforms/partition.py", line 17, in
from cp_kmpp_d0_dist import cp_kmpp_d0_dist
ModuleNotFoundError: No module named 'cp_kmpp_d0_dist'

Hi, the previous issue has been resolved. May I ask if the uncompressed dataset Standford3DDataset_v1.2.zip placed in the s3dis folder？

Thank you very much for your guidance. I think the project may only have the last step left to run, and an error message indicates that my CUDA out of memory. May I ask how much CUDA memory is required for your project to run？
Extracting /home/zhaojing/code/superpoint_transformer-master/data/s3dis/Stanford3dDataset_v1.2.zip
Processing...
0%|                                                                                | 0/10 [03:41<?, ?it/s]
[2023-06-18 16:46:30,636][src.utils.utils][ERROR] -
Traceback (most recent call last):
File "/home/zhaojing/code/superpoint_transformer-master/src/utils/utils.py", line 45, in wrap
metric_dict, object_dict = task_func(cfg=cfg)
File "src/train.py", line 114, in train
trainer.fit(model=model, datamodule=datamodule, ckpt_path=cfg.get("ckpt_path"))
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 531, in fit
call._call_and_handle_interrupt(
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 42, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 570, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 927, in _run
self._data_connector.prepare_data()
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py", line 94, in prepare_data
call._call_lightning_datamodule_hook(trainer, "prepare_data")
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 160, in _call_lightning_datamodule_hook
return fn(*args, **kwargs)
File "/home/zhaojing/code/superpoint_transformer-master/src/datamodules/base.py", line 144, in prepare_data
self.dataset_class(
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/s3dis.py", line 258, in init
super().init(*args, val_mixed_in_train=True, **kwargs)
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/base.py", line 193, in init
super().init(root, transform, pre_transform, pre_filter)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/data/in_memory_dataset.py", line 57, in init
super().init(root, transform, pre_transform, pre_filter, log)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/data/dataset.py", line 97, in init
self._process()
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/base.py", line 493, in _process
self.process()
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/base.py", line 528, in process
self._process_single_cloud(p)
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/base.py", line 559, in _process_single_cloud
nag = self.pre_transform(data)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/transforms/compose.py", line 24, in call
data = transform(data)
File "/home/zhaojing/code/superpoint_transformer-master/src/transforms/transforms.py", line 23, in call
return self._process(x)
File "/home/zhaojing/code/superpoint_transformer-master/src/transforms/sampling.py", line 150, in _process
data = _group_data(
File "/home/zhaojing/code/superpoint_transformer-master/src/transforms/sampling.py", line 255, in _group_data
hist = atomic_to_histogram(item, cluster, n_bins=n_bins)
File "/home/zhaojing/code/superpoint_transformer-master/src/utils/metrics.py", line 73, in atomic_to_histogram
hist = scatter_add(item, idx, dim=0)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/torch_scatter/scatter.py", line 29, in scatter_add
return scatter_sum(src, index, dim, out, dim_size)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/torch_scatter/scatter.py", line 19, in scatter_sum
size[dim] = int(index.max()) + 1
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 4.27 GiB (GPU 0; 11.77 GiB total capacity; 8.82 GiB already allocated; 1.98 GiB free; 9.27 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2023-06-18 16:46:30,648][src.utils.utils][INFO] - Closing loggers...
[2023-06-18 16:46:30,649][src.utils.utils][INFO] - Closing wandb!
wandb: Waiting for W&B process to finish... (success).
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/zhaojing/code/superpoint_transformer-master/logs/train/runs/2023-06-18_16-36-37/wandb/offline-run-20230618_163641-zlgrh3u7
wandb: Find logs at: ./logs/train/runs/2023-06-18_16-36-37/wandb/offline-run-20230618_163641-zlgrh3u7/logs
Error executing job with overrides: ['experiment=s3dis', 'datamodule.fold=5']
Traceback (most recent call last):
File "src/train.py", line 139, in main
metric_dict, _ = train(cfg)
File "/home/zhaojing/code/superpoint_transformer-master/src/utils/utils.py", line 48, in wrap
raise ex
File "/home/zhaojing/code/superpoint_transformer-master/src/utils/utils.py", line 45, in wrap
metric_dict, object_dict = task_func(cfg=cfg)
File "src/train.py", line 114, in train
trainer.fit(model=model, datamodule=datamodule, ckpt_path=cfg.get("ckpt_path"))
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 531, in fit
call._call_and_handle_interrupt(
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 42, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 570, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 927, in _run
self._data_connector.prepare_data()
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py", line 94, in prepare_data
call._call_lightning_datamodule_hook(trainer, "prepare_data")
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 160, in _call_lightning_datamodule_hook
return fn(*args, **kwargs)
File "/home/zhaojing/code/superpoint_transformer-master/src/datamodules/base.py", line 144, in prepare_data
self.dataset_class(
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/s3dis.py", line 258, in init
super().init(*args, val_mixed_in_train=True, **kwargs)
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/base.py", line 193, in init
super().init(root, transform, pre_transform, pre_filter)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/data/in_memory_dataset.py", line 57, in init
super().init(root, transform, pre_transform, pre_filter, log)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/data/dataset.py", line 97, in init
self._process()
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/base.py", line 493, in _process
self.process()
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/base.py", line 528, in process
self._process_single_cloud(p)
File "/home/zhaojing/code/superpoint_transformer-master/src/datasets/base.py", line 559, in _process_single_cloud
nag = self.pre_transform(data)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/transforms/compose.py", line 24, in call
data = transform(data)
File "/home/zhaojing/code/superpoint_transformer-master/src/transforms/transforms.py", line 23, in call
return self._process(x)
File "/home/zhaojing/code/superpoint_transformer-master/src/transforms/sampling.py", line 150, in _process
data = _group_data(
File "/home/zhaojing/code/superpoint_transformer-master/src/transforms/sampling.py", line 255, in _group_data
hist = atomic_to_histogram(item, cluster, n_bins=n_bins)
File "/home/zhaojing/code/superpoint_transformer-master/src/utils/metrics.py", line 73, in atomic_to_histogram
hist = scatter_add(item, idx, dim=0)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/torch_scatter/scatter.py", line 29, in scatter_add
return scatter_sum(src, index, dim, out, dim_size)
File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/torch_scatter/scatter.py", line 19, in scatter_sum
size[dim] = int(index.max()) + 1
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 4.27 GiB (GPU 0; 11.77 GiB total capacity; 8.82 GiB already allocated; 1.98 GiB free; 9.27 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

Hi, our standard configs for this project assume a 32G GPU (eg NVIDIA V100). However, it is possible to run SPT on smaller GPUs. We have tested with an 11G GPU, with some small impact on time and performance.
I just pushed a new commit with configs in configs/experiments for training on 11G-GPU. I tested preprocessing and training on my own 11G device, these should work for your 12G GPU. Make sure you do not have any other process running on your device though.
Besides, you can find a new Troubleshooting section in the README, with some tips if you run into more CUDA OOM errors.
Hope that helps !

Have you managed to run the code using the new configs ?

I close this issue for now. Feel free to re-open it if relevant.


Use Java 8-compatible variant of Guava

There doesn't seem to be a need for the Java 7-compatible variant (-android) of Guava.
This enables the use of immutable Collectors to be used with the Java 8 stream APIs.

We can introduce Java6CompatiblePredicate, though I personally don’t think it matters much. It’s already easy to find the locations where we use this hack, and I wouldn’t want to encourage the proliferation of similar usages.
What we should do finally is separate the Java 6/7-requiring part, and make that not depend on Guava at all.


Health Bar Created

also works for corruption bar Just change a few things

Made a couple of changes everything looks good to me


CNV-35198: Update links in getting started resources

📝 Description
Update links in Overview -> Getting started resources card
🎥 Demo

Please add a video or an image of the behavior/changes

/approve

/cherry-pick release-4.14


io/ioutil: add WriteNopCloser

The type ioutil.NopCloser allows us to add a no-op Close to an existing Reader when a ReadCloser is required.
For symmetry, there should also be a ioutil.WriteNopCloser to do the same for Write.
The name can be argued about. Get out your bikeshed paint brushes.

:red_circle:

The permutations go from viable to esoteric :smirk::
WriteNopCloser
NopWriteCloser
NopCloseWriter
WriteCloseNopper
CloseWriteNopper
CloseNopWriter

I support this proposal out of practical experience. I just ran across a use case where I needed just this function, and settled for a suboptimal design because it didn't exist.
My application produces a log file whose path is configurable via environment variable. If the variable is provided, it's assumed to be a file path and an *os.File (i.e., an io.WriteCloser) is opened; if the variable is not provided I want to use ioutil.Discard. I wanted to abstract this logic into a function that  returns an io.WriteCloser so I could defer x.Close() in the caller:
func main() {
    logWriter := GetLogWriter()
    defer logWriter.Close()
}

func GetLogWriter() io.WriteCloser {
    if path, ok := os.LookupEnv("LOG_PATH"); ok {
        if file, err := os.Create(path); err == nil {
            return file
        }
    }
    return ioutil.WriteNopCloser(ioutil.Discard)
}

But, having no trivial way to get an io.WriteCloser from an ioutil.Discard, GetLogWriter wasn't possible to write[^1] and I settled for some rather convoluted logic directly in the caller:
func main() {

	var logWriter io.Writer = ioutil.Discard

	if path, ok := os.LookupEnv("LOG_PATH"); ok {
		if file, err := os.Create(path); err != nil {
			panic("sad day")
		} else {
			defer file.Close()
			logWriter = file
		}
	}

}

This works, but it's not great. Specifically, the conditional defer looks like a guaranteed future bug because it breaks from the familiar idiomatic usage.
[^1]: Yes, I could have implemented a variant of ioutil.WriteNopCloser on my own, but the point is that I didn't. I'm settling for more dangerous code instead.

Would love to see this PR accepted!

Just ran across a use case for this as well. I have a function that returns a WriteCloser to be used for caching. If the cache file can't be written to I still want everything to succeed without having to do any major alternative paths in the request handler so instead I made a WriterCloser out of Discard basically how this is and log the cache failure separately.

To restate the argument in favor of adding this, the situation where it comes up is that you need to implement an interface method that is defined to return an io.WriteCloser. And all you have is a io.Writer. The implication is that the caller calls Close to say "all the writes are done", which usually triggers processing of the collected writes. So nopping out Close would be an odd thing to do or to make very easy.
The specific case of ioutil.Discard is unfortunate: we probably should have defined that as its own type instead of an io.Writer, and then it could be changed to have a Close method. We could still do that - add a Close method on the underlying implementation - and document that it is safe to use ioutil.Discard.(io.WriteCloser).
Are there examples other than ioutil.Discard?
The situation is different from Read/ReadCloser because closing a reader is more of a courtesy, compared to "now do something with everything I wrote" for closing a writer.
WriteNopCloser may still be worth adding but we should understand the reasons better first.

I saw an interesting use of WriteNopCloser in age. The age.EncryptWithArmor function takes an io.Writer and returns an io.WriteCloser. When closed, it flushes its buffer and writes a message trailer, but does not close the underlying writer.
https://github.com/FiloSottile/age/blob/a070570595a3f0a154f106b1c0ea7bc76ece1d89/internal/age/age.go#L42
func EncryptWithArmor(dst io.Writer, recipients ...Recipient) (io.WriteCloser, error) {

Internally this is implemented by wrapping the user-provided writer in types that handle encryption and base64 encoding, plus, at the bottom of the chain, the equivalent of a WriteNopCloser to stop propagation of the Close call.
https://github.com/FiloSottile/age/blob/a070570595a3f0a154f106b1c0ea7bc76ece1d89/internal/age/age.go#L81-L84
		// stream.Writer takes a WriteCloser, and will propagate Close calls (so
		// that the ArmoredWriter will get closed), but we don't want to expose
		// that behavior to our caller.
		finalDst = format.NopCloser(dst)

@rsc If you were replying to me, that's essentially what I was getting at. In my case I was making a basic file system cache and I wanted to maintain the illusion to the caller that everything passed even if say the program didn't have writer permission so I put a no-op close method on discard and return that as a WriteCloser.
If perhaps I wanted to have the option to compress my cache file with archive/zip then the Create method on ZipWriter also doesn't have a Close since the data is actually flushed when the overall zip object is closed.

@magical, arguably the example in github.com/FiloSotille/age/internal/age indicates an API defect in github.com/FiloSottile/age/internal/stream.
stream.NewWriter could just as easily have accepted an io.Writer and left the decision about whether and when to Close it up to the caller.

Two weeks ago I asked about motivating use cases. So far there has only been one, which on closer examination turns out not to be that compelling. The fundamental problem is that Close on a WriteCloser is very often semantically meaningful, whereas Close on a ReadCloser often has very little semantic meaning. So a nop'ed ReadCloser is helpful while a nop'ed WriteCloser is more often a mistake.
Based on the discussion so far, this seems like a likely decline.
Leaving open for a week for final comments.

No final comments, so declining.


Handling Backend Ids with Extraction and Publishing

When we create an API using Azure Portal and specify a function/logic app as the backend HTTP endpoint, the portal automatically creates a backend in the APIM and gives it a kind of random name suffixed with a guid string.
Now, when we want to extract a source APIM's specs and upload it into another instance, say from DEV to QA, the publishing will fail because those backend-ids don't exist in the QA instance unless created manually. Because Azure Portal creates this randomly generated backend-id names, it gets complicated to automate the process of creating these backends in other APIM instances and furthermore, we need to point them probably to a different LogicApp/FunctionApp, relevant to that environment. For example, QA may have a different instance of the function app than DEV and hence the URL pointed to by the Backend would be different although the backend-id's would be the same..
How should we handle these cases? I have not found any way to override Backends with the configuration files. Is there some recommended way to handle these seamlessly? I tried using Named Values when specifying the Backend, but Azure Portal doesn't want to accept using a named value as part of the "backend-id" in the Policy Editor for an API.

We have a feature branch that implements supper for backends. We are working on merging this soon. Currently being tested.

Thanks @waelkdouh. Is it the "backends-and-policy-fragments" branch? Is the user manual updated and is it in a state where we could download and try out that branch?

It should be in that branch but I need to confirm with the lead developer. No the docs haven't been updated yet. Once the testing is done the docs will be tested. Feel free to test that branch though and let me know if you find any issues there.

This has been released as part of Release v1.0.0

@waelkdouh Thanks for merging the feature branch. I was just about to ask you about URL override after testing the feature branch yesterday. Then I noticed it was added before the merge.
I still have a few more questions though:

What about attributes like resourceId? If we override the URL using the config file, shouldn't we also override the resourceId attribute to point to the correct logic/function app?

What about overriding the name itself. For example if the source APIM has a backend called backend-01-DEV and in the QA environment we want to name the backend as backend-01-QA?

How do you suggest we handle things like keys to access the function/logic-apps. Should those be pre-created in each function app, or could we generate keys to access the functionapp using AZ cli in the pipeline and then dynamically populate the Named Values in our config files? I suppose this would vary environment to environment, but are there any best practices?

I could look into adding the first two directly in the code as well, since it's probably only a couple of lines of change.

Hi. Please feel free to submit a PR and we will definitely look into it. We are working on a more flexible configuration system, which we hope to push a part of V2.

@devszr - thanks for the great feedback.

Good point, and thanks for finding the bug on URL overrides. I'll take a look at the PR you submitted.
As Wael mentioned, v2 will allow more dynamic configurations. As long as your configuration matches the REST API's body, it should be accepted and make the necessary overrides.

Name overrides don't work well with our model, as we use names as primary identifiers. We recommend having the same APIM resource names and changing the endpoints per environment. In your example, we'd recommend having a backend named backend-01 in both DEV and QA and overriding their URL/resource ID.
Note that this concept holds true across all APIM resources we support. If you have myapi-dev in DEV and myapi-qa in QA, we'd see those as two distinct APIs.

I don't think we're opinionated here. My immediate thought would be to have a step in your pipeline read your function app key and inject its value in the configuration YAML. You could use a tokenizer (which we've tested successfully), or just pass an environment variable in the publisher step with this format (which we've not tested).

I definitely understand the pain point. There are a few other resources in APIM that act similarly with portal auto-generated names (app insight loggers, for instance).
Teams that do a code-first approach with our tool don't have this issue, but many devs have a portal-first approach. One solution would be to have an intermediary step: extract the portal resources, change the generated folder/artifact names to their desired state, and then merge the artifacts into your codebase. I know this is less than ideal though.
Would you mind creating a separate issue for name overrides? This is a larger topic than backends, and our solution should be consistent across APIM resources. We'll discuss how to handle this. In the meantime, I've reviewed your PR and asked to remove the name overrides.

@guythetechie I should've resolved your recommended changes in the PR now. Also, opened a new issue for name overrides. #111

@devszr thanks for your contributions. The changes are now part of the following release


Plugin 0.3.4 joint compilation works on OS X, fails on Windows 8.1

Inside my Android project with mixed Java and Groovy source with all source under src/main/groovy and src/androidTest/groovy, a clean build under OS X works correctly. No errors are reported. The exact same project when built on Windows 8.1 fails with 34 errors. A sampling of the errors that repeat multiple times are below:
CrownAlertDialog.java:80: error: cannot find symbol Constants (Constants is a Groovy file)
error: cannot find symbol com.crown.ivhm.R
error: package R does not exist

I'm having the same exact problem: mixed Java and Groovy sources under src/main/groovy builds fine on OS X. But on Windows 8.1 all references from Java to Groovy cause errors:
error: cannot find symbol Centers
...

I'm closing this after merging #35.


Successful lane run returns non-zero exit code when using resign.sh

New Issue Checklist

[x] Updated fastlane to the latest version
[x] I read the Contribution Guidelines
[x] I read docs.fastlane.tools
[x] I searched for existing GitHub issues

Issue Description

When running our release lane, Fastlane is returning a nonzero exit code even though the lane seemingly succeeds. The re-sign succeeds because we can upload to the App Store as part of our deployment process. The upload succeeds because we see the build in the App Store immediately after running this deployment. The best clue I have to what is failing is this:
rm: _floatsignTemp/old-embedded-profile.plist: No such file or directory
This line comes from fastlane/sigh/lib/assets/resign.sh. resign.sh tries to remove the old embedded plist near the end of the script, line 844:
rm -r "$TEMP_DIR/old-embedded-profile.plist"
This should be rm -f like the other items that get deleted, then the error message that occurs when the file doesn't exist would be suppressed.  Also -r is recursive and this command just removes a single file. This is a bug regardless of whether that actually fixes my issue. Another solution might be to understand why the script tries to remove a file that no longer exists or was never created. I can make a quick fix and PR for this if it's just the simple one character change.
It's also possible something under the hood is going on during the upload process that is causing a failure even though the upload succeeds. This seems less likely since running the build in a non-CI environment works without issues.
Thanks for a great tool and for looking into this!
Complete output when running fastlane, including the stack trace and command used

Get started using a Gemfile for fastlane https://docs.fastlane.tools/getting-started/ios/setup/#use-a-gemfile
-----------------------------
--- Step: import_from_git ---
-----------------------------
Cloning remote git repo...
$ GIT_TERMINAL_PROMPT=0 git clone 'ssh://build-fastlane.git' '/var/folders/7m/njbmxl1n7fs8yrnfnz_ds2c0ykp5fl/T/fl_clone20181010-72810-7in5mm/build-fastlane.git' --depth 1 -n
▸ Cloning into '/var/folders/7m/njbmxl1n7fs8yrnfnz_ds2c0ykp5fl/T/fl_clone20181010-72810-7in5mm/build-fastlane.git'...
$ cd '/var/folders/7m/njbmxl1n7fs8yrnfnz_ds2c0ykp5fl/T/fl_clone20181010-72810-7in5mm/build-fastlane.git' && git checkout HEAD 'fastlane/Fastfile'
$ cd '/var/folders/7m/njbmxl1n7fs8yrnfnz_ds2c0ykp5fl/T/fl_clone20181010-72810-7in5mm/build-fastlane.git' && git checkout HEAD 'fastlane/actions'
------------------------------
--- Step: default_platform ---
------------------------------
Driving the lane 'ios release' 🚀
-----------------------------
--- Step: unlock_keychain ---
-----------------------------
Unlocking keychain at path: appstore.keychain
--------------------------------------
--- Step: get_provisioning_profile ---
--------------------------------------
+-------------------------------------+-----------------------------+
|                     Summary for sigh 2.105.2                      |
+-------------------------------------+-----------------------------+
| skip_install                        | true                        |
| adhoc                               | false                       |
| development                         | false                       |
| force                               | false                       |
| app_identifier                      | game                        |
| username                            | name                        |
| team_id                             | id                          |
| ignore_profiles_with_different_name | false                       |
| skip_fetch_profiles                 | false                       |
| skip_certificate_verification       | false                       |
| platform                            | ios                         |
| readonly                            | false                       |
+-------------------------------------+-----------------------------+
Starting login with user 'user'
Successfully logged in
Fetching profiles...
Verifying certificates...
Found 1 matching profile(s)
Downloading provisioning profile...
Successfully downloaded provisioning profile...
AppStore_game.mobileprovision
Setting Provisioning Profile type to 'app-store'
--- Step: resign ---
/usr/local/lib/ruby/gems/2.5.0/gems/fastlane-2.105.2/sigh/lib/assets/resign.sh game.ipa 34160347E30C8A549A36611137421BC9FC7EDBA4 -p /Users/bamboo/bamboo-agent-home/xml-data/pStore_game.mobileprovision         --keychain-path /Users/bamboo/Library/Keychains/appstore.keychain game.ipa
_floatsignTemp/Payload/hoops.app: replacing existing signature
rm: _floatsignTemp/old-embedded-profile.plist: No such file or directory
Successfully signed game.ipa!
Successfully re-signed .ipa 🔏.
--- Step: upload_to_app_store ---
Login to App Store Connect (game)
Login successful
+--------------------------------------+-----------------------------+
|                      deliver 2.105.2 Summary                       |
+--------------------------------------+-----------------------------+
| ipa                                  | game.ipa                    |
| run_precheck_before_submit           | false                       |
| skip_metadata                        | true                        |
| skip_screenshots                     | true                        |
| screenshots_path                     | ./fastlane/screenshots      |
| metadata_path                        | ./fastlane/metadata         |
| app_version                          | 0.0.1                       |
| username                             | user                        |
| app_identifier                       | game                        |
| platform                             | ios                         |
| edit_live                            | false                       |
| use_live_version                     | false                       |
| skip_binary_upload                   | false                       |
| skip_app_version_update              | false                       |
| force                                | false                       |
| overwrite_screenshots                | false                       |
| submit_for_review                    | false                       |
| reject_if_possible                   | false                       |
| automatic_release                    | false                       |
| phased_release                       | false                       |
| team_id                              | id                          |
| dev_portal_team_id                   | id                          |
| precheck_default_rule_level          | warn                        |
| ignore_language_directory_validatio  | false                       |
| n                                    |                             |
| precheck_include_in_app_purchases    | true                        |
+--------------------------------------+-----------------------------+
Making sure the latest version on App Store Connect matches '0.0.1' from the ipa file...
Successfully set the version to '0.0.1'
Uploading binary to App Store Connect
Going to upload updated app to App Store Connect
This might take a few minutes. Please don't interrupt the script.
Failing task since return code of [/var/folders/7m/njbmxl1n7fs8yrnfnz_ds2c0ykp5fl/T/epb6468691893660389944.sh ios release keychain_password:********] was -1 while expected 0

Environment

✅ fastlane environment ✅
Stack

Key
Value

OS
10.11.4

Ruby
2.5.1

Bundler?
false

Git
git version 2.17.0

Installation Source
/usr/local/bin/fastlane

Host
Mac OS X 10.11.4 (15E65)

Ruby Lib Dir
/usr/local/Cellar/ruby/2.5.1/lib

OpenSSL Version
OpenSSL 1.0.2o  27 Mar 2018

Is contained
false

Is homebrew
false

Is installed via Fabric.app
false

Xcode Path
/Applications/Xcode_8.app/Contents/Developer/

Xcode Version
8.0

System Locale

Variable
Value

LANG
en_US.UTF-8
✅

LC_ALL

LANGUAGE

fastlane files:
`./fastlane/Fastfile`
default_platform(:ios)

platform :ios do
    desc "Submit an IPA to the App Store"
    lane :release do |options|
        default_keychain_path = File.expand_path("~/Library/Keychains/appstore.keychain")
        default_signing_identity = "iPhone Distribution"
        default_bundle_path = find_bundle(:ipa)

        unlock_keychain(
            path: (options[:keychain_path] or default_keychain_path),
            password: options[:keychain_password]
        )

        get_provisioning_profile(
            skip_install: true
        )

        resign(
            ipa: (options[:bundle_path] or default_bundle_path),
            signing_identity:  (options[:signing_identity] or default_signing_identity),
            keychain_path: (options[:keychain_path] or default_keychain_path)
        )

        upload_to_app_store(
            ipa: (options[:bundle_path] or default_bundle_path),
            run_precheck_before_submit: false,
            skip_metadata: true,
            skip_screenshots: true
        )

        clean_build_artifacts()
    end
end

error do |lane, exception|
    if lane_context[SharedValues::PLATFORM_NAME] == "ios"
        clean_build_artifacts()
    end
end

def find_bundle(bundle_type)
    bundle_glob = { :ipa => "**/*.ipa", :apk => "**/*.apk" }

    ipas = []

    Dir.chdir("..") do
        ipas = Dir[bundle_glob[bundle_type]]
    end

    if ipas.length > 1
        UI.user_error!("More than one IPA or APK found to upload. Please specify which you want to upload using the bundle_path parameter.")
    end

    if ipas.length == 0
        UI.user_error!("No IPAs or APKs found in the current directory and subdirectories. Please specify a location using the bundle_path parameter.")
    end

    return ipas[0]
end

`./fastlane/Appfile`
##### iOS #####

app_identifier("com.fakegame") # The bundle identifier of your app
<EMAIL_ADDRESS># Your Apple email address

itc_team_id("itc_team id") # iTunes Connect Team ID
team_id("team_id") # Developer Portal Team ID

##### Android #####

package_name("com.fakegame")

fastlane gems

Gem
Version
Update-Status

fastlane
2.106.0
✅ Up-To-Date

Loaded fastlane plugins:
No plugins Loaded
Loaded gems

Gem
Version

did_you_mean
1.2.0

slack-notifier
2.3.2

atomos
0.1.3

CFPropertyList
3.0.0

claide
1.0.2

colored2
3.1.2

nanaimo
0.2.6

xcodeproj
1.6.0

rouge
2.0.7

xcpretty
0.3.0

terminal-notifier
1.8.0

unicode-display_width
1.4.0

terminal-table
1.8.0

plist
3.4.0

public_suffix
2.0.5

addressable
2.5.2

multipart-post
2.0.0

word_wrap
1.0.0

tty-screen
0.6.5

tty-cursor
0.6.0

tty-spinner
0.8.0

babosa
1.0.2

colored
1.2

highline
1.7.10

commander-fastlane
4.4.6

excon
0.62.0

faraday
0.15.2

unf_ext
<IP_ADDRESS>

unf
0.1.4

domain_name
0.5.20180417

http-cookie
1.0.3

faraday-cookie_jar
0.0.6

fastimage
2.1.3

gh_inspector
1.1.3

json
2.1.0

mini_magick
4.5.1

multi_json
1.13.1

multi_xml
0.6.0

rubyzip
1.2.2

security
0.1.3

xcpretty-travis-formatter
1.0.0

dotenv
2.5.0

bundler
1.16.1

faraday_middleware
0.12.2

naturally
2.2.0

simctl
1.6.5

uber
0.1.0

declarative
0.0.10

declarative-option
0.1.0

representable
3.0.4

retriable
3.1.2

mime-types-data
3.2016.0521

mime-types
3.1

little-plugger
1.1.4

logging
2.2.2

jwt
2.1.0

memoist
0.16.0

os
0.9.6

signet
0.8.1

googleauth
0.6.2

httpclient
2.8.3

google-api-client
0.21.2

emoji_regex
0.1.1

msgpack
1.2.4

bootsnap
1.3.2

openssl
2.1.0

date
1.0.0

fileutils
1.0.2

ipaddr
1.2.0

generated on: 2018-10-10

I merged that fix, but I agree that this wouldn't cause an erroneous nonzero exit code

Okay, thanks for the fix. I'll test it when the new version comes out. Sounds like it won't address this. I can do more experimentation to see what the actual cause might be.

@goraths Did you have time to review the new fastlane releases?

I did. I'm not on the latest latest release, but I ran a deployment on 2.106.1, which I believe has the change to fix the rm -r line. It didn't fix the non-zero exit code I was seeing. I'll have to come up with a different test case to repro this to see if I can get better debug output because what I'm seeing is not enough to tell what's going wrong.


Add Additional Documentation on Manual Linux Build

Added clarifying steps for the manual Linux build.

Jay - Approved


typo: Fix the cmd grpc port in the docs

Codecov Report

Merging #1097 (8ed10bd) into main (c48c42b) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main    #1097   +/-   ##
=======================================
  Coverage   56.61%   56.61%
=======================================
  Files         136      136
  Lines        9063     9063
=======================================
  Hits         5131     5131
  Misses       3391     3391
  Partials      541      541

Impacted Files
Coverage Δ

cmd/flags_core.go
42.37% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

This PR doesn't seem to be touching IT. If they're a bit flaky, can someone restart them please?

Restarted, but we can merge anyway.


Include JWT secret in test app startup message

Summary
That should make it easier for people to find the secret and serves as a reminder to set it.
Ticket Link
Fixes #392

@javaguirre which docs are you refering to?

https://developers.mattermost.com/integrate/apps/

@hanzei I see it's ACTUALLY in the docs, I don't know why I didn't get there 🤦
https://developers.mattermost.com/integrate/apps/quickstart/quick-start-jwt/#download-and-start-the-app


Upgrade app.go with upgrade handler using the semver of the current app

Background
TODO:
https://github.com/palomachain/paloma/blob/d9731ff5c2caf4f81ad6d8c15dd5f80544ae0e61/app/app.go#L350-L352
s/palomad/app.Version()/ simplified
Done when

[ ] a list of items, when checked, this ticket should be considered to be finished

moved to Volume


Updating dependencies for larsoft

trigger build with DUNE/duneopdet#25

:heavy_check_mark: CI build for LArSoft Succeeded on slf7 for e20:prof -- details available through the CI dashboard

:heavy_check_mark: CI build for LArSoft Succeeded on slf7 for c7:prof -- details available through the CI dashboard

:warning: CI build for DUNE Warning at phase ci_tests DUNE on slf7 for c7:prof -- details available through the CI dashboard
:rotating_light: For more details about the warning phase, check the ci_tests DUNE phase logs
parent CI build details are available through the CI dashboard

:warning: CI build for DUNE Warning at phase ci_tests DUNE on slf7 for e20:prof -- details available through the CI dashboard
:rotating_light: For more details about the warning phase, check the ci_tests DUNE phase logs
parent CI build details are available through the CI dashboard


training with default.yaml

hi,
want to train this network to my custom data i have prepared everything and all the data just as stated in the repo but when i train it it gives me this error
Traceback (most recent call last):
File "train.py", line 73, in
model = config.get_model(cfg, device=device, dataset=train_dataset)
File "/home/mazharnaveed804/experimental/occupancy_networks/im2mesh/config.py", line 76, in get_model
cfg, device=device, dataset=dataset)
File "/home/mazharnaveed804/experimental/occupancy_networks/im2mesh/onet/config.py", line 35, in get_model
encoder_latent = models.encoder_latent_dict[encoder_latent](
KeyError: None
Now i don't know how can i solve it please help.

it is giving me the same error when i try to train on the preprocessed shapenet data provided in this repo

it is giving me the same error when i try to train on the preprocessed shapenet data provided in this repo

Me too.

I met the same problem too.
After a lot of trial-and-error, I found that the following scripts are able to run successfully.
python3 train.py configs/img/onet.yaml
python3 train.py configs/pointcloud/onet.yaml
python3 train.py configs/voxels/onet.yaml

Just would like to share my findings and hope it will help you.

I met the same problem too. After a lot of trial-and-error, I found that the following scripts are able to run successfully.
python3 train.py configs/img/onet.yaml
python3 train.py configs/pointcloud/onet.yaml
python3 train.py configs/voxels/onet.yaml

Just would like to share my findings and hope it will help you.

Hello, there are a lot of yaml files in the whole configuration file in the file, do they all mean anything different?
We look forward to your reply thanks.


babel script.jsx -o script.js fails for ch10 tooltip code

Had to move script.jsx to a directory outside ch10 tooltip folder to convert script.jsx to script.js with the command: babel script.jsx -o script.js
Issuing the command above inside ch10 tooltip folder fails with error:
Error: Couldn't find preset "react" relative to directory "../tooltip"
    at /usr/lib/node_modules/babel-cli/node_modules/babel-core/lib/transformation/file/options/option-manager.js:293:19
    at Array.map (<anonymous>)
    at OptionManager.resolvePresets (/usr/lib/node_modules/babel-cli/node_modules/babel-core/lib/transformation/file/options/option-manager.js:275:20)
    at OptionManager.mergePresets (/usr/lib/node_modules/babel-cli/node_modules/babel-core/lib/transformation/file/options/option-manager.js:264:10)
    at OptionManager.mergeOptions (/usr/lib/node_modules/babel-cli/node_modules/babel-core/lib/transformation/file/options/option-manager.js:249:14)
    at OptionManager.init (/usr/lib/node_modules/babel-cli/node_modules/babel-core/lib/transformation/file/options/option-manager.js:368:12)
    at File.initOptions (/usr/lib/node_modules/babel-cli/node_modules/babel-core/lib/transformation/file/index.js:212:65)
    at new File (/usr/lib/node_modules/babel-cli/node_modules/babel-core/lib/transformation/file/index.js:135:24)
    at Pipeline.transform (/usr/lib/node_modules/babel-cli/node_modules/babel-core/lib/transformation/pipeline.js:46:16)
    at transform (/usr/lib/node_modules/babel-cli/lib/babel/util.js:50:22)

Install babel-cli and babel-preset-react:
npm i babel-cli babel-preset-react
Then try the command again: babel script.jsx -o script.js

The jsx files need to be transpiled into js (javascript) files. This requires babel to be installed on the operating system in addition to node packages that ideally should be installed to the parent folder containing all project files for the chapter. The package.json file should also be installed at that level with instructions to facilitate transpiling jsx code to js. The basic node packages should be installed via the following command prompt commands at the level of the package.json file that is above all the project folders - in this case inside "ch04" folder:
npm i babel-cli babel-preset-react --save-dev
npm i react react-dom --save-dev
npm audit fix

Inspect the package.json file immediately under section on ( "scripts": { ... } ).
See:
"scripts": {
"test": "echo "Error: no test specified" && exit 1",
"build-clock": "./node_modules/.bin/babel clock/jsx -d clock/js -w",
"build-clock-es6": "./node_modules/.bin/babel clock-es6/jsx -d clock-es6/js -w",
"build-clock-analog-digital": "./node_modules/.bin/babel clock-analog-digital/jsx -d clock-analog-digital/js -w",
"build": "npm run build-clock & npm run build-clock-es6 & npm run build-clock-analog-digital"
},
Note:  JSON comments take the form of an underscore "_" preceding an identifier. For example, if require "build-clock-analog-digital" to be commented-out, then do this: "_build-clock-analog-digital" : { ... }
Once all the jsx files are ready, then run the following command, at the level of the package.json file, for transpiling the jsx files to js for the current project (in this case "clock"):
npm run build-clock
Then launch another console session, inside of the project folder (clock), and issue the following command to start an static server on port 3030:
static -p 3030
Then launch a web browser and point the url to localhost:3030


Any sample which shows the use of TicketedSessionAuthentication

http://pubs.vmware.com/vsphere-50/index.jsp#com.vmware.wssdk.apiref.doc_50/vim.vm.guest.TicketedSessionAuthentication.html
The example may look like this :
auth = vim.vm.guest.TicketedSessionAuthentication(ticket=base_64_string)
guest_auth = si.content.guestOperationsManager.authManager.AcquireCredentialsInGuest(vm=vm, requestedAuth=auth)
    processes = si.content.guestOperationsManager.processManager.ListProcessesInGuest(vm=vm, auth=guest_auth)
    si.content.guestOperationsManager.authManager.ReleaseCredentialsInGuest(vm=vm, auth=guest_auth)

I am not able to proceed forward ! AcquireCredentialsInGuest is returning error "The guest operating system does not support the operation"

I know this is an old (likely abandoned ticket), but if you take a look at the vSphere API doc:

NamePasswordAuthentication contains the information necessary to authenticate within a guest using a name and password. This is the typical method for authentication within a guest and the one currently used by VIX. This method of authentication is stateless.

The 'one currently used by VIX' part makes me think you're not going to find an example.

Hi,
I took a quick look at the API doc:
https://pubs.vmware.com/vsphere-6-5/index.jsp?topic=%2Fcom.vmware.wssdk.apiref.doc%2Fvim.vm.guest.TicketedSessionAuthentication.html
I'm afraid the first line not accurate:
"auth = vim.vm.guest.TicketedSessionAuthentication(ticket=base_64_string)", it should be "vim.vm.guest.SSPIAuthentication".
(1) A TicketedSessionAuthentication object will be returned as the result of a successful call to AcquireCredentialsInGuest.
(2) Then in AcquireCredentialsInGuest, for SSPI authentication, requestAuth should be of the type SSPIAuthentication.
https://pubs.vmware.com/vsphere-6-5/index.jsp?topic=%2Fcom.vmware.wssdk.apiref.doc%2Fvim.vm.guest.SSPIAuthentication.html
I did not try this, but you can have a try on using "SSPIAuthentication", and refer to the info in above link.
thanks,
Diane

I was trying out SSPIAuthentication. But got the same error.

Also I am not sure if I obtained the sspiToken for passing onto vim.vm.guest.SSPIAuthentication in the correct way.
Can anyone please explain how to obtain this sspiToken?

Sample code: https://github.com/RicterZ/PySharpSphere/blob/main/pysharpsphere/lib/sharp_sphere.py#L126-L151


Restrict organization, funder to ROR entities and support "typeahead" / autocompletion

See glossary for full list of fields with typeaheads. Hazel to provide more details on source of truth for org, funder fields. Should admins curate those lists? If so, we need tables and interfaces. What about external services that provide them?
For tags, the source of truth has to be us, right? So we have to either have admins curate the tags, or allow respondents to both use existing tags and add new tags.

Typeahead list of organizations and funders: https://ror.readme.io/docs/create-ror-powered-typeaheads
I think we should separate this into two issues since "tags" will require a custom and editable list.

I think we should separate this into two issues since "tags" will require a custom and editable list.

Good call

How's this issue title @hazelshapiro ?

Looks good thanks!


Grab golint from golang.org

/domain @Betterment/test_track_core
/no-platform
/cc @reggieag @andimrob
We were depending on a version of golint from github and from a branch that no longer exists. This updates our dependency to a version from golang.org, which (I think) is the canonical source.

<< domain LGTM


Validate Rancher HA installs on k3s clusters.

According to our staging documentation, https://staging.rancher.com/docs/rancher/v2.x/en/installation/k8s-install/kubernetes-rke/, we start K3s without deploying traefik. This results in no ingress being deployed and Rancher unaccessible through the UI.
Deploying k3s with traefik by default allows Rancher to be accessed through the UI in an HA install.
curl -sfL https://get.k3s.io | sh -s - server \
--datastore-endpoint="mysql://<username>:<Password>@tcp(<hostname>:3306)/<data-store name>" \

Verified with Rancher server rancher\rancher:master-head version 1ce03e7
Steps:
Create a mysql datastore in aws.
Install k3s on 2 linux instances:
curl -sfL https://get.k3s.io | sh -s - server \
--datastore-endpoint="mysql://<username>:<Password>@tcp(<hostname>:3306)/<data-store name>" \

Set LB and route 53
Set kubectl
Install Rancher:
helm install rancher rancher-alpha/rancher --namespace cattle-system --set hostname=<hostname> --version 2.4.0-alpha1  --set rancherImageTag=master-head

Existing automation passes on a 3 node custom cluster, all roles.

k3s HA install has been validated using mysqkl and postgresql for AWS datastores.

HA install validated for k3s version 1.17.2+k3s1
Traefik version: 1.7.19
kubectl and exec shell work in the following browsers:
Chrome version 80.0.3987.122
Firefox version 73.0.1
Safari version 13.0.5 (15608.5.11)
Previous issues prevent Traefik from being the recommended ingress controller over nginx:
https://github.com/rancher/rancher/issues/7425#issuecomment-272999869

on the setup K3s v1.17.3+k3s1 (HA mode)+ external etcd v3.3 + Rancher v2.4.0-rc1, I can exec shell and view logs of workloads deployed in local and user clusters(cluster provisioned by Rancher).

Waiting on new K3s release with nginx as default provider for Ingress.
Need to re test using this K3s version with Mysql 5.7 and MariaDB.

HA install validated for k3s version v1.17.3-alpha1+k3s1
MariaDB version: 10.3.20
kubectl and exec shell work in the following browsers:
Chrome version 80.0.3987.132
Firefox version 73.0.1
Safari version 13.0.5 (15608.5.11)
We will use Traefik as the default provider for ingress. Testing was done using traefik:1.7.19

hold for validating Rancher in k3s HA in an air-gapped environment.


🐛 Bug Report — node:util#inherits implementation breaks with jws/jswebtoken npm packages

Repro:

<EMAIL_ADDRESS>compat_flags = ["experimental:nodejs_compat_v2"]
install and use jwshttps://www.npmjs.com/package/jws (or use https://www.npmjs.com/package/jsonwebtoken which uses jws`)

Expected: it works
Actual:
 Uncaught TypeError: The "superCtor.prototype" property must be of type object. Received undefined
    at null.<anonymous> (node:util:106:15) in inherits
    at null.<anonymous>
  (.../node_modules/jws/lib/data-stream.js:39:6) in
  node_modules/jws/lib/data-stream.js
    at null.<anonymous> (index.js:12:50) in __require
    at null.<anonymous>
  (.../node_modules/jws/lib/sign-stream.js:3:18) in
  node_modules/jws/lib/sign-stream.js
    at null.<anonymous> (index.js:12:50) in __require
    at null.<anonymous>
  (.../node_modules/jws/index.js:2:18) in
  node_modules/jws/index.js
    at null.<anonymous> (index.js:12:50) in __require
    at null.<anonymous>
  (.../node_modules/jsonwebtoken/decode.js:1:11) in
  node_modules/jsonwebtoken/decode.js
    at null.<anonymous> (index.js:12:50) in __require
    at null.<anonymous>
  (.../node_modules/jsonwebtoken/index.js:2:11) in
  node_modules/jsonwebtoken/index.js
   [code: 10021]

more context: internal CF chat

@jasnell https://github.com/irvinebroque/node-util-repro

My worker code that reproduces the problem:
`
import jwt from "jsonwebtoken";
import JwkToPem from "jwk-to-pem";
import memoizer from "lru-memoizer";
import ms from "ms";
const client = memoizer({
load: async (
kid: string,
callback: (err: Error | null, key: string | null) => string
) => {
await fetch("https://api.cloudflare.com/client/v4/internal/certs")
.then((r) => {
return r.json();
})
.then((r) => {
const resp = r as JWKSResponse;
if (!resp.keys || !resp.keys.length) {
return callback(new Error("no keys in JWKS endpoint response"), null);
}
    const signingKey = resp.keys.find((key) => {
      return key.kid == kid;
    });

    if (!signingKey) {
      return callback(
        new Error("the JWKS endpoint did not contain any signing keys"),
        null
      );
    }

    return callback(null, JwkToPem(signingKey));
  });

},
hash: (kid) => kid,
maxAge: ms("8h"),
max: 4,
});
async function handleRequest(
request: Request,
env: Env,
ctx: ExecutionContext
) {
return new Response("", {
status: 200
});
export default {
fetch: handleRequest,
};
`

Ok, quick investigation shows that the bug here is not in workerd or our Node.js built-ins. The issue is actually in the way wrangler is transforming the source internally. Specifically, the line that is blowing up is util.inherits(DataStream, Stream) in the data-stream.js file in the jws module. However, if you look at the way wrangler is transforming things, the line const Stream = require('stream'), which in Node.js and in Workerd returns the Stream class, the transpiled code is instead producing a copy of the module namespace object..
Specifically, in the transpile code, Stream ends up as...
{
  Duplex: [Getter],
  PassThrough: [Getter],
  Readable: [Getter],
  Stream: [Getter],
  Transform: [Getter],
  Writable: [Getter],
  _isUint8Array: [Getter],
  _uint8ArrayToBuffer: [Getter],
  addAbortSignal: [Getter],
  compose: [Getter],
  destroy: [Getter],
  finished: [Getter],
  isDisturbed: [Getter],
  isErrored: [Getter],
  isReadable: [Getter],
  pipeline: [Getter],
  promises: [Getter]
}

I would suspect that this is a limitation of the the transpiler that is being used.

Issue opened in workers-sdk. Closing this one.

Thanks for looking into this @jasnell, let's continue the convo on the wrangler issue. Thanks for filing it!

FWIW: replacing jswebtoken with jose makes all these issues go away.
I dug through the _worker.js that sveltekit cloudflare adapter generates and found the require('node:buffer') in there which tripped me up.


Update support for RPi

Follow these recommendations, so it works out of the box :smile:
http://www.gepatto.nl/openfl-4-7-on-a-raspberry-pi-2-or-3/

Done :smile:


Implement search/jump to username in submission table

We'd like to request that more tools be added to the instructor's submissions page
to more quickly find the submission of a specific student.  In a course with 600+
students, it's awkward to find a specific student.  This feature could come in one
of several forms:
-- A search box for the student's name/userid
-- Splitting the pages to display by first initial rather than
the number of students on the page so it would be easier to
judge which page to jump to.

@aimenk @jerboaa status on this? Aimen if you haven't looked too much into this and don't mind, I'd be interested in tackling this ;)

@hansonwu Since @aimenk is working on user preferences + showing and hiding columns in various tables I think it would be fair to say that this one is up for grabs :)

Yea feel free to take it :)

@jerboaa What did you mean by on-the-fly? I assume as in actually querying the DB?
Ps. Can you officially assign to me? :)

@hansonwu On-the-fly as in reduce result set as you type. This would mean to issue one query for each character, which I think would not perform nicely.

@jerboaa gotcha :)


Feedback on the web version of the library

Hi @wouterken,
I've developed a simple rocket web app that wraps the converter in a web API: https://github.com/ecostanzi/html2adf-api/blob/main/src/main.rs
Since I've never developed anything with rust I'm interested in having feedback about it. Let me know if you see any problems or improvements in my implementation.
Thank you for your work!

Nice work @ecostanzi! Looks like a good first cut to me. To productionize it there is of course room for some additional logging, error handling etc - the usual suspects - but depending on your use-case that may be overkill!
Just for fun you could probably even take this one step further and deploy it as a wasm module on a Cloudflare worker, to have the conversion run on the edge, to get the converted strings back to your users even faster!


Card requires Frigate running with port 5000 exposed

Is your feature request related to a problem? Please describe.
Frigate doesn´t have any authentication mechanism and by default it exposes port 5000 without any authentication mecanism, so anyone inside the network can access all camera feeds, snapshops, recordings, etc. There are options to add an authentication layer, for instance adding a reverse proxy with basic authentication and stop exposing port 5000, but after doing that this card stops working as it get data from port 5000
Describe the solution you'd like
I would like to have an option to secure Frigate with authentication and still being able to use this card
Describe alternatives you've considered
The only alternative I can see at the moment is leaving the port 5000 exposed

The Frigate card does not communicate directly to Frigate in any way. All communication is via the Frigate integration in Home Assistant, which proxies requests through to the Frigate backend. As long as you protect your Home Assistant endpoint, using the Frigate card should "just work", since the Frigate integration will proxy your traffic to Frigate backend behind the scenes.
The traffic between HA and the Frigate backend itself is not authenticated in any way, but that is not a problem/issue with the card -- rather Frigate itself, in which case your request is actually for this repo: https://github.com/blakeblackshear/frigate .

Thank you for the clarification and such a quick response, I will open this issue/question to Frigate repo, happy to close this issue now

@Heisenberg2980 You are very welcome. Just keep in mind before you open the issue that what I think you're asking for will already work out of the box without exposing anything extra to the internet, it's just that the local connectivity between HA and your Frigate instance (assuming they are on the same local network) will not be authenticated. That may not necessarily be a priority for the Frigate backend developers to address.

@dermotduffy yes you are right, it is not an actual issue but more a security concern, as having something with a port exposed in your local network without any type of security, and specially talking about camera feeds, is a bit worrying. Just as an example, any device within your network (including IOT devices) could access Frigate and send all the data to an internet server

@Heisenberg2980 Completely understand, just wanted to make sure you knew the magnitude of the security concern -- sounds like you do. Good luck!

Also, I'd like to mention that, if you install Frigate through the Home Assistant add-on, the port 5000 will not be exposed to your internal network.

Issue open to Home Assistant Frigate integration: https://github.com/blakeblackshear/frigate-hass-integration/issues/449


chore(derive-alloy): Docs

Description
Small PR adding docs to kona-derive-alloy.

📚 $\text{Stack Overview}$
Pulls submitted in this stack:

#763 👈
main

This comment was automatically generated by st.


INFRA-2348: Fix ECL serialization

https://jira.ihtsdotools.org/browse/INFRA-2348

@zoliMolnar I've just read your comments on the INFRA ticket about non-deterministic behaviour and unit tests. Understood. Thank you.

Thank you!


Dynamic combo box items

The problem
I would like to dynamically populate the combo box items by user input, such as when searching a large collection, retrieved from a remote resource
The solution
Looking at the storybook, it does nota appear possible to hack the combo box web component currently.
A minimal change would require that there is a value-changed kind of event which I could listen to and load matching elements for rendering from an API etc.
Application/website
private
Business priority
Low Priority = release date is not dependent on fix or not upcoming
What time frame would this ideally be needed by (if applicable)
No response
Examples
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct
[X] I checked the current issues for duplicate issues

@emyarod to investigate

@tpluscode would listening to the input event fit your use case? if not can you elaborate on the changes you are expecting?

Sorry for coming back to this after much time.
It does not have to be super complicated. I thought I would go with manually replacing the bx-combo-box-item elements. Here's how that could be
class MyDynamicComboBox {
  @property({ type:Array }) items = []

  render() {
    return html`
      <bx-combo-box @bx-combo-box-filtered="${this.loadItems}">
        ${this.items.map(this.renderItem)}
      </bx-combo-box>
    `
  }

  renderItem(item) {
    return html`<bx-combo-box-item value=${item.value}>${item.label}</bx-combo-box-item>`
  }

  async __loadItems(e) {
    this.items = await fetchItems(e.detail.searchText)
  }
}

As long as I can listen for user input in the text field (bx-combo-box-filtered), I'm good to go.
One small necessary detail would be to debounce the fetchItems call. This could be part of the combo itself but not sure.

Additionally, in case of large collections to search, I know I will be fetching the data in pages.
In such scenario, it would be nice to have an event triggered when the menu-body is scrolled to the bottom. Something in the lines of
<menu-body @scroll=${this.notifyBottom}>

notifyBottom(e) {
  if (e.target.scrollTop + e.target.getBoundingClientRect().height == e.target.scrollHeight) {
    this.dispatchEvent(new CustomEvent('bx-combo-box-scrolled-to-bottom'))
  }
}

I could then handle this event to fetch subsequent pages dynamically in an infinite scroll fashion. WDYT?


Unable to the join video room without giving any of audio or video access permission or both.

Currently I am facing an issue like if I don't give any audio or video permission or both, then I am getting some error.
Is there any way to bypass this permission things and overcome with like if the user gives permission or not, the user can join the video room?
How to work on this scenario?
Please help me on this one. Thanks in advance.

Thanks for the question @imbiswajit!
The behavior you describe is intentional. You can see here, the 'Join Room' button is disabled when isAcquiringLocalTracks is true. We won't let the user join the room while the media permission dialog (provided by the browser) is still pending.
You can get around this by removing the isAcquiringLocalTracks variable from the line I referenced above. However, we don't have a mechanism to publish local media after a user has joined a room. This functionality would have to be built.
Also, the error that is emitted when you deny media permission to the browser doesn't stop any users from joining the room. You can still join a room after denying permissions. But if you want to suppress the error, you can always remove the catch() blocks from the useLocalTracks.ts file (although we don't recommend this).


⚠ Fortnite has degraded performance

In 49a79a8, Fortnite (https://fortnite.com) experienced degraded performance:

HTTP code: 200
Response time: 1604 ms

Resolved: Fortnite performance has improved in fffe1a0.


nip.io not filtered by my DNS provider

My DNS provider filters local IPs, so this prevented *.nip.io address to work correctly and caused the following error instead :
➜  ~ curl http://deis.<IP_ADDRESS>.nip.io/v2/ && echo
curl: (6) Could not resolve host: deis.<IP_ADDRESS>.nip.io

I first tried to use xip.io instead before realizing this sas caused by my DNS provider which, of course, failed to solve the problem.
I could thinks of these workarounds :

In documentation, in case of failure with *.nip.io, request the user to change DNS provider and use another provider such as Google and update DNS settings to something like <IP_ADDRESS> / <IP_ADDRESS>.
Add entries to /etc/hosts

Maybe there are more friendly solutions. However, I am not sure these work-around are OK for all use-cases.

So for background context, we configure DNS to nip.io and xip.io only for development/testing purposes on minikube. For other cloud platforms we use an actual wildcard route with their DNS provider, and in Configuring DNS we don't even make a mention to nip.io/xip.io.
There's not many DNS providers that block xip and nip.io so I think this is a very small subset of devs that would hit this roadblock, so I'd partially be against telling users to switch DNS providers just to run Workflow.
Another potential solution is to install and configure dnsmasq to point at the node's IP. However, that only works on Linux/macOS and not Windows. I'm not sure of a good cross-system solution other than nip.io/xip.io (or actually setting up proper DNS routes).

I agree dnsmasq is far superior as it does not require to mess with DNS settings and it provides a wildcard local domain. It looks like a good alternative to nip.io for a development/testing environment for macOS and Linux. For Windows, it looks like Acrylic DNS Proxy can also provide such a local wildcard.
On macOS, I used these instructions (see. https://gist.github.com/eloypnd/5efc3b590e7c738630fdcf0c10b68072) :
brew install dnsmasq
echo "address=/deis.local/`minikube ip`" >> /usr/local/etc/dnsmasq.conf
sudo brew services start dnsmasq
sudo mkdir /etc/resolver
echo nameserver <IP_ADDRESS> | sudo tee /etc/resolver/local

Then logged out / login session.

Maybe we can mention alternatives in case of nip.io failure and link to details in minikube docs ?

@ybart I think that is a great idea -- a PR would be welcome!  Perhaps in the minikube section and potentially in the deploy-an-app section?

Here it is:
https://github.com/deis/workflow/pull/850


[DO NOT MERGE] Integration testing

Dummy PR to check the current state of the integration test intermittents and try out possible improvements.

/botio integrationtest

/botio-integrationtest

/botio integrationtest

Closing since I have created issues for all findings.


sqs: each job handler gets its own trace

We want each sqs job handled to have its own trace. This previously
worked properly while we used jaeger, assuming that by submitting tasks
to the handler thread pool does not propagate any existing trace.
However, now that we switched to datadog, it actually automagically uses
code generation to propagate traces across asynchronous execution
boundaries. So now the receiver scheduling thread becomes one giant long
trace that includes every job.
See https://github.com/DataDog/dd-trace-java/tree/master/dd-java-agent/instrumentation/java-concurrent/src/main/java/datadog/trace/instrumentation/java/concurrent
This change forces a new root span for each job.

datadog scares me


Push correct slide number when lazy loading

This commit will push the correct slide number on to the lazyLoadedList when swiping right on the first slide. Before this commit it would push -1 instead of the correct slide number.
This PR is different than #635 in that it pushes the correct the index to the lazyLoadedList.
#635 just handles having the wrong index on the lazyLoadedList
fixes #628

This logic was updated in some commit in a previous release.