Try to prevent long confirmation times during high network load

@zathras-crypto brought this up in another thread.
During high network load, or "stress tests", there is some risk that transactions take quite some time to be confirmed, which may result in an unpleasant user experience.
During the active phase of the recent transaction flood, the number of blocks to confirm for faucet transactions increased from usually 2 blocks to up to 164 blocks. My sample size is limited, and the numbers are scattered, but on average the confirmation time was about 60 blocks, or roughly 10 hours. While the faucet uses pretty low fees, it nevertheless shows a massive (relative) impact.

The following commit was proposed: https://github.com/zathras-crypto/omnicore/commit/8c58429cfc76e8d1d2a9ecebe8c31c3bd6faf26f
When sending transactions, then there is a check to count how many transactions there are in the local memory pool with a fee/kB higher than the transaction to send. If this number is higher than a threshold, a warning dialog is shown to the user, suggesting to raise the fee.
I love the idea! My initial thought regarding this topic was that we shouldn't try to mess with the internal fee estimation etc., and this is a brilliant solution to tackle the issue in a very non-intrusive way, with minimal logic.

Transaction fees are one aspect, but there is also the mintxrelayfee, which we use to calculate output values (i.e. those 0.00000546 BTC, 0.00000684 BTC, ... values).
Appearingly pools raised their treshold to reject "spam" transactions during the active phase, and there were a few posts on Reddit, suggesting to raise the relay fee as well.
Addressing this is more complicated, because I don't really see a good way to measure the impact, and unfortunately there is also no nice UI integration. The confirmation times for recent faucet transactions are back down to 3-4 blocks, so I assume this was only temporary, but in the long run we should prepare for the day, where larger parts of the network, or pools, reject transactions with low output values.


Does superagent have a standalone version for browser?

Do I have to use module bundler to use superagent in browser?
Is there somthing like below?
<script src="/path/to/superagent.js"></script>
<script>
var request = new superagent();
request.get('some/end/point').end(function(err, res){/*...*/});
</script>

Yes. node_modules/superagent/superagent.js works in the browser as-is.

For me var request = superagent; instead of var request = new superagent(); did the trick.

@pornel I installed superagent with npm but I don't have such fodler

@anonym24 npm i superagent; ls -l node_modules/superagent/superagent.js works for me. You definitely should have that after successful install.


[docs] the Filter struct needs documentation

The Filter struct implements the parallel iterator filter() combinator. It needs some simple docs that say precisely that and link to filter() method for more information.

Done in #248.


Feature request

Helo 0tick, great work on this I love it I have some feature request I'm working with this on comfy is a painter node and is amazing but I dont like this scrible brush I would like to see brush system like MyPaint is open source and death but the app is amazing  and the brush system is independent library from the app can you integrate it and add mirror feature; here is some example of what Im working on with paint node and my paint sketches.

The pressure sensitive don't work what I can do.

And when I activate pressure the stroke look faceted here is an example without pressure and with pressure

And when I activate pressure the stroke look faceted here is an example without pressure and with pressure

Is this still occuring? I was not able to replicate it. Regarding your feature request you should probably ask again here as this is the original editor and what i based this project on. I am not familiar on how the brush system is implemented and wont be able to integrate a new brush system.


Change time format to XX:XX:XX

The time is currently displayed as 1:2:3. It should be displayed as 01:02:03.

Can I work on this issue?

Sure! Please go ahead. Apologies for delayed response.


Revoke an API Key

What would be the best way to revoke an API key?

You can just delete it

Yes, what if I want to add new column called is_revoked so if it's true then this API key can't access the API, what simple way to do this kind of check on this package. Thanks!

Hmm.. You'll have to modify the ApiGuardController for that to work. But really, you should just be deleting the ApiKey. The ApiKey has soft deletes enabled so it really isn't deleted from the database.

Yes I was thinking of the soft deletes feature to achieve this. Thank you for your quick response.


Add AWS Advance Shield protection checks @michael-dickinson-sainsburys

If there is an active AWS Shield Advanced subscription, check that supported resources are included  as protected resources.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@Apple root

Thank for this contribution @michael-dickinson-sainsburys, this is pretty helpful.
When you say "active AWS Shield subscription", you do mean Shield Advance, don't you?
I'm going to review the checks and add comments. I'll let you know once are ready for your review.

I think i have covered all the changes requested.
I have also included some of the guidance back into the example check to help others that might contribute to the project

Thanks @michael-dickinson-sainsburys this is a pretty good contribution!

I'm making some additional corrections and I'll push them in a few, FYI:

All commands must have $AWSCLI $PROFILE_OPT to guarantee AWS cli full path and profile used.
no resources found are textInfo
Added Avanced Shield to those part without it.
CHECK_SEVERITY didn't have the check id in it.

Additionally:
shield:GetSubscriptionState
shield:DescribeProtection

Are needed in the ProwlerAdditionsPolicy.


Failed item with "Passed by child" parent sets all the children to "Not started"

An item which has any failed status, with a "Passed by child" parent is not set as "Done". Thus, within the house of cards the item is reset to "Not started" and all of its children are set themselves to "Not started".
The scenario is difficult to replicate in a real flow, because the possibility of a failed item having a PBC parent is highly unlikely.
It can, however, be replicated by starting 5 commits,canceling  commit 1 in Jenkins and setting it to PBC in "Debug mode" of the IDE. Then, commit 2 can be failed, and the above mentioned issue occurs, meaning commits 2,3,4 and 5 are set to "Not started".

fixed in PR 75


Support additional address in listeners output

Please provide a description of this PR:
Support for dual-stack listeners.

LGTM

/cherry-pick release-1.18

@jacob-delgado: once the present PR merges, I will cherry-pick it on top of release-1.18 in a new PR and assign it to you.

In response to this:

/cherry-pick release-1.18

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@jacob-delgado: #44648 failed to apply on top of branch "release-1.18":
Applying: print additional addresses
Using index info to reconstruct a base tree...
M	istioctl/pkg/writer/envoy/configdump/listener.go
Falling back to patching base and 3-way merge...
Auto-merging istioctl/pkg/writer/envoy/configdump/listener.go
CONFLICT (content): Merge conflict in istioctl/pkg/writer/envoy/configdump/listener.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 print additional addresses
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-1.18

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@jacob-delgado: new issue created for failed cherrypick: #44784

In response to this:

/cherry-pick release-1.18

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.


Provide more direct mutable access to shape, strides, and owned Vec

It would useful in some cases to have more direct mutable access to the shape, strides, and data representation. This could be resolved with setters that perform safety checks instead of just providing &mut references to the fields themselves. See #425, in particular this comment for some ideas. This does warrant some more discussion, so I'm creating this issue.

Vec -> We have a hard time exposing a Vec by reference. Ideally we should be able to use our own allocator (let's imagine one where we could request a 64-byte alignment for the data, or something like that).
Sounds good. With strides we have the issue of moving to the more type correct isize values for strides. (What we have today is isize values stored in usize memory.)


Bugfix: Error: Search Backend only searching default admin search fields

Describe the bug

Bug:
The bug occurs when I attempt to search any query. An error message appears saying: "Error from the search backend, only showing results from default admin search fields -Error:[Errno -3] Temporary failure in name resolution."
If the search query is a word in the title of a website, it will return results with that word in it.
If it is only in the wget snapshot of the item, it will not return that item.
Context:
I am running ArchiveBox using on  Windows 10 with docker-compose and have launched the web UI which I am successfully accessing at http://<IP_ADDRESS>:8000. As far as I can tell, all the snapshots are functional and there are no pending links. The output directory is on an external hard drive, but there have been no issues reading/writing from this drive (except for speed, though I can't tell if that's just how the Django Web UI is or not).
Relevant Info:
Bug seems similar to @jdcaballerov comment when search enabled but backend failed in his testing (see screenshot 4).
Steps to reproduce

mkdir archivebox && cd archivebox
curl -O https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/master/docker-compose.yml
Edit the docker-compose yml's volumes section to read:
volumes:
  - ./data:/data
  - D:\\archivebox:/mnt/d/archivebox

(unsure if external drive-specific setup needed to reproduce, so wanted to include)
Open a Windows Terminal in administrator mode, navigate to D:/archivebox/, open a Git Bash tab and run the following:
> docker-compose up -d
> docker-compose run archivebox init
> docker-compose run archivebox manage createsuperuser
> docker-compose run archivebox add 'https://www.dailydot.com/parsec/fandom/dieselpunk-steampunk-beginners-guide/'
Navigate to http://<IP_ADDRESS>:8000 and search "beginners' (see screenshot 1). Because it is in the title, it will show up. The error message will also show up.
Search "biopunk" (see screenshot 2). Even though it is in the wget file, it will not show up (see screenshot 3). The error message will show up. I have not done extensive testing on whether different filetype snapshots will get searched or not, but I don't think it picks any of them up if they are not in title.
 If the search backend can't be queried the search will only occur in the url and title, the admin fields.

@alsokpisz as @jdcaballerov mentioned this is likely a DNS resolving issue inside your docker-compose network. Docker on macOS is infamous for having container DNS issues, so I wouldn't be surprised if Docker on Windows is plagued by similar bugs.
To confirm that Sonic is up and running and accessible from the archivebox container, can you run these python commands manually and report back what output you get:
docker-compose run archivebox /usr/local/bin/python3
>>> import socket
>>> HOST = 'sonic'
>>> PORT = 1491
>>> with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
...     s.connect((HOST, PORT))
...     s.sendall(b'Hello, world')
...     data = s.recv(1024)
...
>>> print('Received', repr(data))
Received b'CONNECTED <sonic-server v1.3.0>\r\n'
>>>

Two arrangements, one based on an empty sonic folder which was created by the docker-compose setup, and then I will also try your suggested arrangement.
Arrangement 1:
--archivebox
   | --docker-compose.yml
   | --data
        | -- archive, logs, sources, A...B.conf, A...B.conf.bak, index.sqlite3
   | --etc
        | -- sonic
              | -- config.cfg (folder)
                    | -- config.cfg (file)

Results:
No selectable checkboxes on the left hand side of the UI. Everything is much much faster. Queries no longer produce the error message but produce the same functionality (still only searching title fields) (see screenshot 1). Missing buttons in top right (Archive, Re-archive, etc).
Observing the Docker containers currently running, despite the terminal saying that the Sonic container was started (screenshot 2), my Docker Desktop shows it has stopped (screenshot 3). This is confirmed by the > docker ps (screenshot 4).
Arrangement 2:
--archivebox
   | -- docker-compose.yml
   | --data
        | -- archive, logs, sources, A...B.conf, A...B.conf.bak, index.sqlite3
   | --etc
        | -- sonic
              | -- config.cfg (file)

Several errors in the terminal (see screenshot 5).
Is the tree I've set up above not correct?
Screenshots
Screenshot 1.

Screenshot 2.

Screenshot 3.

Screenshot 4.

Screenshot 5.

Inside of docker is always linux, so having a Windows path in this docker command doesn't make sense: docker-compose run archivebox C:/Python37/python
Run it verbatim as I posted above, and paste in the script line by line (don't make a file):
docker-compose run archivebox /usr/local/bin/python3

>>> ... paste in lines above here

Causes a subargument issue.

try docker-compose run archivebox shell

Result:
Received b'CONNECTED <sonic-server v1.3.0>\r\nENDED '
I tried this when docker ps still shows both services running.

Great! That means both DNS and the TCP socket to the sonic container are working. Try this next in docker-compose run archivebox shell:
>>> from sonic import SearchClient
>>> from archivebox.config import SEARCH_BACKEND_HOST_NAME, SEARCH_BACKEND_PORT, SEARCH_BACKEND_PASSWORD, SONIC_BUCKET, SONIC_COLLECTION
>>> with SearchClient(SEARCH_BACKEND_HOST_NAME, SEARCH_BACKEND_PORT, SEARCH_BACKEND_PASSWORD) as querycl:
>>>    print(querycl.query(SONIC_COLLECTION, SONIC_BUCKET, 'test'))

Results:
['3ad870d4-82b5-4974-a6ce-ee8cc6a235fa', '5d6734a5-1b9d-418a-a215-8e1e1dbdb8e5', '74696c7d-4421-46b8-8f35-9f1c9537ee1b', 'fca5096f-13da-4d94-8afd-5d742d7b3fb4', '6f009e8d-947a-4fa7-94d7-f21a94c2b525']

Seems like any page which is a .pdf, or .jpg causes this error during the index command:
[*] <link.pdf>
[X] An Exception ocurred reading the indexable content='utf-8' codec can't decode byte 0xb5 in position 10: invalid start byte:
[*] <link>
[*] <link>
[X] The search backend threw an exception=ERR invalid_format(PUSH <collection> <bucket> <object> "<text>" [LANG(<locale>)]?)
:
[*] <link>
[*] <link>

And then it hangs.
After removing all the .pdf links and .jpg which I suspected was causing some problems it kept causing the following error:
[X] The search backend threw an exception=ERR invalid_format(PUSH <collection> <bucket> <object> "<text>" [LANG(<locale>)]?)
:
[*] <link>
[*] <link>
*terminal hangs*

Now there are no errors when I run the re-index command, but it will still spend ages on random pages. Notably stuff with 'weirder' components like live webcam feeds or something. I removed those one by one until it managed to get through the 80ish bookmarks in less than 10 minutes.
The returned results were the same as the admin search, but public index still didn't work.

@alsokpisz the public search view is not connected to the search backend for security and performance reasons.

Well that's that then I suppose.
Is there a way to set a "timeout" per link on the docker-compose run archivebox update --index-only command? So it will skip links it spends more than say, a minute trying to index?

Ok this should be somewhat improved in f67a5a2. It will be out with the next v0.6 release soon.
You can also try it early by adding this line to your docker-compose config: build: https://github.com/ArchiveBox/ArchiveBox.git#dev.
Comment back here if you're still having issues with indexing failures/hanging and I'll reopen the issue.

print('Received', repr(data))

This is what I get when I follow the troubleshooting:
>>> print('Received', repr(data)) Traceback (most recent call last):   File "<stdin>", line 1, in <module> NameError: name 'data' is not defined
Anyone know what I'm doing incorrectly?

Looks like you messed up the indentation, make sure to copy paste that whole block together above, or remove the extra newline before that print to be doubly sure. @jdqw210


Error in tiff images while executing python generate_aitod_imgs.py.

Hello Sir,
I followed the steps as mentioned, and go to the execution part of "python generate_aitod_imgs.py". However, in my terminal, the error depicted in the below given image is being shown. Please do let me know of a way in which I can rectify it and whether the error is in some of the tiff images only or something else. Thanks.

Maybe some error occurs when uncompressing the files, some .tiff files may be broken when using the ".zip" file. It seems that using the provided ".tar" file for uncompressing can solve this problem.

I tried it in the manner you suggested, and downloaded the tar files from xview for both the training set and the labels geojson file. However the same error persists.
I have attached another screenshot that shows the tiff images within the train_images folder (and there appears to be no issues with their names). However if you check the error in the perviously added screenshot, you can find the file name ._100.tiff where i guess its name is what is causing the issue but its not there in the train_images set while checking the file directly for that particular image (._100.tiff).
Please do let me know if there's a way I could go about to help resolve this, and any help you provide will be quite valuable.

Hi,
Is it possible to provide an email address? maybe I can directly send you the ai-tod dataset.

Yes sure thank you, my email id is<EMAIL_ADDRESS>You could share it on this one. After which I just need to replace the train_images folder by the one you send right?

Yes

Ok sir I received the onedrive link, and it seems that with regards to its contents (i.e the aitod dataset) there isn't any issue, as this error popping up in the terminal is purely based on the tiff images within the xview dataset.
I am actually using the aitod dataset for running your algorithm of RFLA, for which I followed the instructions of this page to use the aitod dataset. So can I directly use only the images of aitod part and leave out the xview images for its execution? Or I need to include the tiff images also? Because I am not aware of how to solve the issue thats popping up with regards to the unrecognized tiff files.

The link is the whole aitod dataset (as mentioned in the aitod paper, it is constructed partly based on the xview, so the provided data already contains the xview part), it is the same as what is generated from this repo. You can directly use the dataset from the provided link as training data. And you do not need to include the tiff images.

Oh ok sir. So I should directly include the dataset images and its annotations in a folder within the mmdet-rfla cloned folder before running "python setup.py develop" right??

"python setup.py develop" is for installing the repo,  the organization of  training data can refer to the official guide of (mmdetection)[https://github.com/open-mmlab/mmdetection/blob/master/docs/en/2_new_data_model.md], you need to create a folder named data inside the mmdet-rfla folder and put the images and anns into this data folder, please refer to the  link for details.

Ok sir got it thanks for the help. Finally, after adding the data in this manner, I'd move forward by executing the train.py available within the tools folder right? (I tried it already tho, and that too requests the mmcv version to be between 1.3.2 and 1.4.0 and not the latest) Just wanted to clarify this last part about executing the train.py . Thank you.

Yes, you may need to install the mmcv of the required version and execute the train.py command with the config file.

Hello Sir, I was able to get through with the previously mentioned issues, and while I'm simply using the AI-TOD set of images (leaving out the tiff images), there's an issue with regards to unpacking that I'm facing (screenshot attached below). I am still quite new when it comes to analyzing the code and debugging hence I thought of putting it here to get some ideas from anything that you could help me with.
Please do let me know how to approach this and any help will be really appreciated. Thank you.

Hi, maybe you can enter the mmdet-rfla dir and try to execute:
python tools/train.py configs/rfla/aitod_cascade_r50_rfla_rfla_klf_1x.py

Under the mmdetection framework, some errors may occur if the command is not executed under the correct directory.

Hi Sir, thanks for providing the update, however even after executing the said statement in the mmdet-rfla dir, I have come upon a new error, and I'm not sure how this has come to happen, (I did try to delete and redo the setup of cloning the rfla folder and proceeding). Please do let me know if I should simply reset it and uninstall all packages etc and retry it.

Hi, it seems that there is a mismatch between the installed mmcv and the required mmcv of the mmdet. The mmcv used in my mmdet-rfla is mmcv-full v1.3.8, maybe you can try it.

Sir, I'm having some issue with installing previous versions of mmcv-full in my system as well (like 1.3.8 as you mentioned) so for my openmmlab environment i am thinking of specifically reinstalling the appropriate versions of torch, mmcv and other requirements. Could you specify which all versions are you utilizing for the rfla code (just as you specified about mmcv 1.3.8)?

All packages in the rfla env are shown below. Besides, I used RTX3090 gpu with cuda 11.1:
addict                 2.4.0
aitodpycocotools       12.0.3
albumentations         1.2.0
appdirs                1.4.4
asynctest              0.13.0
attrs                  21.4.0
blessings              1.7
certifi                2022.6.15
charset-normalizer     2.1.0
cityscapesScripts      2.2.0
click                  8.1.3
codecov                2.1.12
colorama               0.4.5
coloredlogs            15.0.1
coverage               6.4.1
cycler                 0.10.0
Cython                 0.29.30
flake8                 4.0.1
gpustat                0.6.0
humanfriendly          10.0
idna                   3.3
imagecorruptions       1.1.2
imageio                2.19.3
importlib-metadata     4.2.0
iniconfig              1.1.1
interrogate            1.5.0
isort                  4.3.21
joblib                 1.1.0
kiwisolver             1.3.1
kwarray                0.6.2
matplotlib             3.4.2
mccabe                 0.6.1
mkl-fft                1.3.1
mkl-random             1.2.2
mkl-service            2.4.0
mmcv-full              1.3.8
mmdet                  2.13.0    /home/xuchang/mmdet-rfla
networkx               2.6.3
numpy                  1.21.5
nvidia-ml-py3          7.352.0
olefile                0.46
onnx                   1.7.0
onnxruntime            1.5.1
opencv-python-headless <IP_ADDRESS>
packaging              21.3
Pillow                 9.2.0
pip                    21.2.2
pluggy                 1.0.0
protobuf               4.21.2
psutil                 5.9.1
py                     1.11.0
pycocotools            2.0.4
pycodestyle            2.8.0
pyflakes               2.4.0
pyparsing              2.4.7
pyquaternion           0.9.9
pytest                 7.1.2
python-dateutil        2.8.1
PyWavelets             1.3.0
PyYAML                 5.4.1
qudida                 0.0.4
requests               2.28.1
scikit-image           0.18.3
scikit-learn           1.0.2
scipy                  1.7.3
setuptools             61.2.0
six                    1.16.0
sklearn                0.0
tabulate               0.8.10
terminaltables         3.1.0
threadpoolctl          3.1.0
tifffile               2021.11.2
toml                   0.10.2
tomli                  2.0.1
torch                  1.10.0
torchaudio             0.10.0
torchvision            0.11.0
tqdm                   4.64.0
typing                 <IP_ADDRESS>
typing_extensions      4.3.0
ubelt                  1.1.2
urllib3                1.26.10
wheel                  0.37.1
xdoctest               1.0.0
yapf                   0.31.0
zipp                   3.8.0

Hi Sir, so I was able to re-install the packages and try the training execution. In this manner, it ran through and through for the given 12 epochs, using "python tools/train.py configs/rfla/aitod_faster_r50_rfla_kld_1x.py". But at this point a RunTimeError turns up (shown as below).

Even though I'm trying to make the changes in the bbox_nms.py folder by adding .cpu() in the inds variables, it isn't working, so I thought of asking you about the same.
The said error carries forward to when I'm trying to evaluate it as well using "python tools/test.py configs/rfla/aitod_faster_r50_rfla_kld_1x.py work_dirs/aitod_faster_r50_rfla_kld_1x/epoch_8.pth --eval mAP".

It seems that this error occurs in the inference stage when using the get_bboxes() function. Maybe you need to verify whether the inputs of the multiclass_nms() function in the bbox_head.py (as indicated in the error "line 371") are in the same device, if not, you need to transfer the input to the same device.

Hey sir, I am able unable to understand exactly how to transfer the input to the same device, as mentioned in my previous comment, I was trying to use .cpu() to get the inds (indices) to the cpu part, however I'm unable to do the same in the function mentioned in your comment above (in bbox_head.py). Can you uggest how exactly should I go about transfering to either the cpu or gpu?

Hi, to my best knowledge, this error occurs when the "inds" and its indexed variables are not in the same device. To tackle this problem, you can try to debug by printing out the device of correlated variables and check whether they are in the same device. Then, you can transfer the problematic variable by command "to(device)", for example, the "device" can be obtained by the command "inds.device". This checking process can be done in bbox_nms, or you can check whether the input in the  multiclass_nms() are in the same device as the "inds".


Grab 'name' key from joinColumn in association mapping for column name.

Each element in the joinColumns array in an association mapping is an associative array. When pulling out the column name, it needs to access array key 'name'.

Hi @jamescdavis - thanks for your contribution!
If I understand correctly this currently results in an error or empty column name? Could you write a regression test reproducing this problem?

I have discovered there are more changes needed to get auditing of entities identified through foreign entities to work, so I am closing this pull request and I will open another (which will include a regression test). The changes in this pull request had worked only because the foreign entity I was testing on happened to have a __toString() that returned the id. My next pull request will include a complete fix.


Implelent log levels

Add log levels
Improve log messages
Add LOG_LEVEL variable in .env.example file
Prevent diverging of branches dev and main and merge features (as discussed in #8)

Thanks for the PR!
I will add more features to debug mode as well.
(Example: when DEBUG mode is on, /getcontext dissapears in telegram bot's menu)


Severe Regression in 3.16.1 Calico - Fixed in 3.16.5

What kind of request is this (question/bug/enhancement/feature request):
Bug
Steps to reproduce (least amount of steps as possible):
Perform a rolling update to etcd (similar to our cluster upgrades) during an update to v1.19
Result:
https://github.com/projectcalico/calico/issues/4109
You will very likely hit this bug which causes watcher errors repeatedly and never passed back (they are swallowed by go-calico)
Upgrading to 3.16.5 should resolve this, I propose we update KDM to have a release addressing this bug as it will impact customers looking to move to v1.19
Other details that may be helpful:
The customer upgraded to v1.19 at 6pm on the 20th and immediately hit this issue, it took time to build up corresponding with the nature of the bug causing a build-up in watchers
gz#13734

On 2.5.2 -

Deployed a Canal cluster on 1.18 k8s version - 3 etcd, 1 control and 3 worker nodes. Enabled cluster monitoring.
Upgraded k8s to 1.19.4-rancher1-1
No logs seen on calico node and calico-kube-controller pods.
No increase in metrics after upgrade.

on rancher server - v2.5.4-rc4 , 1.19.4-rancher1-2 clusters created with "calico" network uses calico - v3.16.5 . Automation runs on these clusters succeed.

Automation checks on Canal clusters - Node driver look good on 2.5.4-rc4 on 1.19.4-rancher1-2 clusters

Closing the issues after https://github.com/rancher/rancher/issues/30324#issuecomment-743783468 and https://github.com/rancher/rancher/issues/30324#issuecomment-743911437 validations


Server recieves but client does not

So im running on AWS beanstalk using apache.
I have my autobahn server runinng on port 8080, as an apache proxy to point to that. It looks a something like this:
<VirtualHost *:80>
    ServerName websocket.foo.com

    ProxyPass /ws ws://<IP_ADDRESS>:8080/ws
    ProxyPassReverse /ws ws://<IP_ADDRESS>:8080/ws
</VirtualHost>

My server runs fine and when i run it in debug mode i can see when a client enters with the following request:
GET /ws/?agent_id=9 HTTP/1.1
Origin: https://app.foo.com
Upgrade: WebSocket
Accept-Language: en-US,en;q=0.8,da;q=0.6
Accept-Encoding: gzip, deflate, sdch
X-Forwarded-Host: websocket.foo.com
X-Forwarded-Port: 443
X-Forwarded-For: x, x
Sec-Websocket-Version: 13
Connection: Upgrade
Sec-Websocket-Key: kZsC4ekgMyfRhXibwhHzeg==
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.107 Safari/537.36
Host: <IP_ADDRESS>:8080
X-Forwarded-Proto: https
Cookie: __cfduid=d1467ccb6824ee8a59e57cbd245af817a1433702820; _gat=1; _ga=GA1.2.196486585.1433702963
Pragma: no-cache
Cache-Control: no-cache
Sec-Websocket-Extensions: permessage-deflate; client_max_window_bits
X-Forwarded-Server: websocket.foo.com

The server now wants to send a response:
HTTP/1.1 101 Switching Protocols
Server: AutobahnPython/0.10.4
X-Powered-By: AutobahnPython/0.10.4
Upgrade: WebSocket
Connection: Upgrade
Sec-WebSocket-Accept: ifFa888u7HRDZgWPYGEhZQkrMhk=

This is were i stop understanding what is going on, because the client ends up with the following error code:
WebSocket connection to 'wss://websocket.foo.com/ws/?agent_id=9' failed: Error during WebSocket handshake: Unexpected response code: 408

Connection closed (wasClean = false, code = 1006, reason = '')

The server however seems to send whatever i tell it to with messages lige these:
2015-07-26 01:58:14+0200 [-] TX Frame to tcp4:<IP_ADDRESS>:52585 : fin = True, rsv = 0, opcode = 1, mask = -, length = 2, repeat_length = None, chopsize = None, sync = False, payload = []
2015-07-26 01:58:14+0200 [-] TX Octets to tcp4:<IP_ADDRESS>:52585 : sync = False, octets = 81025b5d
2015-07-26 01:58:14+0200 [-] TX Frame to tcp4:<IP_ADDRESS>:52605 : fin = True, rsv = 0, opcode = 1, mask = -, length = 2, repeat_length = None, chopsize = None, sync = False, payload = []
2015-07-26 01:58:14+0200 [-] TX Octets to tcp4:<IP_ADDRESS>:52605 : sync = False, octets = 81025b5d

So what is going on?

This is more an APache question. the client gets a HTTP 408 (a timeout) .. I have no clue what Apache is doing here. sorry, we don't use Apache anymore ..

Yeah, changed to docker and started using nginx. Works like a charm.

glad it works for you! using nginx instead of apache: a wise decision;)

Hmm, i went back to this i seem to have the same issue with nginx.

You might try setting the externalPort to the external visible port your Nginx is listening on (if that differs from the port of the actual Autobahn WebSocket server behind). Please see here

We have done that all along :)

On 07 Sep 2015, at 22:53, Tobias Oberstein<EMAIL_ADDRESS>wrote:
You might try setting the externalPort to the external visible port your Nginx is listening on (if that differs from the port of the actual Autobahn WebSocket server behind). Please see here
—
Reply to this email directly or view it on GitHub.


Added two extra functions

added event synchronization function
add event_elapsed_time

Signed-off-by: Dr. Mathieu Taillefumier<EMAIL_ADDRESS>

wanted to add memcpyToSymbol but the hip version has an extra parameter compared to the cuda one


💅 possible difference in scope between biome useliteralkeys and eslint dot-notation rules

Environment information
CLI:
  Version:                      1.7.3
  Color support:                true

Platform:
  CPU Architecture:             x86_64
  OS:                           windows

Environment:
  BIOME_LOG_DIR:                unset
  NO_COLOR:                     unset
  TERM:                         unset
  JS_RUNTIME_VERSION:           "v20.13.1"
  JS_RUNTIME_NAME:              "node"
  NODE_PACKAGE_MANAGER:         "npm/10.5.2"

Biome Configuration:
  Status:                       Loaded successfully
  Formatter disabled:           false
  Linter disabled:              false
  Organize imports disabled:    false
  VCS disabled:                 true

Linter:
  Recommended:                  false
  All:                          false
  Rules:                        unset

Workspace:
  Open Documents:               0

Rule name
useLiteralKeys
Playground link
https://biomejs.dev/playground/?lineWidth=100&indentStyle=space&quoteProperties=preserve&indentWidth=4&code=YwBvAG4AcwB0ACAAdABlAHMAdAAgAD0AIAB7AAoAIAAgACAAIAAiAHQAZQBzAHQAIgA6ACAAMQAsAAoAIAAgACAAIAAiAHQAZQBzAHQAIAAyACIAOgAgADIALAAKACAAIAAgACAAIgB0AGUAcwB0ADMAIgA6ACAAMQAsAAoAfQA7AAoACgB0AGUAcwB0AFsAIgB0AGUAcwB0ACIAXQAgAD0AIAAyAA%3D%3D
Expected result
Hey folks,
There seems to be a possible difference between the scope of biome's useLiteralKeys linter rule and the scope of eslint's dot-notation linter rule. In particular, eslint seems to only apply the rule when the property of an object is being accessed, but biome seems to apply the rule both when the property is being accessed AND when the properties are being defined/created within a new object. For instance:
const test = {
    "hello": 1
}

In this example, eslint won't throw an error, but biome will and suggest to simplify by not using a string literal. You can view results in ESLint here. If that's the intended nature, all good, though would prefer parity with eslint as much as possible to help teams migrating; if that's not the intended nature, then there is probably a bug.
Thanks for your great work on this project, I love using it!
Code of Conduct

[X] I agree to follow Biome's Code of Conduct

I believe this is intentional, but it does not seem to be properly documented.
I question this behavior because our formatter is able to remove unnecessary quotes around properties.
In fact, useLiteralKeys conflicts with the quoteProperties configuration of our JavaScript formatter.
So, personally, I am in favor of reducing the scope of useLiteralKeys to property access.

So, personally, I am in favor of reducing the scope of useLiteralKeys to property access.

It's not very clear what reducing scope means. If I understand correctly, the code action should only remove the brackets and keep the quotes, correct? E.g.
// original
const a = {
  ["f"]: b
}

New code action
const a = {
-  f: b
+  'f': b
}

It's not very clear what reducing scope means.

The rule could only apply to property access:
- a.b["c"];
+ a.b.c;

Your example could be untouched.
However, I noticed that our formatter doesn't remove brackets when they are not needed. This follows Prettier. Should we diverge from Prettier here? It could make sense?

In fact, useLiteralKeys conflicts with the quoteProperties configuration of our JavaScript formatter.

This is actually what tipped me off that something might be funky in the first place. I had set quoteProperties: "preserve" and ran it on this code:
const test = {
    "test": 1,
    "test 2": 2,
    "test3": 1,
};

The formatter preserved the quotes around the properties, but the linter threw errors. The only way to fix was to unquote the "test" and "test3" properties, which seems to contradict the intended nature of quoteProperties: "preserve"
Reducing the scope of useLiteralKeys makes sense to me.

Yeah sounds good. We could consider this a fix, since it oversteps the formatter boundaries


position.speed showing negative values in iPad Pro

🐛 Bug Report
Don't know but on iPad pro position.speed shows negative values, however on other iPhones and Android phone it works fine.
Expected behavior
It shouldn't show negative values.
Reproduction steps
var locationOptions = LocationOptions(
  accuracy: LocationAccuracy.bestForNavigation,
  distanceFilter: 0,
  timeInterval: 0,
);

geoLocator.getPositionStream(locationOptions).listen((Position position) {
  print("Heading = ${position.heading}"); // works fine
  print("Speed = ${position.speed}"); // shows -1.0
  print("Longitude = ${position.longitude}"); // works fine
});

Version:
2.1.0
Platform:
iPad Pro 10.5 inches 2017 running iOS 12.0.1

WiFi enabled iPad doesn't have GPS receiver, so position.speed is always going to be -1. I installed other apps from the app store, and no app was able to provide the speed and at the same time, my iPhone was showing speed. Tested in different conditions but same result.


[5.9] [Walker] Don't visit VarDecls encountered in freestanding macro expansions

Explanation: VarDecls are always walked via their pattern bindings, so we end up
with double-visitation if we also visit them in freestanding macro expansions, causing a compiler assertion.
Scope: Narrow; only code within macro expansions is affected.
Risk: Low due to narrow scope.
Issue:  rdar://109376102.
Original pull request: https://github.com/apple/swift/pull/66202

@swift-ci please test


Set an environment variable in swift test to indicate which testing library is in use.

This PR sets a new environment variable "SWIFT_PM_TEST_LIBRARY" to the name of the active testing library ("XCTest" or "swift-testing".) Code in these libraries can then use the presence of this environment variable to adjust their behaviour.
Resolves rdar://128272138.

@swift-ci please test

@swift-ci please test

@swift-ci please test Windows

@swift-ci please test

@swift-ci please test Windows


Update Data transformations

Review and update as necessary to document all data transformations in ML.NET

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: a4185570-21cc-dcdb-62fd-d05bceb62f81
Version Independent ID: 4f0fe52a-a771-d044-593d-2bb65df15e0d
Content: Data transformations - ML.NET
Content Source: docs/machine-learning/resources/transforms.md
Product: dotnet-ml
GitHub Login: @luisquintanilla
Microsoft Alias: luquinta

@jwood803 can you take a look at this. Thanks.


Improve CI workflow

This is the implementation of the changes discussed in #311:

Extracting the cargo fmt and cargo clippy steps into their own jobs.

@grtcdr: What do you think? Is this OK, or should we combine the two jobs?

Fail clippy on warnings (required some minor code changes).
Execute the "Test" and "Doctor" change on all platforms that support it.

Additionally, I removed the strip variable from the matrix, because it was never used.

The "GNU/Linux x86_64 (x86_64-unknown-linux-gnu)" job seems to fail the "Doctor" stage because of an actual bug in libmacchina: https://github.com/Macchina-CLI/libmacchina/blob/main/src/linux/mod.rs#L831

This looks good to me, let's merge :)


Carousel support

Hi! Does your code has a carousel support?
"You can also render a carousel, allowing you to send a list of items in a compact way."
https://developers.facebook.com/docs/messenger-platform/product-overview
I tried to find it, unsuccessfully.

I'm working on big improvement of API, but I can't find implementation of "Carousel" in documentation, could you please post direct link?
Isn't that just structured template? I will look into it and I will add support, if I find how to send it :D But right now I unfortunately don't have time, but weekend is ahead :)

Ok, thank you! I didn't find too. But I guess it's just an array of structured templates or something similar.
Will wait your update!

Hi! Is there any news about update? Sorry for disturbing.

Just send multiple elements (https://developers.facebook.com/docs/messenger-platform/send-api-reference/generic-template).


Add stream selection support for tf.contrib.ffmpeg.decode_video

This fix is a follow up to #16101. In #16101, stream selection support has been added for tf.contrib.ffmpeg.decode_audio.
However, it was still not possible to selectively decode a perticular stream with tf.contrib.ffmpeg.decode_vidio.
This fix adds an additional attribute stream which could be used to specify the stream of the video to decode. By default stream='' which leaves the decision to ffmpeg.
This fix is related to comment https://github.com/tensorflow/tensorflow/pull/16101#issuecomment-357448137
Signed-off-by: Yong Tang<EMAIL_ADDRESS>

Nagging Assignee @fredbertsch: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @fredbertsch: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @fredbertsch: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Reviewer @fredbertsch: It has been 14 days with no activity and the awaiting review label was assigned. Can you please take a look?

Nagging Reviewer @fredbertsch: It has been 92 days with no activity and the awaiting review label was assigned. Can you please take a look?

Nagging Reviewer @fredbertsch: It has been 124 days with no activity and the awaiting review label was assigned. Can you please take a look?

Nagging Reviewer @fredbertsch: It has been 139 days with no activity and the awaiting review label was assigned. Can you please take a look?

Nagging Reviewer%(reviewers): You have been added as a reviewer to this pull request. Please add your review or reassign. It has been  154ays with no activity and the awaiting review label has been applied.

Nagging Reviewer @fredbertsch: You have been added as a reviewer to this pull request. Please add your review or reassign. It has been 169 days with no activity and the awaiting review label has been applied.

Nagging Reviewer @fredbertsch: You have been added as a reviewer to this pull request. Please add your review or reassign. It has been 184 days with no activity and the awaiting review label has been applied.

Nagging Reviewer @fredbertsch: You have been added as a reviewer to this pull request. Please add your review or reassign. It has been 199 days with no activity and the awaiting review label has been applied.

Nagging Reviewer @fredbertsch: You have been added as a reviewer to this pull request. Please add your review or reassign. It has been 214 days with no activity and the awaiting review label has been applied.

Nagging Reviewer @fredbertsch: You have been added as a reviewer to this pull request. Please add your review or reassign. It has been 229 days with no activity and the awaiting review label has been applied.

Nagging Reviewer @fredbertsch: You have been added as a reviewer to this pull request. Please add your review or reassign. It has been 244 days with no activity and the awaiting review label has been applied.

Nagging Reviewer @fredbertsch: You have been added as a reviewer to this pull request. Please add your review or reassign. It has been 259 days with no activity and the awaiting review label has been applied.

Nagging Reviewer @fredbertsch: You have been added as a reviewer to this pull request. Please add your review or reassign. It has been 274 days with no activity and the awaiting review label has been applied.

Nagging Reviewer @fredbertsch: You have been added as a reviewer to this pull request. Please add your review or reassign. It has been 289 days with no activity and the awaiting review label has been applied.

Nagging Reviewer @fredbertsch: You have been added as a reviewer to this pull request. Please add your review or reassign. It has been 304 days with no activity and the awaiting review label has been applied.

@yongtang Thanks for your contribution and apologies for the delay in reverting. Can you please sync with latest TensorFlow:Master? @fredbertsch Can you please take a look? Thanks!


[Fix] Closing error in MXNet when indexing results in an empty array

Description
This is to fix the resource closing issue in MXNet engine:
MXNetError: Check failed: delay_alloc:
which happened in the following code
        try (NDManager manager = NDManager.newBaseManager()) {
            NDArray dumb1 = manager.create(new Shape(2, 2)).get(":4, 3:4");
            NDArray dumb2 = manager.create(new Shape(2)).get("3:3");
            NDArray dumb3 = manager.create(new Shape(2)).get("2:3");
        }

The fix is to check the indices within java.

Codecov Report

Merging #1966 (54cdfbd) into master (bb5073f) will decrease coverage by 2.00%.
The diff coverage is 67.97%.

@@             Coverage Diff              @@
##             master    #1966      +/-   ##
============================================
- Coverage     72.08%   70.08%   -2.01%
- Complexity     5126     5824     +698
============================================
  Files           473      574     +101
  Lines         21970    25833    +3863
  Branches       2351     2783     +432
============================================
+ Hits          15838    18104    +2266
- Misses         4925     6361    +1436
- Partials       1207     1368     +161

Impacted Files
Coverage Δ

api/src/main/java/ai/djl/modality/cv/Image.java
69.23% <ø> (-4.11%)
:arrow_down:

...rc/main/java/ai/djl/modality/cv/MultiBoxPrior.java
76.00% <ø> (ø)

...rc/main/java/ai/djl/modality/cv/output/Joints.java
71.42% <ø> (ø)

.../main/java/ai/djl/modality/cv/output/Landmark.java
100.00% <ø> (ø)

...main/java/ai/djl/modality/cv/output/Rectangle.java
72.41% <0.00%> (ø)

...i/djl/modality/cv/translator/BigGANTranslator.java
21.42% <0.00%> (-5.24%)
:arrow_down:

...odality/cv/translator/BigGANTranslatorFactory.java
33.33% <0.00%> (+8.33%)
:arrow_up:

...nslator/InstanceSegmentationTranslatorFactory.java
14.28% <0.00%> (-3.90%)
:arrow_down:

.../cv/translator/SemanticSegmentationTranslator.java
0.00% <0.00%> (ø)

.../cv/translator/StyleTransferTranslatorFactory.java
40.00% <ø> (ø)

... and 476 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


chore(icefiredb-pubsub): update pubsubStore.go

recieved -> received

recieved -> received

Thank you so much.


Marketplace Owner - User Creation

Currently, the Create Admin/MPO User function and Edit Admin/MPO User function are just vaporware, and the Create and Save actions don't actually create/edit an Admin/MPO User.

As an Admin User, when I click Create Admin/MPO User from the Admin/MPO User list page, allow the Admin user to fill out the existing form (Admin/MPO User Username, First Name, Last Name, Phone, Email, and active boolean)
As an Admin User, when I click Save, my Admin/MPO User should be created, and I should now see my Admin/MPO User on the Admin/MPO User list page.
As an Admin User, when I click Discard Changes, an Admin/MPO User should not be created and I should return to the Admin/MPO User list page
As an Admin User, when I click on an existing Admin/MPO User to edit it, allow the Admin user to make changes to the existing form (Admin/MPO User Username, First Name, Last Name, Phone, Email, and active boolean)
As an Admin User, when I click Save, my Admin/MPO User should be updated with the changes I made. Remain on the Admin/MPO User detail page after saving.
As an Admin User, when I click Discard Changes, an Admin/MPO User should not be updated and I should return to the Buyer User list page

Note:  The Admin user creating other Admin Users must have proper permissions to see and create other Admin users for their org.

This has been merged into development and is complete


Fail to read NSO binary

I'm trying to work with a Switch binary here but I keep getting the error
Unhandled exception. System.Exception: Found more than 1 pointer as pCodegenModules
I'm unsure what to do here or if there is nothing else I can do.
I am using a build from the development branch, but this issue also occurs with release 2022.1.0#8

[Info] [Program] Running on Win32NT
[Info] [Plugins] Loading plugins from G:\JDUnity\Cpp2IL\Cpp2IL\bin\Debug\net6.0\win-x64\Plugins...
[Info] [Plugins] Using Plugin: Cpp2IL Built-In
[Warn] [Program] Using force options, I sure hope you know what you're doing!
[Info] [Library] Initializing Metadata...
[Info] [Library]        Using actual IL2CPP Metadata version 29
[Info] [Library] Initialized Metadata in 192ms
[Info] [Library] Using binary type Nintendo Switch Object (from LibCpp2IL)
[Info] [Library] Searching Binary for Required Data...
[Info] [Library]        Decompressing NSO file...
Unhandled exception. System.Exception: Found more than 1 pointer as pCodegenModules
   at LibCpp2IL.BinarySearcher.FindCodeRegistrationPost2019() in G:\JDUnity\Cpp2IL\LibCpp2IL\BinarySearcher.cs:line 196
   at LibCpp2IL.Il2CppBinary.FindCodeAndMetadataReg(Int32 methodCount, Int32 typeDefinitionsCount) in G:\JDUnity\Cpp2IL\LibCpp2IL\Il2CppBinary.cs:line 448
   at LibCpp2IL.LibCpp2IlBinaryRegistry.CreateAndInit(Byte[] buffer, Il2CppMetadata metadata) in G:\JDUnity\Cpp2IL\LibCpp2IL\LibCpp2IlBinaryRegistry.cs:line 68
   at LibCpp2IL.LibCpp2IlMain.Initialize(Byte[] binaryBytes, Byte[] metadataBytes, UnityVersion unityVersion) in G:\JDUnity\Cpp2IL\LibCpp2IL\LibCpp2IlMain.cs:line 159
   at LibCpp2IL.LibCpp2IlMain.LoadFromFile(String pePath, String metadataPath, UnityVersion unityVersion) in G:\JDUnity\Cpp2IL\LibCpp2IL\LibCpp2IlMain.cs:line 205
   at Cpp2IL.Core.Cpp2IlApi.InitializeLibCpp2Il(String assemblyPath, String metadataPath, UnityVersion unityVersion, Boolean allowUserToInputAddresses) in G:\JDUnity\Cpp2IL\Cpp2IL.Core\Cpp2IlApi.cs:line 60
   at Cpp2IL.Program.MainWithArgs(Cpp2IlRuntimeArgs runtimeArgs) in G:\JDUnity\Cpp2IL\Cpp2IL\Program.cs:line 415
   at Cpp2IL.Program.Main(String[] args) in G:\JDUnity\Cpp2IL\Cpp2IL\Program.cs:line 361```

Could you upload a log file also using the --verbose option?

logfile.log

Hmm that looks like quite a painful binary, I'm afraid I don't have any immediate suggestions without having my hands on the binary. You could join my discord (linked in the readme) and we can discuss more closely there, but my free time is often rather limited these days so I don't know how immediately I'd be able to help

I have similar issue finding more than one pointer to pCodegenModules
I've managed to get a working dll with this other project
https://github.com/Perfare/Il2CppDumper
The binary I struggled with was the v1.1.0 of title ID: 0100B0100E26C000, build ID: D6A683602B518969


clipboard  is not used in android 4.2.2.

Minimal example

Fork this JSFiddle and reproduce your issue.

I used clipboard.js in my mobile project. It works well in the operating system of android 4.4.4, but it fail in android 4.2.2.
the difference of android 4.2.2 is that when I clicked the btn, I saw the target object's background change to blue and then the blue disappeared. When I wanted to pasted it, I failed.

Please, provide the browser version too.

I got a user-agent from the browser:
User-Agent: Mozilla/5.0 (Linux; U; Android 4.2.2; en-us; HUAWEI G750-T01 Build/HuaweiG750-T01) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30

Sorry, I wasn't able to reproduce this problem. But feel free to send a pull request. Thanks for reporting @chenEdgar!


Initial Doc and Buy Tikcet flow

This is moreover the flow discused friday abot how to make the vuy ticket flow in a smart way.
The layout result is not good, i think we can improve it a lot, by spliting it into more services or something.
Note:
Sorry for make a pr for two things

Documentation can now be visualized.
Also a template example decision is now there


Reference-based IPs

Reference-based IPs
The goal of this recommendation is to add the capability to build IPs that can transport metadata about content files without having to include the files themselves.
Essentially, the IP will only carry pointers or references to the actual files. This approach has been referred to as "Shallow IPs".
In theory, this functionality is already possible without altering the current implementation, but it would be beneficial to include a section in the CSIP detailing how to achieve this. This recommendation affects all information packages (SIP, AIP and DIP).
Use cases
One of the primary use cases supporting this recommendation concerns I/O logistics and performance. Pre-ingest is responsible for organising data into SIPs before submitting them to the archive. The traditional approach involves copying the data into a physically bound E-ARK SIP, i.e.. a package that includes all the referenced files within its physical boundaries.
This SIP is then submitted to the archive, which generates E-ARK AIPs. After verifying that the ingest was successful, the system must then remove the E-ARK SIP and the original data. Additionally, the system must handle ingest failures and the potential need to regenerate SIPs. This process can lead to several issues, most notably the requirement for significant storage space, which could reach up to three times the original size of the data if we consider the uningested data, the SIP, and the AIP copies.
Moreover, the I/O operations needed to move the data around can place unnecessary stress on the archival system and storage infrastructure.
Instead of duplicating the data within the archival system, Shallow IPs allow for the efficient referencing of data stored in external systems, thereby reducing the need for extensive storage and minimizing the load on the repository’s I/O operations.
This approach ensures that data can be efficiently managed and accessed without unnecessary duplication, improving performance and reducing resource consumption. However, it also increases the risk of data loss, as the repository lacks full control over the data. In certain scenarios, though, this risk may be acceptable in exchange for reducing the use of expensive storage, e.g., when the pre-ingest area and the archival storage area share the same infrastructure.
Given that the OAIS repository is able to retrieve the file whenever it needs for any access or preservation operations the capacity of referring external content can greatly reduce the amount of local resources needed to manage a OAIS system or can allow modern storage systems to be used as a backend for storing E-ARK AIPs. It can also reduce the overall amount of storage needed for a institution when the content is both in the current information management system (e.g., ERMS) and also in the OAIS archive.
For example, a TV broadcast station may have its own archive, which demands significant storage space. While backups and remote replicas are already in place, they may wish to incorporate this data into an OAIS archive to benefit from enhanced preservation features (e.g. Representation Information). However, duplicating the storage or transferring the content into the OAIS archive, and modifying the production system to rely on the OAIS archive, would not be feasible due to the prohibitive costs of such strategies.
The option would be for the OAIS archive to refer to the content in the productive system, allow curators to create shallow SIPs and submit them to the OAIS archive. The OAIS archive would need to access the external content and execute ingest workflow validations.
Preservation actions like fixity checks, file format identification, file format validation, and file format conversion could be done using the external data as input as well. However, every outcome of these operations shall become a local files (including the outcome of the file format conversion actions). We recommend keeping all (descriptive, preservation, other) metadata local to the AIPs, but allow representation data to be external/remote.
CSIP principles 1.4 and 3.6 state that:

"the Information Package SHOULD be scalable" and that
"[...] it is clear that any given technical implementation will become obsolete in time, for example, as new transfer methods and storage solutions emerge. As such this specification does not prohibit the take-up of any emerging logical or physical technical solutions."

Taking these principles into account, we suggest a modification to the CSIP text (which would apply to SIP, AIP, and DIP) to allow representation data to be non-local.
We propose altering the following sections in the Common IP specification:
Section 4. CSIP structure
Update text from:
The preferred implementation of the logical model described in Principle 3.6 is a strict physical (folder) structure that precisely follows the logical structure. While the specification does not prohibit alternative implementations of the conceptual model, the practice is not recommended.
The main reason for this implementation decision is that a fixed and documented folder structure makes the package layout clear to both human users and automated tools. The main benefit this clarity is that many archival tasks (e.g. file format risk analysis), can be executed directly on the data portion of the package structure, as opposed to first processing potentially large amounts of metadata for file locations. This allows for more efficient processing which is valuable in the case of large collections and bulk operations. A fixed folder structure, therefore, provides efficiency and scalability.
Many data storage solutions do not explicitly support folder structures, but use other means for structuring and storing AIP data and metadata. However, the purpose of this specification is to facilitate and support Information Package interoperability. When storage solutions do not support the implementation of the package structure for native AIP storage, it is still possible to implement the physical structure for SIPs and DIPs. While systems need to implement transformations between Common Specification IPs and internal AIPs it allows interoperability between tools that support the common specification, easy transfer of IPs to new repository systems or storage solutions and the establishment of multi-repository duplicated storage solutions.
To:
The preferred implementation of the logical model described in Principle 3.6 is a strict physical (folder) structure that precisely follows the logical structure. While the specification does not prohibit alternative implementations of the conceptual model, the practice is not recommended. However, it is recognised that reference-based packages, where files can be physically separated and referred to by links, are also viable.
The main reason for preferring a fixed and documented folder structure is that it makes the package layout clear to both human users and automated tools. This clarity is crucial as it allows many archival tasks (e.g., file format risk analysis) to be executed directly on the data portion of the package structure, without the need to first process potentially large amounts of metadata to locate files. This leads to more efficient processing, which is valuable in the case of large collections and bulk operations. A fixed folder structure, therefore, provides efficiency and scalability.
While many data storage solutions do not explicitly support folder structures and may use alternative methods, such as Content Addressable Storage systems (CAS) that allow files to be stored separately and retrieved by a unique identifier (and not by its name and location), the purpose of this specification remains to facilitate and support Information Package interoperability. Even when storage solutions do not support the implementation of the folder structures for native AIP storage, it is still possible to implement the physical structure for SIPs and DIPs mainly for transport reasons.
Systems that use reference-based packages should ensure that they implement the necessary processes to consolidate referenced files into a local physically structure, enabling interoperability between tools that support the common specification. This also allows for the easy transfer of IPs to new repository systems or storage solutions and the establishment of multi-repository duplicated storage solutions.
Section 4.3 Implementing referenced-based information packages (NEW SECTION)
In certain implementation contexts, it is advantageous for the files included in an information package - whether it be an SIP, AIP, or DIP - to be referenced via links rather than physically bound together in a folder structure.
One of the primary use cases supporting this recommendation concerns I/O logistics and performance. Pre-ingest is responsible for organising data into SIPs before submitting them to the archive. The traditional approach involves copying the data into a physically bound E-ARK SIP, i.e.. a package that includes all the referenced files within its physical boundaries.
This SIP is then submitted to the archive, which generates E-ARK AIPs. After verifying that the ingest was successful, the system must then remove the E-ARK SIP and the original data. Additionally, the system must handle ingest failures and the potential need to regenerate SIPs. This process can lead to several issues, most notably the requirement for significant storage space, which could reach up to three times the original size of the data if we consider the uningested data, the SIP, and the AIP copies.
Moreover, the I/O operations needed to move the data around can place unnecessary stress on the archival system and storage infrastructure.
Referenced-based IPs, also known as Shallow IPs, offer a solution by allowing for the efficient referencing of data stored in external systems, thereby reducing the need for extensive storage and minimising the strain on the repository’s I/O operations. This approach enables data to be managed and accessed without unnecessary duplication, leading to improved performance and reduced resource consumption. However, it is important to note that this approach carries an increased risk of data loss, as the repository does not maintain full control over the physical storage of the data and also reduced performance when data needs to be fetched from a low bandwidth communication channel. Despite this risk, in certain scenarios, the trade-off may be justified in order to reduce the use of costly resources.
Referenced-based IPs can significantly optimise file storage usage, particularly in scenarios where large volumes of data remain actively used by other systems that function as both processing and archival platforms.
For instance, in environments where data is stored in Content Addressable Storage (CAS) or cloud-based systems like Amazon S3 and OpenStack Swift, the advantages of Referenced-based IPs are evident. These systems inherently manage data through references or identifiers, making Referenced-based IPs an ideal solution for efficient storage and retrieval without the need for duplicating data before and after archiving.
Examples
Typically, the METS file included in an Information Package contains relative URLs that reference files included within the package itself.
Below is an example of an IP that references files available locally within the IP.
<file
    ID="uuid-DD985B25-9328-4F4E-B387-A6F20CA44500" MIMETYPE="application/xml"
    SIZE="3180"
    CREATED="2020-12-22T10:28:45.668Z"
    CHECKSUM="F1F5BB6003165CDD8F6C1FCC32F8FD1F965E1681010F3B9806D9460BCFFA8A3C"
    CHECKSUMTYPE="SHA-256">
    <FLocat
        LOCTYPE="URL"
        xlink:type="simple"
        xlink:href="schemas/xlink.xsd"/>
</file>

However, the METS specification supports several options for LOCTYPE, and the URL option supports the full expressiveness of the URL standard. To create a reference-based information package one should make use of this expressiveness and specify the full location of external files.
The following excerpt is an example of an IP that references files available remotely instead of stored in a local folder structure:
<file
    ID="uuid-DD985B25-9328-4F4E-B387-A6F20CA44500"
    MIMETYPE="application/xml" SIZE="3180"
    CREATED="2020-12-22T10:28:45.668Z"
    CHECKSUM="F1F5BB6003165CDD8F6C1FCC32F8FD1F965E1681010F3B9806D9460BCFFA8A3C"
    CHECKSUMTYPE="SHA-256">
    <FLocat
        LOCTYPE="URL"
        xlink:type="simple"
        xlink:href="https://www.w3.org/XML/2008/06/xlink.xsd"/>
</file>

Other protocols may also be used. The following example, depicts how files may be located on a shared drive, accessible via the NFS protocol:
<file
    ID="uuid-DD985B25-9328-4F4E-B387-A6F20CA44500"
    MIMETYPE="application/xml" SIZE="3180"
    CREATED="2020-12-22T10:28:45.668Z"
    CHECKSUM="F1F5BB6003165CDD8F6C1FCC32F8FD1F965E1681010F3B9806D9460BCFFA8A3C"
    CHECKSUMTYPE="SHA-256">
    <FLocat
        LOCTYPE="URL"
        xlink:type="simple"
        xlink:href="nfs://shared-host/schemas/xlink.xsd"/>
</file>

It is helpful to discuss this explicitly. The TV Broadcaster provides a good use case.
“referenced via links rather than physically bound together in a folder structure” - from the METS example, everything is referenced using a link (FLcat) – both files included in the IP as well as ones that are not. The difference is that the included files have a relative URL. Perhaps there is another way to make the difference clear.
The METS examples would be clearer and easier to understand if they referred to a real content file rather than an XML schema file. The differences would be clearer if they were highlighted or even if the text just showed the new href. As it is, the reader has to parse it carefully and confirming that the ID and CHECKSUMs are the same is not readily done by eye!
“place unnecessary stress on the archival system and storage infrastructure” – What does this mean? The text already mentions the cost of multiple copies of the data.
I think that the text may underplay the risks that can come from referencing the content. Take the TV Broadcaster, for example. Imagine that disk space is tight and a big event risks filling up available storage. It is easy to make an operational decision to delete files that seem unlikely to be used. Or a recording is embarrassing to a presenter and they put pressure on operations staff to delete it. Or the operational version is directly edited. It is also harder to ensure that reference-based copies are physically distinct.
Using remote/cloud-based storage that is part of the archival solution does not introduce the risks that you might see when using copies in an operational system.
“the package layout clear to both human users and automated tools” – as an aside, I guess that this is more about being a lowest-common-denominator technology (file-system with files and folders) and that easily using tools designed for another context.
A final editing pass should catch the minor errors in the very clear writing.

I believe the proposal is reasonable. I am familiar with requirements of this kind related to archive systems that use an S3 storage backend. However, regarding the AIP, I would suggest adding a subordinate requirement that it must always be possible to generate an all-inclusive AIP, which can be divided or split for very large AIPs. The reasoning behind this is that when using external storage for AIPs, it is then still possible to safeguard all-inclusive AIPs on more affordable long-term storage options, such as tape drives, for example.

Revised text taking into account Adam and Sven's comments. @karinbredenberg I believe the text is good enough to initiate the voting process.
Implementing Reference-Based Information Packages (IPs)
The goal of this recommendation is to add the capability to build IPs that can transport metadata about content files without having to include the content files themselves.
Essentially, the IP will only carry pointers or references to the actual content files. This approach has been previously referred to as "Shallow IPs".
In theory, this functionality is already possible without altering the current implementation, however it would be beneficial to include a section in the CSIP detailing how to achieve this. This recommendation affects all information packages (SIP, AIP and DIP).
Use cases
One of the primary use cases supporting this recommendation concerns I/O performance and storage space. Pre-ingest is responsible for process of organising data into SIPs before submitting them to the OAIS. The traditional approach involves copying data into a physically bound E-ARK SIP, i.e., a package that includes all the metadata and content files within its physical boundaries.
The SIP is then submitted to the OAIS, which transforms it into proper AIPs. Typically, after verifying that the ingest was successful, the system eliminates the original SIP and all its included data and metadata.
The OAIS must be able to handle ingest failures and the potential need to reingest SIPs. This process can lead to several issues, most notably the requirement for significant storage space, which could reach up to three times the original size of the data if we consider the uningested data, the SIP, and the AIP copies.
Moreover, I/O operations needed to move the data from its original location to the temporary ingest storage can place unnecessary stress on the archival storage infrastructure.
Rather than duplicating data within the archival system, Shallow IPs allow for the efficient referencing of data stored in external systems, thereby reducing the need for extensive storage and minimizing I/O operations.
This approach ensures that data can be efficiently managed and accessed without unnecessary duplication, improving performance and reducing resource consumption. However, it increases the risk of data loss, as the repository lacks full control over the data. In certain scenarios, though, this risk may be acceptable in exchange for reducing the use of large amounts of expensive storage, e.g., when the pre-ingest area and the archival storage area share the same infrastructure.
As long as the OAIS is able to retrieve the file whenever it needs for any access or preservation operations the capacity of referring external content can greatly reduce the amount of local resources needed to manage a OAIS system or can allow modern storage systems to be used as a backend for storing E-ARK AIPs. It can also reduce the overall amount of storage needed for a institution when the content is both in the current information management system (e.g., on Electronic Records Management Systems) and also in the OAIS.
For example, a TV broadcast station may have its own archival storage, which already demands significant storage space. While backups and remote replicas are already in place, they may wish to incorporate this data into an OAIS to benefit from enhanced preservation features (e.g. Representation Information and Preservation Planning). Duplicating the storage or transferring the content into the OAIS, and modifying the production system to rely on the OAIS archival storage would not be feasible due to the prohibitive costs of such a strategy.
Please note that this approach does not come without risks. Taking the TV broadcaster as an example, one can imagine that if disk space is running tight, an operational decision may be made to delete files that seem unlikely to be used without the proper concern about the preservation policy in place by the OAIS. An important characteristic of this strategy is immutability, which means that the content should not change once it has been ingested into the OAIS. The organisation should employ safeguards to ensure the integrity of the content stored in external environments.
The option would be for the OAIS to maintain references to the files managed by the productive system. This can be accomplished by ingesting Reference-based SIPs that contain pointers to the content files instead of the files them selves. In this scenario, the OAIS would not have full control over the content files. Instead it would have to request those files from an external archival storage given a well know access protocol.
Preservation actions like fixity checks, file format identification, file format validation, and file format conversion would have be done using the external data as input. However, every outcome of these operations shall become local files (including the outcome of the file format conversion actions). Therefore, we recommend keeping all metadata local to the AIPs within the OAIS (descriptive, preservation, other), but allow representation data to be externally managed by another archival storage.
The Common Specification for Information Packages (CSIP) principles 1.4 and 3.6 state that:

"the Information Package SHOULD be scalable" and that
"[...] it is clear that any given technical implementation will become obsolete in time, for example, as new transfer methods and storage solutions emerge. As such this specification does not prohibit the take-up of any emerging logical or physical technical solutions."

Taking these principles into account, we suggest a modification of the CSIP (which would apply to SIP, AIP, and DIP) to allow representation data to be non-local.
To attain this goal, we propose altering the following sections in the CSIP:
Section 4. CSIP structure
Update text from:
The preferred implementation of the logical model described in Principle 3.6 is a strict physical (folder) structure that precisely follows the logical structure. While the specification does not prohibit alternative implementations of the conceptual model, the practice is not recommended.
The main reason for this implementation decision is that a fixed and documented folder structure makes the package layout clear to both human users and automated tools. The main benefit this clarity is that many archival tasks (e.g. file format risk analysis), can be executed directly on the data portion of the package structure, as opposed to first processing potentially large amounts of metadata for file locations. This allows for more efficient processing which is valuable in the case of large collections and bulk operations. A fixed folder structure, therefore, provides efficiency and scalability.
Many data storage solutions do not explicitly support folder structures, but use other means for structuring and storing AIP data and metadata. However, the purpose of this specification is to facilitate and support Information Package interoperability. When storage solutions do not support the implementation of the package structure for native AIP storage, it is still possible to implement the physical structure for SIPs and DIPs. While systems need to implement transformations between Common Specification IPs and internal AIPs it allows interoperability between tools that support the common specification, easy transfer of IPs to new repository systems or storage solutions and the establishment of multi-repository duplicated storage solutions.
To:
The preferred implementation of the logical model described in Principle 3.6 is a strict physical (folder) structure that precisely follows the logical structure. While the specification does not prohibit alternative implementations of the conceptual model, the practice is not recommended. However, it is recognised that Reference-based packages, where files can be physically separated and referred to by links, are also viable.
The main reason for preferring a fixed and documented folder structure is that it makes the package layout clear to both human users and automated tools. This clarity is crucial as it allows many archival tasks (e.g., file format risk analysis) to be executed directly on the data portion of the package structure, without the need to first process potentially large amounts of metadata to locate files. This leads to more efficient processing, which is valuable in the case of large collections and bulk operations. A fixed folder structure, therefore, provides efficiency and scalability.
While many data storage solutions do not explicitly support folder structures and may use alternative methods, such as Content Addressable Storage systems (CAS) that allow files to be stored separately and retrieved by a unique identifier (and not by its name and location), the purpose of this specification remains to facilitate and support Information Package interoperability. Even when storage solutions lack support for the implementation of folder structures for native AIP storage, it is still possible to implement the physical structure for SIPs and DIPs mainly for transport reasons.
Systems that use Reference-based packages MUST ensure that they implement the necessary processes to consolidate referenced files into an all-inclusive local physical structure, enabling interoperability between tools that support the common specification. This approach enables easy transfer of IPs to new repository systems or to more affordable long-term storage options such as tapes or cold data storage systems and the establishment of multi-site duplicated storage solutions.
Section 4.3 Implementing referenced-based information packages (NEW SECTION)
In certain implementation contexts, it is advantageous for the files included in an information package - whether it be an SIP, AIP, or DIP - to be referenced via links rather than physically bound together in a folder structure.
One of the primary use cases supporting this recommendation concerns I/O performance and storage space. Pre-ingest is responsible for the process of organising data into SIPs before submitting them to the archive. The traditional approach involves copying the data into a physically bound E-ARK SIP, i.e.m a package that includes all the metadata and content files within its physical boundaries.
The SIP is then submitted to the archive, which transforms it into proper AIPs. Typically, after verifying that the ingest was successful, the system eliminates the original SIP and all its included data and metadata.
The archive must be able to handle ingest failures and the potential need to reingest SIPs. This process can lead to several issues, most notably the need for a significant amount of storage space, which could reach up to three times the original size of the data if we consider the uningested data, the SIP being transferred, and the AIP copies.
Moreover, I/O operations needed to move the data from its original location to the temporary ingest storage can place unnecessary stress on the archival storage infrastructure.
Rather than duplicating data within the archival system, Shallow IPs allow for the efficient referencing of data stored in external systems, thereby reducing the need for extensive storage space and minimizing I/O operations.
Referenced-based IPs, also known as Shallow IPs, offer a solution by allowing for the referencing of data stored in external systems without the need to copy them into the archival storage of the OAIS. This approach enables data to be managed and accessed without unnecessary duplication, leading to improved performance and reduced resource consumption. However, it is important to note that this approach carries an increased risk of data loss, as the repository does not maintain full control over the physical storage where the data being managed and also reduced performance when data needs to be fetched through a low bandwidth communication channel. Despite these risks, in certain scenarios, the trade-off may be justified in order to reduce the use of costly resources and time.
Referenced-based IPs can significantly optimise file storage usage, particularly in scenarios where large volumes of data remain actively used by other systems that may function both as processing and archival storage platforms.
For instance, in environments where data is stored in Content Addressable Storage (CAS) or cloud-based systems like Amazon S3 and OpenStack Swift, the advantages of referenced-based IPs are evident. These systems inherently manage data through references or identifiers, making referenced-based IPs an ideal solution for efficient storage and retrieval without the need for duplicating data before and after archiving.
Examples
Typically, the METS file included in an Information Package contains relative URLs that reference files included within the package itself. In the following example, the <FLcat> element uses the href attribute to refer a local file by using a relative path to the file within the information package.
<file
    [...]
    <FLocat
        LOCTYPE="URL"
        xlink:type="simple"
        xlink:href="representations/rep1/data/council-meetings/2024-01-05/File1.doc"/>
</file>

The METS specification supports several options for LOCTYPE attribute such as URN, URL, PURL, HANDLE, DOI, and OTHER.. The URL option supports the full expressiveness of the Uniform Resource Locator standard. To create a Reference-based information package one should make use of this expressiveness and specify the full location of the external files.
The following excerpt is an example of an IP that references files available remotely instead of stored in a local folder structure. One can easily spot this difference in the href attribute as it references a file via a a full resource locator composed of scheme, server, port and a full path to the file within the server.
<file
    [...]
    <FLocat
        LOCTYPE="URL"
        xlink:type="simple"
        xlink:href="https://production-system-host:80/council-meetings/2024-01-05/File1.doc"/>
</file>

Other protocols may also be used. The following example, depicts how files may be referenced on shared folder accessible via the NFS protocol:
<file
    [...]
    <FLocat
        LOCTYPE="URL"
        xlink:type="simple"
        xlink:href="nfs://shared-storage-host/council-meetings/2024-01-05/File1.doc"/>
</file>

The suggestion is:

To update CSIP with information about referencing content housed outside of the IP
Add a new chapter with description of how referencing outside the IP is done

Board members acknowledgment of the issue:
Tick the box in front of you name to indicate that you have looked at the suggestion.

[x] Karin Bredenberg (Kommunalförbundet Sydarkivera, chair)
[ ] Anders Bo Nielsen (National Archives of Denmark)
[ ] Anja Paulič (National Archives of Slovenia)
[ ] Arne-Kristian Groven (National Archives of Norway)
[ ] Gregor Zavrsnik (Geoarh)
[ ] Janet Anderson (Highbury Research & Development Ltd.)
[ ] Maya Bangerter (Swiss Federal Archives)
[ ] Miguel Ferreira (KEEP Solutions)
[ ] Stephen Mackey (Penwern Limited)
[ ] Sven Schlarb (Austrian Institute of Technology)

Voting
(Decision making will be carried out on the basis of majority voting by all eligible members of the Board acknowledging the issue. In the case of a tied vote, decisions will be made at the discretion of the Chair)
Tick the box in front of you name to say yes to the suggestion.

[ ] Karin Bredenberg (Kommunalförbundet Sydarkivera, chair)
[ ] Anders Bo Nielsen (National Archives of Denmark)
[ ] Anja Paulič (National Archives of Slovenia)
[ ] Arne-Kristian Groven (National Archives of Norway)
[ ] Gregor Zavrsnik (Geoarh)
[ ] Janet Anderson (Highbury Research & Development Ltd.)
[ ] Maya Bangerter (Swiss Federal Archives)
[ ] Miguel Ferreira (KEEP Solutions)
[ ] Stephen Mackey (Penwern Limited)
[ ] Sven Schlarb (Austrian Institute of Technology)

I was not involved in the consortium when the CSIP was originlly drafted, but my interpretation of the specification as a user is that it has a preference for physical IPs (i.e. ones that contain content data and preservation description information) but in deference to software designs that do not manage data wihin the OAIS as physical folder structures it allows for this implementation as long as physical SIPs can be ingested and physical DIPs exported. An argument can be made that the CSIP defines a folder structure but does not mandate that this actually contains the content data, however if so the following would not be stated:
The main reason for this implementation decision is that a fixed and documented folder structure makes
the package layout clear to both human users and automated tools. The main benefit this clarity is that
many archival tasks (e.g. file format risk analysis), can be executed directly on the data portion of the package
structure, as opposed to first processing potentially large amounts of metadata for file locations. This allows
for more efficient processing which is valuable in the case of large collections and bulk operations. A fixed
folder structure, therefore, provides efficiency and scalability.
Specifically:

I see this proposal as changing a fundemental principle of the CSIP rather than slightly adjusting it and it requires more debate. Such a change if carried, requires significant re-writes of the CSIP (and probably other specifications), otherwise the CSIP and other specifications will look (in my view) contradictory.

I disagree to the term 'Shallow IP', not a common term for me, this would be better called a 'Virtual IP' as it is not a physical package.

The section 4.3 on implementation is appropriate for a guideline, not the specification.

This change is proposed for the CSIP and so should apply to all packages: SIP, AIP, DIP and consequently proposes virtual SIPs and DIPs as well as AIPs. So, the OAIS sends or receives  packages that contain no data and references links to external storage systems. This is not a model for an OIAS that I recognise.

Proposal:
The reference model for an OAIS is well defined and understood and contains all of the necessary functions, data and metadata. The OAIS accepts SIPs containing data and distributes DIPs,containing data. AIPs are held in the OAIS. The boundaries of the OAIS are not restricted, so a system can be envisaged where different functions are performed in different systems and even different locations but are governed by the management functions of the OAIS. Any 'external systems' used must thus be considered part of the OAIS and any data exchange with truly external systems must be via submission and dissemination packages. I think that any guideline notes on virtual AIPs should be in the AIP specification or guideline alongside the guidance for splitting AIPs but should not be applicable to the SIP and DIP. CSIP should be unchanged but if the philosophy is changing a more fundemental rewrite is necessary.


Fix input/output error

hcitool lescan creates an error if the bluez is already running. The solution I found was
hciconfig hci0 down
hciconfig hci0 up

I signed it!

Thanks @PrabhanshuAttri for this. I'm not that familiar with this aspect of hciconfig. Does this mean that your fix resets the BLE connection? How often does this happen? Just once on startup or each time the command is invoked?

@PrabhanshuAttri: this can be caused by the Bluetooth daemon running. Can you first try to stop the daemon with "sudo systemctl stop bluetooth" or "sudo /etc/init.d/bluetooth stop" to see if that resolves the issue?

@scottjenson every time we do a 'lescan' and if hci0 was 'up' then it throws this error. @perja12 I think this would be a better fix for this. If the hci0  was already down then running these commands won't create any issue. It's just a minor fix.

Bringing hci0 down and and back up will loose other bluetooth connections that might already being used.

I think we are already putting it on non-connectable mode while advertising the link. Resetting hci0 was the only solution I found. Do we need to advertise and scanning URL, simultaneously?

I'm actually ok with this solution. I just wanted a bit of review to make sure there wasn't something I was missing. Unless anyone else has strong feelings, I'd like to approve this tomorrow. Speak up (with solutions) or forever hold your peace ;-)

I guess I am missing the use case. Is this on a dedicated embedded Linux just being used as a  beacon. Or is it a full blown Linux workstation / laptop where other bluetooth devices may be connected as in keyboads, mice, receiving data from other beacons, etc.
If the former, then who cares. If the latter, then something else might hiccup during the restart.
If this is just a script for experimenting with, I don't have a problem either while one figures out why the lescan seems to be misbehaving. It should just work.

@tolson2000 Yes, unfortunately, that will happen, if we reset it. This fix stops the advertising-url. I think it's a low-cost solution to test and use Eddystone beacons. For production uses, the user must buy the real beacons.

@scottjenson what do you think about resetting it?


fix: convert type/format rule to spectral style

This PR adds these new spectral rules:

array-of-arrays
valid-type-format
and removes the old style corresponding rules.

:tada: This PR is included in version 0.2.1 :tada:
The release is available on npm package (@latest dist-tag)
Your semantic-release bot :package::rocket:

:tada: This PR is included in version 0.54.1 :tada:
The release is available on npm package (@latest dist-tag)
Your semantic-release bot :package::rocket: