New Alpaca 13b LoRA, newest more precise training.

Hi @tloen ,
I know there are several 13b lora linked already.
They seem of varying quality however (some Rank 4, some old training style, only 2 modules).
Here is my newly trained one with its training settings:
https://huggingface.co/Angainor/alpaca-lora-13b
Rank 16, 10 epochs, 4 modules.

On generation with oogabooga, I get the following error:
RuntimeError: probability tensor contains either `inf`, `nan` or element < 0

Not sure why. I've used several other 13b lora w/o problems in 8-bit.

Thanks for the notice.
I did not try with oogabooga.
The GPUs are busy right now, but I'll redo some tests from the HF repo once the job is over and report.

Thanks for sharing @AngainorDev. Was looking at the HF model repo and saw the graphs, curious what exactly was the loss?  My best loss training has been 0.78, can't seem to break that lower with the cleaned dataset, but will try now with your hyperparams

curious what exactly was the loss?

Default one.
Things with an incidence could be:

No validation (For this precise case, Alpaca dataset, aiming at diversity I find it better not to let samples apart and ignore validation, since the goal is to learn different enough stuff for it generalize). I also save at every epoch - not by steps - So I'm able to pick an earlier checkpoint if needed.
Hard coded warmup steps (could be auto-computed) to 404 so it fitted exactly the first epoch (with my own batch params)
No "Group by len" (faster but non random samples)
Ofc , with more LoRA params: 10 epochs, 3 are not enough.

Thanks for the notice. I did not try with oogabooga.
The GPUs are busy right now, but I'll redo some tests from the HF repo once the job is over and report.
Edit: Just checked, works from this repo generate (and HF hosted lora) I also installed the current oogabooga and it works as well.

Thanks. Don't know what was wrong, but it works now. Must have just needed a reboot of my PC. Sorry for the confusion.

Added this to the README. I was a little hesitant to include r and the modules at first because I didn't want to endorse downloads I haven't tested, but it'll probably be useful going forward to distinguish between "rough" and "refined" LoRAs. Might write a script to collect the information automatically. Of note: I think @deep-diver also updated his weights to r=16 and q/k/v/o.

Yes just did today at https://huggingface.co/chansung/alpaca-lora-13b and will upload 30b version at https://huggingface.co/chansung/alpaca-lora-30b as well tomorrow. I simply used the same script as
python finetune.py \ --base_model='decapoda-research/llama-7b-hf' \ --num_epochs=10 \ --cutoff_len=512 \ --group_by_length \ --output_dir='./lora-alpaca' \ --lora_target_modules='[q_proj,k_proj,v_proj,o_proj]' \ --lora_r=16 \ --micro_batch_size=8

If you are curious, here is the playground with 13B model (ignore the wrong contents in the markdown saying 30B)
https://notebooksf.jarvislabs.ai/BuOu_VbEuUHb09VEVHhfnFq4-PMhBRVCcfHBRCOrq7c4O9GI4dIGoidvNf76UsRL/

@AngainorDev  Without eval, how can you tell your model training is not just overshooting everything with too much training? During my testing with high epoch and 250k sized training data, eval became must worse and starts regressing after 2-3 epochs even through train/loss was still at steady level.  In my case, depending on size of training data size, high epochs is not the best option.

@diegomontoya The training code as provided has load_best_checkpoint set to True, which should pick the checkpoint with the lowest eval loss.

@tloen  Yes, but @AngainorDev  provided the refined trained weights with --val_set_size=0 which I assume will disable evaluation and thus my question on who he was able to determine the checkpoint he chose was the best one since no eval was generated? Maybe I am mistaken to what --val_set_size=0  means?

just uploaded fine-tuned 30B version. The final loss is as below
: {'loss': 0.5584, 'learning_rate': 0.0, 'epoch': 9.99}
here is the link
: https://huggingface.co/chansung/alpaca-lora-30b/tree/main
It was trained on A100 40G * 8 DGX workstation for about 32 hours.
if there is a way to mark something like "this version is trained on the cleaned dataset(4/6/2022) with the latest known to be working better technique(lora_r, target_modules, epochs)"

@AngainorDev - Are the evals you're talking about different than the kind mentioned here? https://github.com/tloen/alpaca-lora/issues/147

These are generic evals, I'd want a dedicated alpaca style eval dataset, with instructions similar to (and different from) what alpaca training has.

There is this repo,
I tweaked the test script to work with our loras
https://github.com/manyoso/haltt4llm/pull/6
But this one mainly focuses on hallucinations with a specific multiplce choice template that is not known to Alpaca, so it has a hard time answering "IDK" or "None of the above".
Still a good move, but this only checks for factual knowledge, no other capabilities.


Unknown error: Press "OK" to return to document list.

Hi, someone know how fix it?
I'm using traefik as loadbalancer, here an capture and my docker-compose.yml file:
log
## document server (/var/log/onlyoffice/documentserver/nginx.error.log)
2017/11/11 14:11:56 [crit] 407#0: *108 connect() to [::1]:8080 failed (99: Cannot assign requested address) while connecting to upstream, client: <IP_ADDRESS>, server: , request: "GET /2017-10-28-08-40/spellchecker/doc/95_1510408155/c/info?t=1510409516194 HTTP/1.1", upstream: "http://[::1]:8080/doc/95_1510408155/c/info", host: "demo.nextcloud.com", referrer: "https://demo.nextcloud.com/ds-vpath/2017-10-28-08-40/web-apps/apps/documenteditor/main/index.html?_dc=2017-10-28-08-40&lang=en&customer=ONLYOFFICE&frameEditorId=iframeEditor"

if I try do or change something on the admin area, get this message:
Error when trying to connect (Mixed Active Content is not allowed. HTTPS address for Document Server is required.) Nextcloud Search Basic settings Monitoring Sharing Security Theming Encryption Workflow Activity Usage survey Logging Additional settings Tips & tricks

version: "3"

networks:
  proxy:
    external: true
  internal:
    external: false

volumes:
  document_data:
  document_log:
  app_data:
  nextcloud-db-data:

services:
  app:
    restart: always
    container_name: app-server
    image: nextcloud:fpm
    stdin_open: true
    tty: true
    expose:
      - '80'
      - '9000'
    networks:
      - onlyoffice
    volumes:
      - app_data:/var/www/html
    networks:
      - proxy
      - internal
    labels:
      - traefik.enable=false

  onlyoffice-document-server:
    restart: always
    container_name: onlyoffice-document-server
    image: onlyoffice/documentserver:latest
    stdin_open: true
    tty: true
    networks:
      - onlyoffice
    expose:
      - '80'
      - '443'
    volumes:
      - document_data:/var/www/onlyoffice/Data
      - document_log:/var/log/onlyoffice
    networks:
      - proxy
      - internal
    labels:
      - traefik.docker.network=proxy

  nextcloud-db:
    image: postgres
    volumes:
      - nextcloud-db-data:/var/lib/postgresql/data
    environment:
      - POSTGRES_USER=nextcloud
      - POSTGRES_PASSWORD=nextcloud
      - POSTGRES_DB=nextcloud
      - POSTGRES_HOST=nextcloud-db
    networks:
      - internal
    labels:
      - traefik.enable=false

  nginx:
    restart: always
    container_name: nginx-server
    image: nginx
    stdin_open: true
    tty: true
    expose:
    - '80'
    - '443'
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
      - app_data:/var/www/html
    labels:
      - traefik.backend=demo.nextcloud.com
      - traefik.frontend.rule=Host:demo.nextcloud.com
      - traefik.docker.network=proxy
    networks:
      - internal
      - proxy

this issue was solved here:
https://github.com/ONLYOFFICE/Docker-DocumentServer/issues/80

Hi,
with the same yml file, I have the same error message when I want to change the ONLYOFFICE configuration. For example, if I check the box: open file in the same tab Then, it is not even possible to modify a document.
Without the reverse proxy (Traefik), I can change the configuration without problems.
I modified the nginx.conf file as shown but it does not change anything.

@houstek Could you please create a new issue, please do not forget to include all necessary information about the issue such as logs, os, versions, behavior.
THX


Handle a missing statusPort setting for the injector as statusPort 0

Please provide a description for what this PR is for.
We create our own custom values files to enable specific settings and configure the proxy for our needs. The last time we forgot to set the statusPort (https://github.com/istio/istio/blob/master/install/kubernetes/helm/istio/values.yaml#L229) which resulted in the following error message at replicaSets:
  Type     Reason        Age               From                   Message
  ----     ------        ----              ----                   -------
  Warning  FailedCreate  30s               replicaset-controller  Error creating: Pod "test-v2-dummy-istio-dev-694589b8fc-zjz95" is invalid: spec.containers[1].readinessProbe.httpGet.port: Invalid value: "<nil>": must contain only alpha-numeric characters (a-z, 0-9), and hyphens (-)
  Warning  FailedCreate  30s               replicaset-controller  Error creating: Pod "test-v2-dummy-istio-dev-694589b8fc-66v7l" is invalid: spec.containers[1].readinessProbe.httpGet.port: Invalid value: "<nil>": must contain only alpha-numeric characters (a-z, 0-9), and hyphens (-)
  Warning  FailedCreate  30s               replicaset-controller  Error creating: Pod "test-v2-dummy-istio-dev-694589b8fc-nbnzh" is invalid: spec.containers[1].readinessProbe.httpGet.port: Invalid value: "<nil>": must contain only alpha-numeric characters (a-z, 0-9), and hyphens (-)
  Warning  FailedCreate  30s               replicaset-controller  Error creating: Pod "test-v2-dummy-istio-dev-694589b8fc-brztf" is invalid: spec.containers[1].readinessProbe.httpGet.port: Invalid value: "<nil>": must contain only alpha-numeric characters (a-z, 0-9), and hyphens (-)
  Warning  FailedCreate  30s               replicaset-controller  Error creating: Pod "test-v2-dummy-istio-dev-694589b8fc-x94gg" is invalid: spec.containers[1].readinessProbe.httpGet.port: Invalid value: "<nil>": must contain only alpha-numeric characters (a-z, 0-9), and hyphens (-)
  Warning  FailedCreate  30s               replicaset-controller  Error creating: Pod "test-v2-dummy-istio-dev-694589b8fc-kqc5b" is invalid: spec.containers[1].readinessProbe.httpGet.port: Invalid value: "<nil>": must contain only alpha-numeric characters (a-z, 0-9), and hyphens (-)
  Warning  FailedCreate  30s               replicaset-controller  Error creating: Pod "test-v2-dummy-istio-dev-694589b8fc-w5pzv" is invalid: spec.containers[1].readinessProbe.httpGet.port: Invalid value: "<nil>": must contain only alpha-numeric characters (a-z, 0-9), and hyphens (-)
  Warning  FailedCreate  29s               replicaset-controller  Error creating: Pod "test-v2-dummy-istio-dev-694589b8fc-sp7xp" is invalid: spec.containers[1].readinessProbe.httpGet.port: Invalid value: "<nil>": must contain only alpha-numeric characters (a-z, 0-9), and hyphens (-)
  Warning  FailedCreate  29s               replicaset-controller  Error creating: Pod "test-v2-dummy-istio-dev-694589b8fc-8ct25" is invalid: spec.containers[1].readinessProbe.httpGet.port: Invalid value: "<nil>": must contain only alpha-numeric characters (a-z, 0-9), and hyphens (-)
  Warning  FailedCreate  9s (x4 over 27s)  replicaset-controller  (combined from similar events): Error creating: Pod "test-v2-dummy-istio-dev-694589b8fc-c2tdj" is invalid: spec.containers[1].readinessProbe.httpGet.port: Invalid value: "<nil>": must contain only alpha-numeric characters (a-z, 0-9), and hyphens (-)

The error is not really helpful, since our test application has no readinessProbe we knew that the error must be in the injector template. I would prefer that a missing statusPort setting is handle as no statusPort (e.g. 0).
And to help us figure out who should review this PR, please
put an X in all the areas that this PR affects.
[X] Configuration Infrastructure
[ ] Docs
[X] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

@johscheuer: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

release-test_istio
7f1a85ca118d17e0adc84488a636778db12b10d0
link
/test release-test_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

@johscheuer: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

integ-security-k8s-tests_istio
7f1a85ca118d17e0adc84488a636778db12b10d0
link
/test integ-security-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest


nix run nixpkgs#kmail fails to launch -- The Email program encountered a fatal error and will terminate now. The error was: Unable to obtain agent type ''.

Describe the bug
Testing out kmail - cannot do so because kmail does not appear to pull in necessary runtime dependencies.
Steps To Reproduce
Steps to reproduce the behavior:

Set up NixOS, do not add anything related to akonadi to your system.environmentPackages.
run nix run nixpkgs#kmail
Observe this fatal error:

The Email program encountered a fatal error and will terminate now.
The error was:
Unable to obtain agent type ''.

Expected behavior
kmail pulls in necessary dependencies and launches gracefully.
Screenshots

Additional context
See:

#102167 (same error message, seems unrelated)
https://discourse.nixos.org/t/kde-on-nixos-issues/14871/7 (More perspectives on this bug)

Notify maintainers

@ttuegel @vandenoever @nyanloutre
Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
winston@icarus ~ $ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.55, NixOS, 23.05 (Stoat), 23.05.20230927.e3f57c8`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.13.5`
 - nixpkgs: `/nix/store/x87x8b1dfvmzwgapjqphkgw27am77b37-source`

This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/kde-on-nixos-issues/14871/12

KMail doesn't crash for me anymore, but I still access my mails (#292450).

Is this still an issue?

@k900 I've just run into this on nixos-unstable, so I think it's still an issue. If I just use the kmail package, it cannot find the account wizard, so I added kmail-account-wizard. Then it wants akonadi which I then added. And then it gets to this error.
The logs are the same for me as in the screenshot above but I'll post them as text:
org.kde.pim.akonadicore: Failed to request resource "" : "Cannot connect to the Akonadi service. (Unable to obtain agent type ''.)"
org.kde.pim.akonadicore: "Unable to obtain agent type ''."
org.kde.pim.akonadicore: Failed SpecialCollectionsRequestJob::slotResult "Cannot connect to the Akonadi service. (Unable to obtain agent type ''.)"
org.kde.pim.mailcommon: "The Email program encountered a fatal error and will terminate now.\nThe error was:\nUnable to obtain agent type ''."

So there's an additional issue with some dependencies that do not seem to be added automatically, which would also be a welcome improvement.
I'd love to help out, but I have no idea where to start. Let me know if there's anything I can do and thanks for all your amazing work on getting plasma to work on nixos :)

nixos-unstable doesn't have the fix yet, and you need to also install kdepim-runtime.

Ah apologies, I'll try it right away. Which ref should I use to test it?

nixos-unstable-small, though nixos-unstable should also update soon.

This is fixed for me and we are going to add a proper option as part of #299152
So we can close this from my point of view.

It is fixed for me as well (sorry forgot to report back).

This problem still occurs in NixOS 23.11
After running nix-shell -p kmail and then kmail a KMail error dialogue states
The Email program encountered a fatal error and will terminate now.
The error was:
Unable to obtain agent type ''.

Screenshot:

$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.82, NixOS, 23.11 (Tapir), 23.11.5648.44733514b72e`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.1`
 - channels(root): `"agenix, home-manager-23.11.tar.gz, nixos-23.11"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

This is not the same issue, and not the same version of KMail. It's very unlikely we'll backport Plasma 6.

The OP neither mentions Plasma 6 nor a specific version of KMail. My error occurs with the current stable version, i.e. 23.11. Therefore, I assume(d) that this is the correct issue.

I'm afraid I need to re-open this:
$ LANG=C nix run nixpkgs#kmail
qt.qpa.wayland: Wayland does not support QWindow::requestActivate()
org.kde.pim.akonadicore: "QLocalSocket: Remote closed" "/run/user/1000/akonadi/akonadiserver-cmd.socket"
org.kde.pim.akonadicore: Socket error occurred: "QLocalSocket: Remote closed"
org.kde.pim.akonadicore: Job error:  "Cannot connect to the Akonadi service." for collection: QVector()
We have an error during reading password  "Entry not found"
org.kde.pim.akonadicore: "QLocalSocket: Remote closed" "/run/user/1000/akonadi/akonadiserver-cmd.socket"
org.kde.pim.akonadicore: Socket error occurred: "QLocalSocket: Remote closed"
org.kde.pim.akonadicore: ""
org.kde.pim.akonadicore: Failed SpecialCollectionsRequestJob::slotResult "Cannot connect to the Akonadi service."
org.kde.pim.mailcommon: "The Email program encountered a fatal error and will terminate now"
We have an error during reading password  "Entry not found"
We have an error during reading password  "Entry not found"

I'm on NixOS 24.05 with plasma6 and just deleted all my ~/.config/.akonadi* and ~/.local/share/akonadi* to switch over to the sqlite driver.
Slightly related: I already have KMail, Kontact, and several PIM packages installed into my home-manager profile or my system packages. When I run kmail from there without the nix run part, it successfully opens.
But when trying to add a new e-mail account I got the error window Could not start the account wizard. Please make sure you have AccountWizard properly installed. until I've explicitly added the kmail-account-wizard package.
See my system config for the packages I've installed.
System details:
$ nix-shell -p nix-info --run "nix-info -m"                                                                                130 ↵
 - system: `"x86_64-linux"`
 - host os: `Linux 6.8.12, NixOS, 24.05 (Uakari), 24.05.20240725.a1cc729`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Lix, like Nix) 2.90.0`
 - channels(root): `"nixos"`
 - nixpkgs: `/nix/store/sj9yrq21wbbfr5715hys3laa2qd6x471-source`

That's what the programs.kde-pim option was added for, that's currently the only recommended way to get the required Akonadi services running.

See https://github.com/NixOS/nixpkgs/pull/330521 for 24.05.

Thanks for the swift backport, I was just about to ask whether I had missed a note or announcement on that.
To get the word out on this, it might make sense to add it to the docs or release notes. Technically this is now already part of 24.05, so the Uakari release notes can be an appropriate place; but at it is a delayed backport it also makes sense announcing this in the 24.11 release notes again.


Simulator: Track Statistics Immutably

Summary
Can anyone give a hint on how to immutably keep track of the statistics?
Description
My current approach is to write a Recorder class and pass it to every relevant Event, which involves fields that are not declared final. I know I could wrap those fields to an ArrayList though, to get pass CodeCrunch, but are there any really immutable ways of doing it?

Hi,
I think you're on the right track. Just return immutable Recorder class after any event is passed in.
For instance, I have a Statistics.java with the constructor Stats(double waitingTime. int numCustomersServed, numCustomersLeft.
In the spirit of immutability, let's say if I pass inside a Leave event, then I have a method called left() which returns a new Statistics instance, return new Statistics(this.waitingTime, this.numCustomersServed, this.numCustomersLeft + 1)
This will enable your declared variables to be both private and final. Hope this answered your question.

I don't collect data until the event queue is empty. Instead, I add every ServeEvent to a list and every WaitEvent  to another list when they are polled from the event queue. After everything is settled, I calculate the difference of time stamp between each pair of WaitEvent and ServeEvent associated to the same customer to get the average waiting time. The number of customers served is just the size the the list consisting of ServeEvent, and the number of customers who left is the number of customers - this size.

I also have a dedicated stats class, itself containing a PriorityQueue which logs all events. It has a different Comparable method compared to the 'true' PriorityQueue involved in the simulation. Everything that happens in the 'true' PriorityQueue gets flushed into this when resolved, which gives me control over how whether to include certain things (like a RestEvent). Like @Hongyi6328, my duration spent waiting is also 'retroactively' calculated, though the logic itself is done method-side, before a modified WaitEvent is added to to the event logs.

Thank you!


Create JS Resources Doc

Example links to include:

https://www.freecodecamp.org/news/why-i-left-gulp-and-grunt-for-npm-scripts-3d6853dd22b8/
https://github.com/steveukx/git-js
https://www.freecodecamp.org/news/introduction-to-npm-scripts-1dbb2ae01633/
https://www.npmjs.com/package/shelljs
https://www.twilio.com/blog/npm-scripts
https://docgov.dev/posts/npm-scripts/
https://www.freecodecamp.org/news/introduction-to-npm-scripts-1dbb2ae01633/
https://blog.npmjs.org/post/118810260230/building-a-simple-command-line-tool-with-npm

More JS links

https://www.regextester.com/
https://www.typescriptlang.org/play

Maybe we should make this a more general resource...  here are some more useful links
Markdown to PDF Links

https://emmanuelbosquet.com/2022/awesome_pdf_with_eisvogel/
https://forum.obsidian.md/t/generate-custom-pdfs-with-pandoc-panrun-and-the-eisvogel-latex-template/22237

reference regarding --- (triple dashes) syntax in YAML https://camel.readthedocs.io/en/latest/yamlref.html#general-syntax

Another helpful set of notes for mocha

to run only one group of tests change describe(...) to describe.only(...) and execute tests with npx mocha
to run only one test change it(...) to it.only(...) and execute tests with npx mocha


find file dialog obscures last row option when path is longer than 2 rows

Working normally:

Last row obsured:

Oooh, this is fun. Will take a look this weekend.

Closing this as I've stopped using childframes for ivy/swiper.


Hlb hotfix

Hardlight bow hotfix, arrows are reflected by reflective vest, was supposed to be part of the buff PR

Change disabler arrow description because it says that is stuns in 2 hits, it's in hardlight_bow_protos file.

@Aviu00 it is done


Bluetooth: Controller: Periodic adv. sync. degraded performance on latest main branch

Describe the bug
In the below table it can be seen that current main branch of Zephyr is not working as we expect. We expect the performance to be as with nRF Connect 2.0 or better.
This is tested with https://github.com/zephyrproject-rtos/zephyr/pull/49429 code, which is a small addition to the main branch as of today.

15 Tags nRF Connect 2.0
15 Tags Zephyr main

20 Tags nRF Connect 2.0
20 Tags Zephyr main

One difference I can see in the logs are that for Zephyr main branch I get many prints as below.
That is sync_term callback but the le address is invalid (garbage).
PER_ADV_SYNC[4]: [DEVICE]: 00:D9:AC:00:00:00 (public-id) sync terminated reason: 62
[00:00:17.917,205] <err> u_df_scan: Got sync_term from none queued tag!

Please also mention any information which could help others to understand
the problem you're facing:

What target platform are you using? - Zephyr SDK and NRF52833
Is this a regression? - Bisect is in progress...

To Reproduce
Build below application based on the direction_finding_connectionless_rx sample (the application does not enable CTE sampling by default but can still reproduce the issue).
aoa_receiver_multiple_configureable.zip
There are some options in the top of the main.c file, default is using periodic adv. list. which is also from where the images above are taken using.
NOTE if you build for other commits than https://github.com/zephyrproject-rtos/zephyr/pull/49429 the line CONFIG_BT_CTLR_SYNC_PERIODIC_SKIP_ON_SCAN_AUX=y needs to be commented out in prj.conf
Have about 10 tags sending periodic advertisements (I used about 50ms interval and about 30 bytes per. adv. data payload, this is probably not needed) for example use direction_finding_connectionless_tx sample, the periodic_adv should also work, but I have not tried.
Expected behavior
Equal behaviour or better in Zephyr main branch compared to Zephyr version used in nRF Connect 2.0
Impact
Degraded performance when tracking many tags in an application
Logs and console output
ncs_2-0-20-tags.log
ncs-2-0-15-tags.log
zephyr-main-15-tags.log
zephyr-main-20-tags.log
Environment (please complete the following information):

OS: Windows
Toolchain: Zephyr SDK
Commit SHA or Version used: Used the code as in https://github.com/zephyrproject-rtos/zephyr/pull/49429

@jakkra Thank you again for analysing.
It does not make sense as it is only switching to new Link Layer Control Procedure implementation as default that only is used in connection oriented usecases (also, for connection oriented direction finding BT_LL_SW_LLCP was always the default).
Could you check if reverting https://github.com/zephyrproject-rtos/zephyr/commit/76396496b6d9ea24a1e644565c5225222e63984f help? (I am guessing here based on the only recent changes related to periodic advertising sync).

Ok thanks, I'll come back tomorrow with more info, perhaps it was more of a coincidence the performance change in this commit. I'll do some more investigation.

@cvinayak commit 76396496b6d9ea24a1e644565c5225222e63984f is indeed the problem, reverting it the behaviour is same/similar to nRF  Connect 2.0.
However I'm still getting some (currupt?):
(114925) PER_ADV_SYNC[9]: [DEVICE]: 00:D9:AC:00:00:00 (public-id) sync terminated reason: 62
[00:01:54.925,933] <err> u_df_scan: Got sync_term from none queued tag!

I'll backtrack when it started.

@cvinayak actually the invalid PER_ADV_SYNC[9]: [DEVICE]: 00:D9:AC:00:00:00 (public-id) sync terminated reason: 62 is not something new, I see it now on Zephyr version used in nRF Connect 2.0 also. Just that I saw it a lot before reverting 76396496b6d9ea24a1e644565c5225222e63984f and thought it was new.
Do you have any input regarding this? Want me to open a new issue? It's not a huge problem as I for now can handle

@jakkra to confirm, https://github.com/zephyrproject-rtos/zephyr/commit/76396496b6d9ea24a1e644565c5225222e63984f is the only change for which a fix is needed, right? if so, I will upstream a PR to add a Kconfig to permit your application to not use the reduced time reservations for periodic sync PDU reception.
PER_ADV_SYNC[9]: [DEVICE]: 00:D9:AC:00:00:00 (public-id) sync terminated reason: 62 is a typical issue that can occur due to the fact that the Bluetooth Specification only permits 6 periodic interval of missed PDUs before declaring that a sync establishment has failed; If there are more than 6 peer Periodic Advertising devices that have overlapping intervals, sync established to be failed would occur. (You may confirm by only using 5 peer advertisers whether a repeated the 5 periodic sync establishments for a long duration tests scenario do not report any failures of reason 62).

@cvinayak yes commit 76396496b6d9ea24a1e644565c5225222e63984f fixes the issues with degraded sync performance.
About sync term and the reason 62, I'm more talking about that the MAC address is invalid/garbage and not corresponding to any tags being tracked. The bt_le_per_adv_sync_term_info bt_le_per_adv_sync parameters are mostly just 0's.

@jakkra sorry I misunderstood, yes please create a new github issue for the invalid device address in the sync terminated.
@Thalley Is there any chance the host could be filling the invalid device address in the sync terminated callback when sync fails to be established?
Could the reference to addr that is assigned to the info and then the instance is deleted be the cause? https://github.com/zephyrproject-rtos/zephyr/blob/main/subsys/bluetooth/host/scan.c#L909-L917

Hmm, the only scenario that I can see is that if the scanning fails in bt_le_per_adv_sync_create and that the sync is established before the host manages to call bt_le_per_adv_sync_delete (which would cancel it), as the address is only set if the scanning succeeds or if the device is already scanning:
	if (!atomic_test_bit(bt_dev.flags, BT_DEV_SCANNING)) {
		err = bt_le_scan_update(true);

		if (err) {
			bt_le_per_adv_sync_delete(per_adv_sync);
			return err;
		}
	}

	*out_sync = per_adv_sync;
	bt_addr_le_copy(&per_adv_sync->addr, &param->addr);
	per_adv_sync->sid = param->sid;

Hmm, actually if the address supplied is garbage, which would probably result in sync terminated with error code 62 (BT_HCI_ERR_CONN_FAIL_TO_ESTAB), then that would explain it.
Please verify that the address in the sync parameters is correctly set (potentially provide an HCI log as well).

@cvinayak @Thalley I think I found the issue. Since I'm using perioidic adv. list the current get_pending_per_adv_sync() will have a none populated le addr.
struct bt_le_per_adv_sync_param sync_create_param;
sync_create_param.options = BT_LE_PER_ADV_SYNC_OPT_USE_PER_ADV_LIST;
sync_create_param.skip = 0;
sync_create_param.sid = 0; // Not used but needs to be 0 otherwise error
sync_create_param.timeout = SYNC_LOST_TIMEOUT_10MS_UNIT;
memset(&sync_create_param.addr, 0xFF, sizeof(bt_addr_le_t)); // <= Added this now
err = bt_le_per_adv_sync_create(&sync_create_param, &syncForList);

With memset(&sync_create_param.addr, 0xFF, sizeof(bt_addr_le_t)); part and the address in sync_term is now 0xFF .. 0xFF... Previously I did not touch the sync_create_param.addr, hence the random bytes.
So the problem is here https://github.com/zephyrproject-rtos/zephyr/blob/main/subsys/bluetooth/host/scan.c#L910 that
term_info.addr = &pending_per_adv_sync->addr; should be populated with the address coming from the HCI event instead of pending_per_adv_sync =>
term_info.addr = &evt->adv_addr;
term_info.sid = evt->sid;

Ah, yes, that makes sense. Indeed it should use the event address (the event address should be compared to the pending_per_adv_sync->addr when the list is not being used.
For using the list, I guess pending_per_adv_sync->addr should be set to the NONE address in case of using the list.

yes commit https://github.com/zephyrproject-rtos/zephyr/commit/76396496b6d9ea24a1e644565c5225222e63984f fixes the issues with degraded sync performance.

@jakkra Please test your application with #49579 and let me know if it solves the performance issue.
You can also check if the following improves Extended Advertising Report generation by increasing CONFIG_BT_CTLR_SCAN_AUX_SET:
CONFIG_BT_CTLR_ADVANCED_FEATURES=y
CONFIG_BT_CTLR_SCAN_AUX_SET=2

@cvinayak I have now tested https://github.com/zephyrproject-rtos/zephyr/pull/49579 with my sample application (attached to this issue) and it looks promising. Will test with our main application on monday also.
250ms per adv interval and 45 tags.

@cvinayak yes I was able to sync to around ~40/45 tags simultaniously, as you can see in the plot they come and go a bit (blue dots goes to yellow means sync was lost), but around 40 seems to be the limit for 250ms per.adv interval.

hope you are using a supervision timeout > (250 ms* 45)
Do you mean supervision timeout for an additional BLE connection, or parameters inside of bt_le_per_adv_sync_param?
What I do in the sample application is to just add all tags to per. adv. list and never removing them, hence no sync timeout really used (I do this in our "real" application though).

I just wrote a long reply here https://github.com/zephyrproject-rtos/zephyr/issues/39376#issuecomment-1230088110 after testing those two, at the end I wrote about a new problem I found:

When syncing with more than about 5 tags with per.adv interval of about 10ms then after a while (not long, like 10s) no events are generated from the controller (no scan rsp, no ext. scan rsp, no sync term, no cte etc.).

When syncing with more than about 5 tags with per.adv interval of about 10ms then after a while (not long, like 10s) no events are generated from the controller (no scan rsp, no ext. scan rsp, no sync term, no cte etc.).

Ok. Let me try to debug this and if I find something will address it in #49579

@jakkra I have tested 5 tags synchronized under simulation under two conditions; one, all 5 tags have overlapping periodic advertising and all the 5 tags got synchronized; two, 5 tags with a progressive offset over time and only 4 tags got synchronized. But in both cases I was using duplicate filtering, stopping and starting scanning,  saw no issues, and periodic_sync device discovered extended advertising all along (except the one that was not synchronized). In the second case 5th tag was not discovered and this is possible if the auxiliary PDU of the 5th was overlapping any of the periodic advertising sync constantly. Keeping the periodic interval at 10 ms and extended advertising at a multiple of 10 ms would cause constant overlap between extended advertising event and periodic advertising event.
Extended Advertising auxiliary PDU transmissions are implemented as periodic events, a design choice to support non-overlapping multiple Extended and Periodic sets as a broadcaster. This means multiple broadcaster devices with same extended advertising interval will have overlapping auxiliary PDUs and make it difficult for a Periodic Sync device in discovering these overlapping broadcasters.
My testing yesterday was under simulation in BabbleSim. But on a target I suspect the overlapping may be too much for CPU processing between overlapping events. Something that was discovered during conformance testing of mixed multiple sets of extended advertising, periodic advertising synchronization and on multiple PHYs.
On target, the temporary solution was https://github.com/zephyrproject-rtos/zephyr/commit/a40a403a0dfc8fdf573b4ba436d51d84d68a8bd1
There is room for improvement in how ticker module schedules closely expiring timeouts. Currently, a minimum of 3 ticks is used between consecutive ticker expiry callbacks implemented by:
https://github.com/zephyrproject-rtos/zephyr/blob/be7173a49054e34ee02d1e7ec713aabb6acc89a7/subsys/bluetooth/controller/ticker/ticker.c#L2405-L2429
Use of minimal time reservation causes many just-in-time scheduling decisions being taken and consecutive ticker expiry that consume time leading to the winning event being delayed to an amount that it cancels itself not being able to meet the hard real-time instant required to start the radio event.
To keep the just-in-time scheduling mitigation at O(1), i.e. scheduling decision take only based on time reservation, we have to not use the minimal time reservations, need #49579:
CONFIG_BT_CTLR_ADVANCED_FEATURES=y
CONFIG_BT_CTLR_SCAN_AUX_SYNC_RESERVE_MIN=n

Please let me know if the above Kconfig setting is acceptable by your application.

@cvinayak thanks for the detailed explaination.

Keeping the periodic interval at 10 ms and extended advertising at a multiple of 10 ms would cause constant overlap between extended advertising event and periodic advertising event.

I'll make sure that extended adv. int. is not a multiple of periodic adv. interval. In the tags I have extended adv. int. set to 200ms, so in case of 10ms periodic adv. int. then what you say can occur. However regarding the issue I talked about It's the same problem at 7.5ms and up to somewhere around 10ms+.
Take a look on this log:
7.5ms per. adv interval used, 200ms extended interval.
It's ok that some tag is never seen due to reasons you describe. However the problem is that something seems to break and NO events at all are genereated from controller. Even after I power of all transmitters, no sync term event or any other event is sent. I made some comments in the log wrapped around ----'s.
*** Booting Zephyr OS build zephyr-v3.1.0-4071-ge64c3ad1c2fc  ***
Starting Connectionless Locator Demo
Bluetooth initialization...success
Scan callbacks register...success.
Periodic Advertising callbacks register...success.
Start scanning...success
[00:00:00.004,547] <inf> main: Creating Sync for tags in list
Waiting for periodic advertising...
Scan is running...
[00:00:00.111,145] <err> main: scan_recv: per. adv. interval: 6
[00:00:00.111,389] <dbg> main: scan_recv: Added tag from per adv list
[00:00:00.272,705] <err> main: scan_recv: per. adv. interval: 6
[00:00:00.272,918] <dbg> main: scan_recv: Added tag from per adv list
[00:00:00.531,311] <err> main: scan_recv: per. adv. interval: 6
(539) PER_ADV_SYNC[0]: [DEVICE]: 33:46:8A:57:AC:25 (random) synced, Interval 0x0006 (7 ms), PHY LE 1M
[00:00:00.540,069] <dbg> main: sync_cb: Removed tag from per adv list
[00:00:00.540,069] <inf> main: Creating Sync for tags in list
Enable receiving of CTE...
(540) success. CTE receive enabled.
[00:00:00.796,844] <wrn> bt_hci_core: Malformed data
[00:00:00.834,259] <err> main: scan_recv: per. adv. interval: 6
[00:00:00.834,472] <dbg> main: scan_recv: Added tag from per adv list
[00:00:00.951,110] <err> main: scan_recv: per. adv. interval: 6
Running...
TAGS:28,sync...,sync...,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,
Queued: 2 Synced: 1 Num CTE with err: 0/28
[00:00:01.254,364] <err> main: scan_recv: per. adv. interval: 6
(1264) PER_ADV_SYNC[1]: [DEVICE]: 21:ED:DB:E2:70:50 (random) synced, Interval 0x0006 (7 ms), PHY LE 1M
[00:00:01.264,862] <dbg> main: sync_cb: Removed tag from per adv list
[00:00:01.264,892] <inf> main: Creating Sync for tags in list
Enable receiving of CTE...
(1265) success. CTE receive enabled.
[00:00:01.464,263] <err> main: scan_recv: per. adv. interval: 6
[00:00:01.581,085] <err> main: scan_recv: per. adv. interval: 6
[00:00:01.699,951] <wrn> bt_hci_core: Malformed data
[00:00:01.952,728] <err> main: scan_recv: per. adv. interval: 6
(1970) PER_ADV_SYNC[2]: [DEVICE]: 3D:78:84:09:C1:4D (random) synced, Interval 0x0006 (7 ms), PHY LE 1M
[00:00:01.970,306] <dbg> main: sync_cb: Removed tag from per adv list
[00:00:01.970,306] <inf> main: Creating Sync for tags in list
Enable receiving of CTE...
(1970) success. CTE receive enabled.
Scan is running...
Running...
TAGS:49,0,28,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,
Queued: 0 Synced: 3 Num CTE with err: 0/77
[00:00:02.211,059] <err> main: scan_recv: per. adv. interval: 6
[00:00:02.304,260] <err> main: scan_recv: per. adv. interval: 6
[00:00:02.372,589] <err> main: scan_recv: per. adv. interval: 6
[00:00:02.631,072] <err> main: scan_recv: per. adv. interval: 6
[00:00:02.724,243] <err> main: scan_recv: per. adv. interval: 6
[00:00:03.002,563] <err> main: scan_recv: per. adv. interval: 6
Running...
TAGS:26,25,23,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,
Queued: 0 Synced: 3 Num CTE with err: 0/75
[00:00:03.261,047] <err> main: scan_recv: per. adv. interval: 6
[00:00:03.354,248] <err> main: scan_recv: per. adv. interval: 6
[00:00:03.503,662] <wrn> bt_hci_core: Malformed data
[00:00:03.774,261] <err> main: scan_recv: per. adv. interval: 6
[00:00:03.842,498] <err> main: scan_recv: per. adv. interval: 6
Running...
TAGS:25,23,21,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,
Queued: 0 Synced: 3 Num CTE with err: 0/69
[00:00:04.101,043] <err> main: scan_recv: per. adv. interval: 6
[00:00:04.170,562] <wrn> bt_hci_core: Malformed data
[00:00:04.194,274] <err> main: scan_recv: per. adv. interval: 6
[00:00:04.262,481] <err> main: scan_recv: per. adv. interval: 6
Scan is running...
[00:00:04.485,198] <err> main: scan_recv: per. adv. interval: 6
[00:00:04.485,412] <dbg> main: scan_recv: Added tag from per adv list
[00:00:04.521,026] <err> main: scan_recv: per. adv. interval: 6
[00:00:04.614,440] <err> main: scan_recv: per. adv. interval: 6
[00:00:04.628,692] <wrn> bt_hci_core: Malformed data
[00:00:04.892,456] <err> main: scan_recv: per. adv. interval: 6
Running...
TAGS:27,25,25,sync...,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,
Queued: 1 Synced: 3 Num CTE with err: 0/77
[00:00:05.034,332] <err> main: scan_recv: per. adv. interval: 6
[00:00:05.301,177] <wrn> bt_hci_core: Malformed data
[00:00:05.454,315] <err> main: scan_recv: per. adv. interval: 6
[00:00:05.955,200] <err> main: scan_recv: per. adv. interval: 6
(5980) PER_ADV_SYNC[3]: [DEVICE]: 03:AE:43:82:9C:BC (random) synced, Interval 0x0006 (7 ms), PHY LE 1M
[00:00:05.980,712] <dbg> main: sync_cb: Removed tag from per adv list
[00:00:05.980,712] <inf> main: Creating Sync for tags in list
Enable receiving of CTE...
(5981) success. CTE receive enabled.
[00:00:05.990,997] <err> main: scan_recv: per. adv. interval: 6
Running...
TAGS:31,27,25,0,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,
Queued: 0 Synced: 4 Num CTE with err: 0/84
[00:00:06.084,442] <err> main: scan_recv: per. adv. interval: 6
[00:00:06.206,146] <wrn> bt_hci_core: Malformed data
[00:00:06.362,396] <err> main: scan_recv: per. adv. interval: 6
[00:00:06.433,654] <wrn> bt_hci_core: Malformed data
[00:00:06.504,455] <err> main: scan_recv: per. adv. interval: 6
Scan is running...
[00:00:06.782,379] <err> main: scan_recv: per. adv. interval: 6
Running...
TAGS:17,16,17,16,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,
----------------------------Here everything stops working----------------------------
Queued: 0 Synced: 4 Num CTE with err: 0/66
Running...
TAGS:0,0,0,0,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,
Queued: 0 Synced: 4 Num CTE with err: 0/0
Running...
TAGS:0,0,0,0,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,
Queued: 0 Synced: 4 Num CTE with err: 0/0
Running...
TAGS:0,0,0,0,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,
----------------------------Around here I reset all tags.----------------------------
Queued: 0 Synced: 4 Num CTE with err: 0/0
Running...
TAGS:0,0,0,0,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,
Queued: 0 Synced: 4 Num CTE with err: 0/0
Running...
TAGS:0,0,0,0,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,
Queued: 0 Synced: 4 Num CTE with err: 0/0
Running...
TAGS:0,0,0,0,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,
Queued: 0 Synced: 4 Num CTE with err: 0/0
Running...
TAGS:0,0,0,0,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,
Queued: 0 Synced: 4 Num CTE with err: 0/0
Running...
TAGS:0,0,0,0,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,
Queued: 0 Synced: 4 Num CTE with err: 0/0
Running...
TAGS:0,0,0,0,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,
Queued: 0 Synced: 4 Num CTE with err: 0/0
Running...
TAGS:0,0,0,0,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,
Queued: 0 Synced: 4 Num CTE with err: 0/0
Running...
TAGS:0,0,0,0,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,
Queued: 0 Synced: 4 Num CTE with err: 0/0
Running...
TAGS:0,0,0,0,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,
Queued: 0 Synced: 4 Num CTE with err: 0/0
Running...
TAGS:0,0,0,0,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,
Queued: 0 Synced: 4 Num CTE with err: 0/0
Running...
TAGS:0,0,0,0,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,
Queued: 0 Synced: 4 Num CTE with err: 0/0
Running...
TAGS:0,0,0,0,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,
Queued: 0 Synced: 4 Num CTE with err: 0/0
Running...
TAGS:0,0,0,0,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,
Queued: 0 Synced: 4 Num CTE with err: 0/0
Running...
TAGS:0,0,0,0,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,
Queued: 0 Synced: 4 Num CTE with err: 0/0
Running...
TAGS:0,0,0,0,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,
Queued: 0 Synced: 4 Num CTE with err: 0/0

the problem is that something seems to break and NO events at all are genereated from controller. Even after I power of all transmitters, no sync term event or any other event is sent.

I have a hunch that there is a flaw in the implementation of all abort_cb that is closing the radio event that is aborted by an overlapping event. The implementation is not taking the control path that should be checking for supervision/synchronization timeout.
Let me do some investigation and get back.

Okey great, thanks. Just a quick question regarding
CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC vs. CONFIG_CLOCK_CONTROL_NRF_K32SRC_XTAL I know that the biggest difference is power consumption, but in case of periodic sync with many tags, will it make a difference?
Is my understanding correct that if internal K32RC is used then the time radio is used will be longer (due to window widening?) hence higher power consumption, BUT this should also case the radio being "wasted", or is this time negliable ?

Larger window widening = higher power consumption. Radio is listening earlier and longer to detect missing PDUs than when the clock accuracy is better. Hence, lower clock accuracy means each radio event needs higher event length to use.
What does this mean for Zephyr Controller, the time reservation is fixed (minimal by default). If a PDU arrived late in the window , overflows beyond the time reservation and with a chance to overlap with another event, then the just-in-time negotiation would get used.
Also what I mean is with low clock accuracy more radio event would slide into each other to overlap often.

@cvinayak Yes sure, I'll do that, thanks.

@cvinayak created https://github.com/zephyrproject-rtos/zephyr/issues/49915


License

When logging a word with a special configuration, it should show the license.

Instead, a procedure was created.


tests: disable keyring instead of setting a fail keyring

Related to python-poetry/poetry#9820

Do we have coverage elsewhere for failing keyring backends?

This is poetry-plugin-export. And this is the default config fixture for all tests. I do not think that we need coverage for failing keyring backends here. I assume we just want to deactivate keyring for the tests because as we all know it can cause trouble.
I am only doing this change because we should not import keyring without declaring it as dev dependency.


New interactive window points to root folder / instead of workspaceFolder

Applies To

[ ] Notebooks (.ipynb files)
[X] Interactive Window and/or Cell Scripts (.py files with #%% markers)

What happened?
When opening a new interactive window, pwd points to / rather than the workspace folder.
PS. I already found a workaround in a another issue, which ironically seems to have been fixed! Either there's something wrong on my end or the bug has somehow resurfaced:
"jupyter.notebookFileRoot": "${workspaceFolder}",

VS Code Version
Version: 1.76.1 Commit: 5e805b79fcb6ba4c2d23712967df89a089da575b Date: 2023-03-08T16:31:31.010Z Electron: 19.1.11 Chromium: 102.0.5005.196 Node.js: 16.14.2 V8: <IP_ADDRESS>-electron.0 OS: Linux x64 5.17.5-76051705-generic Sandboxed: No
Jupyter Extension Version
v2023.2.1200692131
Jupyter logs
info 09:50:13.000: Starting interactive window for resource '' with controller '.jvsc74a57bd0b58bd5559424689280ce24ff6229e536533c877108d283a6d2846312dfd182d7./home/amin/mambaforge/envs/sci/python./home/amin/mambaforge/envs/sci/python.-m#ipykernel_launcher (Interactive)'
info 09:50:13.539: Starting Jupyter Session startUsingPythonInterpreter, .jvsc74a57bd0b58bd5559424689280ce24ff6229e536533c877108d283a6d2846312dfd182d7./home/amin/mambaforge/envs/sci/python./home/amin/mambaforge/envs/sci/python.-m#ipykernel_launcher (Python Path: /home/amin/mambaforge/envs/sci, EnvType: Conda, EnvName: 'sci', Version: 3.9.13 | packaged by conda-forge | (main, May 27 2022, 16:58:50)
[GCC 10.3.0]) for '/Interactive-1.interactive' (disableUI=false)
info 09:50:13.562: Process Execution: > ~/mambaforge/envs/sci/bin/python -c "import ipykernel; print(ipykernel.__version__); print("5dc3a68c-e34e-4080-9c3e-2a532b2ccb4d"); print(ipykernel.__file__)"
> ~/mambaforge/envs/sci/bin/python -c "import ipykernel; print(ipykernel.__version__); print("5dc3a68c-e34e-4080-9c3e-2a532b2ccb4d"); print(ipykernel.__file__)"
info 09:50:13.605: Process Execution: > ~/mambaforge/envs/sci/bin/python -m ipykernel_launcher --ip=<IP_ADDRESS> --stdin=9003 --control=9001 --hb=9000 --Session.signature_scheme="hmac-sha256" --Session.key=b"20d84fa9-e7ae-403a-8313-dd563833483c" --shell=9002 --transport="tcp" --iopub=9004 --f=/home/amin/.local/share/jupyter/runtime/kernel-v2-13803537JMmKTR4f6nf.json
> ~/mambaforge/envs/sci/bin/python -m ipykernel_launcher --ip=<IP_ADDRESS> --stdin=9003 --control=9001 --hb=9000 --Session.signature_scheme="hmac-sha256" --Session.key=b"20d84fa9-e7ae-403a-8313-dd563833483c" --shell=9002 --transport="tcp" --iopub=9004 --f=/home/amin/.local/share/jupyter/runtime/kernel-v2-13803537JMmKTR4f6nf.json
info 09:50:13.606: Process Execution: cwd: ~/Code/CL-model
cwd: ~/Code/CL-model
info 09:50:13.791: ipykernel version & path 6.15.2, ~/mambaforge/envs/sci/lib/python3.9/site-packages/ipykernel/__init__.py for /home/amin/mambaforge/envs/sci/bin/python
info 09:50:14.282: Started Kernel sci (Python 3.9.13) (pid: 1382210)

Coding Language and Runtime Version
Python 3.9.13 | packaged by conda-forge | (main, May 27 2022, 16:58:50)
Language Extension Version (if applicable)
v2023.4.1
Anaconda Version (if applicable)
No response
Running Jupyter locally or remotely?
Locally

seems to be working fine for me.  Does the default value (${fileDirname}) for that setting work if you run a file in the interactive window?

Hi,
I have the exact same issue - the value in my extension is the default value as suggested:
"jupyter.notebookFileRoot": "${fileDirname}"

When I run a file in the interactive window, it works without problems ¯_(ツ)_/¯

@amunger, I don't work with workspaces, I just open the root directory of the project I want to work on in VS Code.

Duplicate of https://github.com/microsoft/vscode-jupyter/issues/13057


Can you add guidance for determining how to add a 2nd app gateway to a private subnet?

I am unable to determine how to programmatically find the next unused IP address in a private subnet which already contains a V2 application gateway, because the output of $Subnet.ipConfigurations.privateIPAddress is empty, despite some IP addresses being in use by the existing V2 App Gateway.
I can determine the next available IP through trial and error, but this is not scalable for automation.
Thanks!

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 1b1e0765-a7aa-6e76-3aca-751dc8a531dc
Version Independent ID: e95861b3-cfd3-2b75-95e8-1c0176ba79f2
Content: Quickstart: Direct web traffic using the portal - Azure Application Gateway
Content Source: articles/application-gateway/quick-create-portal.md
Service: application-gateway
GitHub Login: @greg-lindsay
Microsoft Alias: greglin

@pgbfnf
Thanks for your feedback! We will investigate and update as appropriate.

Hi @pgbfnf We already have a guidance document for determining the available IP address in a Application gateway subnet. Please refer: Azure Application Gateway infrastructure configuration | Microsoft Learn
Please do take a look at tip in this doc which says:

We don't have a readily available solution or command to find the available IP address in a subnet programmatically.
We are closing this issue for now. If there are further questions regarding this matter, please reply and we will gladly continue the discussion.

Very disappointing. You're essentially saying this is not possible (programmatically) and you have no plans to make it possible.

@pgbfnf
Thanks for your contribution. Please add your feedback in below link, so our production team can review it and update the same.
Ideas · Community (azure.com)


Docker container not accepting POST request

First of all, thanks to Jeff for your amazing work and all the documentation work you do, it's fantastic and deeply appreciated as I learn a lot from it!
I have the same problem as #6 I think.
I logged into the serial port of the device and noticed that the POST request got 404'ed too.
I managed to get the error code to change to 403 by hopping into the container and creating the folder /var/www/html/sensors/airgradient:d024a3/measures/ as well as changing the permissions with chown -R 33:33 .
but for some reason the container do not 'accept' the POST request:
{"wifi":-69,"pm02":2,"rco2":1151,"atmp":23.20,"rhum":35}
http://<IP_ADDRESS>:9926/sensors/airgradient:d024a3/measures
301
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

<title>301 Moved Permanently</title>

Moved Permanently

The document has moved <a href="http://<IP_ADDRESS>:9925/sensors/airgradient:d024a3/measures/">here.

<address>Apache/2.4.51 (Debian) Server at <IP_ADDRESS> Port 9925</address>

My guess is that this has to do with the changes done in the last pull request to support multiple airgradient devices, but that the docker run command needs to be updated. Is this accurate? How could I go into debugging my issue?

This might help you out delete  /airgradient:d024a3/measures then it will work.

I had the same problem as #6 but it's because the Docker commands pull a generic "php:8-apache" image. It's looking for files that are in the Git repo, in the "/var/www/html" directory. I created and copied the contents of "index.php", "metrics", and ".htaccess" into the html directory and it started seeing 200 from the POST requests in the logs. I also pointed Prometheus at the "/metrics" endpoint and was receiving data successfully.
One other thing to note is the Docker CLI and Compose seem to run different commands so I combined them into
docker run -d -p 9925:80 --name airgradient -v /path/to/direcotry:/var/www/html php:8-apache /bin/bash -c 'mkdir /sensors; chown -R 33:33 /sensors; chown -R 33:33 html; a2enmod rewrite; apache2-foreground'
making sure to set the directory to bind. In the CLI from the README this is "-v "$PWD":/var/www/html"

Spot on @jorddenn ! I just had realized that index.php was not in the /var/www/html/ folder and had just put it there when I saw your message. I added "metrics" and ".htaccess" and started seing 200 status right away!
For future other un-informed and confused readers: the .ino file instructs the device to http POST under the URL http://containeraddress:port/sensors/airgradient:d024a3/measures
But the index.php script placed at the root of the website (/var/www/html/) intercepts the request, read the path ( $_GET['path'] ), and if the base folder is '/sensors', then it processes the POST request. When you query http://containeraddress:port/metrics, you do not query a folder but the php file 'metrics' which serves you the information stored and processed by index.php.
Given that these files are not exported by the container build instruction provided, you get 404 if you try to POST or GET at these endpoints.
I will propose a PR with a modified docker-compose file.


When can the MRKL Agent be supported?

When can the MRKL Agent be supported?

@junjiem what is your use case? Many things are already supported: RAG, tools, etc.

Duplicate of https://github.com/langchain4j/langchain4j/issues/193


Fix incremental compiler not detecting removal of a jar on classpath

This commit fixes #1913 by making the incremental compiler rely on the previous compilation jar snapshot
to detect whether a jar has been removed or not. The rationale for this change is that the incremental task
inputs detection has a bug when the collection is ordered: while it will properly detect that something has
changed, the information it returns is incorrect: it would notify of modified jars for jars which haven't
changed, notify of removal jars which have not been removed, and eventually miss the real removal.
A proper fix for incremental task inputs change detection will be issued later. In the meantime, we will
use the information the incremental compiler already knows as the basis for analysis.
This pull request also incidentally fixes #1761 , which was actually due to wrong information reported by IncrementalTaskInputs.

I think we need a bit more test coverage:

Behaviour when the classpath order is changed.
Behaviour when using directories instead of jars (ie move the test coverage up to the superclass)


[Discussion] Planning Architecture

Discussion
In relation to #1677, we need to create the design overview of planning.
According to @dejanpan , we can start this by defining inputs/outputs and logics in these 4 layers.

Global planning
Behaviour planning
Local planning
Control

Thanks for his suggestion, for example,
https://github.com/CPFL/Autoware/issues/1677#issuecomment-435721786

Global planning => on demand service, interface to apriori map and traffic information,
Behaviour planning => open problem, Darpa urban challenge: rule-based, other companies: search-based method, our favorite: robust POMDP
Local planning => some deterministic sampling-based methods (GMT) or form of MPC, pre-recorded path, RRT
Control

Inputs to this module:
track predictions
map
pose

I think the Motion Layer in present Autoware architecture (https://github.com/CPFL/Autoware/wiki/Overview) includes 3. Local planning and 4. Control .
I created the draft design about these and commented at  https://github.com/CPFL/Autoware/issues/1677#issuecomment-438166365 .

@makokal fyi

Responding to @dejanpan 's request for comment:
I think the main quality of a good stack or architecture is that it should be easy to extend.
I think this is especially important as Autoware is a very ambitious open source project.
Generally, to achieve a stack that is easy to extend, the following is very helpful:

Separation of concerns: each logical entity (class, node etc.) does one thing:

Easier to understand

Well defined interfaces

With respect to this, I think from a conceptual, robotics algorithms level, the standard planning stack of:

Global planning
Behavior planning
Motion planning
Control
Vehicle interface

Is a good one, as it logically separates planning into distinct algorithm domains, making it easy to reason about planning from an algorithm perspective.
How, within each of the five components, each one is structured is a matter of debate, whether it be more class-based, or node based. I personally favor a more monumental, class-based means of separating concerns.
With respect to this, I would advocate that we define standard interfaces (e.g. topic names and types) to each component of the stack.
If we can achieve this, then we would be able to interchange algorithms at each level from various contributors.
The thinking then is that we could have a hierarchy of launch files, e.g.
# planning.launch.py
global_planner = 'global_planner.launch.py'
behavior_planner = 'behavior_planner.launch.py'
# .. etc..

Where then each particular launch file can e.g. dispatch to many nodes, as in @aohsato 's design, or a single node, if it is from another contributor who designs it that way.
That said, here is a first proposal for interface messages. Here, I am favoring standard messages where possible:

Global planning

Input: Waypoint?
Input: [PoseWithCovarianceStamped]
Output: Waypoint?

Behavior planning

Input: [PoseWithCovarianceStamped]
Input: Waypoint?(https://github.com/ros2/common_interfaces/blob/master/geometry_msgs/msg/PoseWithCovarianceStamped.msg)
Output:

Motion planning

Input:
Input(Optional): PoseWithCovarianceStamped
Output: JointTrajectory

Control

Input: PoseWithCovarianceStamped
Input: JointTrajectory
Output: AccelStamped

Vehicle interface

Input: AccelStamped
Output: Direct commands to the drive-by-wire interface (e.g. voltage signal)

Remarks:

The input for a vehicle interface is because in my experience there you typically map voltage to an acceleration command
JointTrajectory allows you to capture higher derivative information i.e. in the case of trajectory planning
PoseWithCovarianceStamp can be used instead of PoseStamped because uncertainty maybe used in implementer's algorithms. IMO the memory burden and communication overhead should be negligible in this message
Global pose is an optional input for the Motion planner as in most senses it is a local planner, so a fixed position is not strictly necessary depending on how you set up the problem
Input/output for global planner depends on how locations are broadly represented in the global frame and in the map. For this we could maybe use something like Waypoint with is extremely broad, but I think this, along with the interfaces for the behavior planner are open questions

Just some suggestions and discussion points. I think on the motion/local planning and control level, those proposals are fairly concrete, as the interfaces here a pretty well understood. At the higher behavior and global planning levels, it might be challenging to design a good interface, since the solutions and problem settings become a lot more broad.

Hi @cho3 @dejanpan @aohsato , I am late to this conversation so I thought I start with aligning the task.
In my opinion, let's separate the design first into functionality before going into details on the interface types. This means hashing out the role of these modules first and also deciding which ones are processes vs libraries. A process communication with a library does not have to go through ROS which is sometimes very useful. Afterwards the interface details become more apparent.
Here is a first shot at this.
Processes

Mission planner

Determine the sequence of lanes and lane segment for a concrete task like drive to parking lot.

Behavior/Maneuver planner

Assess the scene with help of predictions to determine relevant constraints for motion planning, i.e. produce motion plan specification.
All calls motion planner and assesses the output using additional libraries like a constraint checker to measure adherence to say safety rules.

Controller/Tracker

Track a given motion plan and continuously report status upwards.
Vehicle interface should be abstracted away from this to allow for interchanging of controllers, e.g. versions of pure pursuit.

For these process commands from flow the top (mission -> behavior -> motion -> control) with simultaneous increase in information update. Controller runs fastest (100hz, 50hz). Status information is reported upwards. This ensures a single source of 'truth' to allow for consistent decisions.
Libraries

Motion planning

Ego lane planner
Lane change planner
Parking planner
etc, loadable and switchable via the behavior process.

Prediction (from perception)
Constraint checker
Scene model

Representation of the map around Ego and convenience functions for manipulation it. This heavily relies on a proper abstraction of the underlying map format.

With this separation, ROS communication is only needed between the processes.
Additional note: parameter handling is crucial across the full hierarchy here, so I highly recommend using namespaces like /motion_planner/ego_lane/param_x

@aohsato @cho3 @dejanpan comments on ^^?

@makokal :turkey: :turkey: :turkey: Your comment fell a little awkwardly before the short holidays :turkey: :turkey: :turkey:.
Regarding what is written, I broadly agree. I was under the false impression that some of these were a given, but it's best to be as clear as possible.
As far as general development practice goes:

All code should generally be as decoupled as possible to minimize complexity
This implies any ROS code should generally be a boilerplate wrapper around some class in a library

RE Mission/global planner:

I generally agree with how you set up the problem here
I'm not sure if you're intending this or not, but I have some concern that a "lane sequence" might be taken to be too binding depending on the lower level planning setup (e.g. use case I'm thinking here is driving on a long highway--your lane sequence is just one for 100 miles, then an exit)
For above, we might want to consider adding some label of "constraint" for some elements of the sequence that denote that a certain lane segment must be satisfied by the subplanners, but that's more an interface discussion

RE Behavior planner:

Generally agree with the output (i.e. setting up relevant constraints for motion planner as a result)
Again, depending on how the problem is set up, map information may/should be needed here
Constraint checker makes sense--we could probably shove the dependency on maps into the constraint checker and into prediction
If I'm understanding you correctly, you're aiming for the maneuver/motion planner to be embedded in the behavior planner--I don't strictly agree with this:

My main concern is that in pre-specifying a certain architecture, you are implying a certain kind of solution, which I think is not strictly what we're going for with Autoware
The way I understand it, this tight coupling lends itself more to a classical/deterministic planning paradigm, where it makes sense to have many or more calls to a maneuver/motion planner
This would sort of preclude a more decoupled, probabilistic planning paradigm, where the requested maneuver can succeed or fail
Of course, one thing we can say is that the behavior planner and maneuver planner are tightly coupled, so we make no prescriptions on how the two interact, only what the inputs to the two combined are, and what we expect the output of the two to be in aggregate
Further, if we keep things as nice libraries (especially motion planners) with a consistent interface, then it should be trivial to break out motion planners into nodes for a more decoupled intermediate planning paradigm

RE Controller:

Agree, but I'm not entirely sure what status would be reported upward, tracking error?

RE libraries:

All sound good
For something like motion planning, again I would prescribe some kind of consistent interface to make life easier, and I'm perfectly fine with making them loadable and switchable as long as it doesn't preclude other modes of using these libraries (e.g. as standalone nodes, etc.)
Seems to me like scene model and constraint checker are at least lightly coupled--we would probably need to get a good understanding/list of constraints and behaviors and iterate back and forth (from a planning architecture perspective) to get a better scene model

I guess my main comments in short are:

We should go further and make everything libraries, and have ROS code only be boilerplate
I think we should be careful about over prescribing structure in the architecture so we don't block off potential solutions
We should maybe treat motion + behavior planning as a unit from the input/output perspective, with some recommendations on how they might interact
We need a planning interface (look at OMPL?)
Scene model is important--we should look at how other people do it too

@cho3 I concur very much.
RE- global/mission planner - Yes, lane sequence is no the right abstraction, I think waypoints (with some possible semantic meaning) is a good start.
RE-Behaviors:

I think that embedding motion planner in a maneuver planner is just one of many possible architectural setups. I agree that interfaces for motion planning should we well defined to allow for both (use in bp, as well as standalone). Having motion planning as a library of planners encourages this separation.

RE- controllers

I envision status updates about the health of the node itself as well as any online metrics. Again just interfaces are key here and the concrete implementations can be left to user. The idea is to equip high level modules with some capacity to intervene for some recoverable cases instead of just resorting to say emergency stop is things go bad.

I also agree about keeping things as ROS agnostic as possible and only adding wrappers to tie the modules to ROS at the end.
For scene modeling , it is the piece hard to find examples of, yet is can vastly affect the interfaces in motion planning (e.g. representation of obstacles)
I haven't looked deeply in OMPL, but AFAIK it is mostly geared towards manipulation planning, so I am not sure how much of that interface fits AD.
Example MP interface I had in mind:
class MotionPlanner {
// ...
  ReturnType plan(const WorkspaceType& ws, const MPTask& task, const MPConstraints& constraints, ...) const = 0;
//...
};

The have a factory for registering and initializing motion planners.
...
  MPType create(const MPParams& params, ...);
...

RE global planner:

I think how we handle this might depend on what paradigm is chosen for handling maps
Mainly I see 2-3 ways of interacting with maps:

Shared database (e.g. each relevant node can access/query the same map database)
Decoupled (e.g. only top level node, such as global planner, has access to map, and only relevant info is passed via messages down)
Server (similar to 1, but map is a ROS server, rather than some indeterminate object in shared memory)

1 probably wouldn't work out too well in a multi-machine context (of course you could have one instance per machine)
The upshot of 1/3 is that it simplifies the messages passed around in the planning stack, though you're pushing the coupling away to the map abstraction
If we pursue a 1/3 style approach (where each relevant node has direct access to the map DB), then I would suggest something like Uint64MultiArray for a first pass. The main reason is that the assumption that each map entity should be uniquely identifiable by 64 bits seems fairly reasonable, and plus this is a built-in type which makes life a little bit easier as a first pass
If we shoot for something like 2, then I have no idea. I think for that we would have to come up with a couple variations of map-based prediction, localization, and behavior planning to get a sense for what would be needed there

RE-controllers:

Do you have any concrete, high level behaviors you're envisioning here? Or even in the case of recovery behavior for each component of the planning stack? I ask this because I think it's impossible to design a reasonable interface without some concrete use-cases, even if we push the problem of implementation to others

RE-interface:

OMPL has a very reasonable API and separation of concerns for motion planning that at least bears some studying
MoveIt! IMO is more ROS-y in terms of it's paradigms, and less useful for general motion planning
Some other things we would need to be aware of and think about in designing an interface are different kinds of planners and planning paradigms, e.g.

Trajectory planning (combined lateral and longitudinal) vs path + velocity planning decomposition (e.g. what Autoware currently does)
Optimization-based methods vs sampling-based methods vs hybrid (n-stage) methods
etc...

I personally take more of an optimization-based view of motion planning (and everything else for that matter), so from this view, just some random things to consider for an interface/problem definition:

objective function (e.g. progress, smoothness, weights etc..)
state space/motion model (e.g. dubins bicycle, first order wheels etc.., what level of derivative to control..)
time horizon
discretization level (in time or space domain)
initial conditions
state space constraints (e.g. 0 <= x <= 10)
obstacle constraints
waypoint constraints

RE-example-interface

If I'm understanding your interface correctly, why would we need workspace to be a parameter rather than keeping the workspace as a set of class variables? Or do you mean workspace to mean something like the broad solution domain of the problem (e.g. 0 <= x <= 10)?

RE global planner:
* I think how we handle this might depend on what paradigm is chosen for handling maps

* Mainly I see 2-3 ways of interacting with maps:

  1. Shared database (e.g. each relevant node can access/query the same map database)
  2. Decoupled (e.g. only top level node, such as global planner, has access to map, and only relevant info is passed via messages down)
  3. Server (similar to 1, but map is a ROS server, rather than some indeterminate object in shared memory)

* 1 probably wouldn't work out too well in a multi-machine context (of course you could have one instance per machine)

* The upshot of 1/3 is that it simplifies the messages passed around in the planning stack, though you're pushing the coupling away to the map abstraction

* If we pursue a 1/3 style approach (where each relevant node has direct access to the map DB), then I would suggest something like [Uint64MultiArray](http://docs.ros.org/jade/api/std_msgs/html/msg/UInt64MultiArray.html) for a first pass. The main reason is that the assumption that each map entity should be uniquely identifiable by 64 bits seems fairly reasonable, and plus this is a built-in type which makes life a little bit easier as a first pass

* If we shoot for something like 2, then I have no idea. I think for that we would have to come up with a couple variations of map-based prediction, localization, and behavior planning to get a sense for what would be needed there

I agree with the options. Just want to add one more thing that the full map or a large portion of it is not needed by most of these components. Only routing and lane level reasoning need access to large portions of the map and they mostly use topological information and some geometric info. I think these can be represented in some object build from a map (which could be living in a server/database etc) and the object is updated rarely.
More intermediate components like motion planners, behavior planner and controllers do not really need access to a map but rather just a definition of workspace. These could again be objects build from either the map portion used in routing or a combination. These workspace objects will need to be constructed every time a planning 'call' is to be made so it is important the process if efficient.
For optimization based planners for instance, these workspace object could then represent continuous spaces.
This decouples how localization interacts with maps. The routing and planning modules here then only consume output of a localization module (as some 'State' object, again an interface).

RE-controllers:
* Do you have any concrete, high level behaviors you're envisioning here? Or even in the case of recovery behavior for each component of the planning stack? I ask this because I think it's impossible to design a reasonable interface without some concrete use-cases, even if we push the problem of implementation to others

Mainly state of the controller process, as in health and online tracking errors. The idea is to provide a channel for a user to propagate such diagnostics upwards and do something with them if need be or just log for debug purposes.

RE-interface:
* [OMPL](http://ompl.kavrakilab.org/api_overview.html) has a very reasonable API and separation of concerns for motion planning that at least bears some studying

Took a look, I like their models for spaces, objective functions and all. I think we can borrow a lot from these. What I didn't see (maybe didnt look hard enough) was arc length parameterization type spaces, i.e. Frenet coordinate which are prevalent in AV.

* [MoveIt!](http://moveit.ros.org/documentation/concepts/) IMO is more ROS-y in terms of it's paradigms, and less useful for general motion planning

* Some other things we would need to be aware of and think about in designing an interface are different kinds of planners and planning paradigms, e.g.

  * Trajectory planning (combined lateral and longitudinal) vs path + velocity planning decomposition (e.g. what Autoware currently does)
  * Optimization-based methods vs sampling-based methods vs hybrid (n-stage) methods
  * etc...

* I personally take more of an optimization-based view of motion planning (and everything else for that matter), so from this view, just some random things to consider for an interface/problem definition:

I like the optimization approach very much too. I think we can have some basic interfaces for all of these and have a reference implementation of an optimization based planner from which other users can derive and add more.

  * objective function (e.g. progress, smoothness, weights etc..)
  * state space/motion model (e.g. dubins bicycle, first order wheels etc.., what level of derivative to control..)
  * time horizon
  * discretization level (in time or space domain)
  * initial conditions
  * state space constraints (e.g. 0 <= x <= 10)
  * obstacle constraints
  * waypoint constraints

RE-example-interface
* If I'm understanding your interface correctly, why would we need workspace to be a parameter rather than keeping the workspace as a set of class variables? Or do you mean workspace to mean something like the broad solution domain of the problem (e.g. 0 <= x <= 10)?

Partially answered above, but the idea is to have planners separate run time changaeble things from configuration things. Params are set at initialization and kept as member variables (with hooks to updates at some arbitrary rate), then the problem/task description which entails the workspace, constraints, etc are set on each call. This allows to also log these and be able to recover the 'exact world' the planner used to solve the task.

Finally,  I think the discussion so far has been very fruitful, how should we go forward now?
UML style design?

@makokal Glad this discussion was helpful.
I guess for next steps we need to consider what the purpose of this discussion to hash out an architecture was. IMO the point of hashing out an architecture is so that we can establish standards to:

Make the code quality better
Make it easier for developers to develop algorithms that can interact with other developer's algorithms

In light of that I guess some kind of document would be best. IMO it should probably have:

General development guidelines (e.g. ROS code should only be boilerplate, turn on all warnings, not doing XYZ will get your MR rejected, etc. most open source projects have developer guidelines IIRC)
High level description of architecture (e.g. simple block diagrams) + high level descriptions of what each component does (high level UML here?)
More detailed descriptions of the architecture and components (class-level UML here?):

the more concrete expected behavior of each component
the API's for commonly defined libraries (i.e. motion planning)
the interface (i.e. message definitions)
some sample use cases and expected behavior in each case

@makokal Glad this discussion was helpful.
I guess for next steps we need to consider what the purpose of this discussion to hash out an architecture was. IMO the point of hashing out an architecture is so that we can establish standards to:
1. Make the code quality better

2. Make it easier for developers to develop algorithms that can interact with other developer's algorithms

Very well articulated.

In light of that I guess some kind of document would be best. IMO it should probably have:
1. General development guidelines (e.g. ROS code should only be boilerplate, turn on all warnings, not doing XYZ will get your MR rejected, etc. most open source projects have developer guidelines IIRC)

Should we do this on this repo, maybe via a series of Markdown docs or elsewhere?

2. High level description of architecture (e.g. simple block diagrams) + high level descriptions of what each component does (high level UML here?)

3. More detailed descriptions of the architecture and components (class-level UML here?):

   * the more concrete expected behavior of each component
   * the API's for commonly defined libraries (i.e. motion planning)
   * the interface (i.e. message definitions)
   * some sample use cases and expected behavior in each case

I think we should get this tarted right away so that any lingering issues/decisions can be surfaced sooner.

Let me get at it.

Dear @makokal , @cho3  this is very great discussion.
I am looking forward to see your design document.
Let me share with you some thoughts:

Planning has very wide range of implementations and standards. so fixing the design on what you prefer or like (for example Optimization based trajectory planner) is not the way to go. you should think more generally.

You should not include vehicle_control/path_following in the planning design, it is different problem.

Planning modules should support simulation, planning based on HD maps, planning based on Occupancy Grid (free space planners).

- Kindly check OpenPlanner, it is part of Autoware:

Design and scientific bases, check paper https://www.fujipress.jp/jrm/rb/robot002900040668/

check the tutorial and illustration videos on YouTube. https://youtu.be/FKM8v79X3_s

Initially, OpenPlanner design was to separate Global Planning (way_planner) and Local Planning (dp_planner). you can still find the nodes and tested in simulation mode.

Secondly from last year we had discussion with Autoware team to divide dp_planner to several nodes (similar proposal as @dejanpan suggested. check the pull request: https://github.com/CPFL/Autoware/pull/1400

There is one library which contains the Planning functions and Logic op_planner. the nodes only concerned with message passing, parameters, synchronization, visualization. (same as suggested) with the addition that I like to separate the visualization in the future.
The only enforcement for others to implement their own sub modules is the nodes interface definition:
OpenPlanner.Nodes.pdf


reduced country list on map page

https://redpen.io/mn9c5f1f7edc6c579d

@markherringer already fix and push staging


missing typo in chain comand.

Short Description:
missing chain command.

Sorry it same as #3057.


Oyodo instead of Ooyodo

This is the only version where Oyodo is used as the romaji instead of Ooyodo which is seen everywhere else. Any particular reason for this?
It's just a minor thing, but it bugs me enough that I feel compelled to change it manually myself everytime I update.

Translate is work of many people so it may have some inconsistency. If you found any of it, please let me know. Thank you.

Since I'm not using the translation feature since I'm a Japanese speaker, just FYI.
I dare say "Oyodo" is correct and "Ooyodo" is incorrect.
US Navy officialy called her "Oyodo":
http://www.history.navy.mil/our-collections/artifacts/research-guides/us-naval-technical-mission-to-japan-reports-in-the-navy-department-library.html#ships
(Also, there are some other documents found at this site.)
Wikipedia(en) calls her "Oyodo".
http://en.wikipedia.org/wiki/Japanese_cruiser_Ōyodo
And we write down "大淀川" as Oyodo River and "大淀区" as Oyodo-ku.
Lastly, "Ooyodo" is not correct Japanese Romaji, so I'm not sure how to pronounce this notation in Japanese.
HTH,  but I think it's OK to leave it as it is if the majority of English-speaking Teitokus prefer to do so.

Yeah that's why I asked if there was a reason for this. There's often a lot of discrepancies when お/ょう gets translated into romaji it seems. But I agree, in this case Ooyodo is probably preferred because it's consistent with the English community.

Japanese speakers tend to use O or Ō while Western general users tend to use Ō or Oo. It is a matter of preferences.


getMarkdownHeadersInCell does not handle headers with HTML tags

The notebookOutlineEntryFactory.ts handles markdown headers with HTML tags, but not other parts (such as collapsing mardown header cells, see foldingModel.ts).
			const fullContent = cell.getText().substring(0, 10000);
			for (const { depth, text } of getMarkdownHeadersInCell(fullContent)) {
				hasHeader = true;
				entries.push(new OutlineEntry(index++, depth, cell, text, false, false));
			}

			if (!hasHeader) {
				// no markdown syntax headers, try to find html tags
				const match = fullContent.match(/<h([1-6]).*>(.*)<\/h\1>/i);
				if (match) {
					hasHeader = true;
					const level = parseInt(match[1]);
					const text = match[2].trim();
					entries.push(new OutlineEntry(index++, level, cell, text, false, false));
				}
			}

Thus, if we have HTML tags for outline headers, then we do not render collapsible regions in markdown for such headers.
Not sure if this is a bug or not, but filing this here,
Feel free to close this if this is invalid.

Good catch, I'll see about unifying that behavior. We should likely just support html tags in the folding model. Might've just been an oversight way back when this was written and didn't grab the updates with the outlineEntryFactory.


Stale task parameters showing up in plugin configuration.

Version
3.11.0
Environment info
Environment Info:

  System:
    OS: Windows 10
    CPU: (16) x64 AMD Ryzen 7 3800X 8-Core Processor
  Binaries:
    Node: 12.11.1 - C:\Program Files\nodejs\node.EXE
    Yarn: Not Found
    npm: 6.11.3 - C:\Program Files\nodejs\npm.CMD
  Browsers:
    Edge: 44.18362.387.0
  npmPackages:
    @vue/babel-helper-vue-jsx-merge-props:  1.0.0
    @vue/babel-plugin-transform-vue-jsx:  1.0.0
    @vue/babel-preset-app:  3.11.0
    @vue/babel-preset-jsx:  1.1.0
    @vue/babel-sugar-functional-vue:  1.0.0
    @vue/babel-sugar-inject-h:  1.0.0
    @vue/babel-sugar-v-model:  1.0.0
    @vue/babel-sugar-v-on:  1.1.0
    @vue/cli-overlay:  3.11.0
    @vue/cli-plugin-babel: ^3.11.0 => 3.11.0
    @vue/cli-plugin-eslint: ^3.11.0 => 3.11.0
    @vue/cli-service: ^3.11.0 => 3.11.0
    @vue/cli-shared-utils:  3.11.0
    @vue/component-compiler-utils:  3.0.0
    @vue/preload-webpack-plugin:  1.1.1
    @vue/web-component-wrapper:  1.2.0
    eslint-plugin-vue: ^5.0.0 => 5.2.3 (4.7.1)
    vue: ^2.6.10 => 2.6.10
    vue-cli-plugin-buefy: ^0.3.7 => 0.3.7
    vue-eslint-parser:  2.0.3 (5.0.0)
    vue-hot-reload-api:  2.3.4
    vue-loader:  15.7.1
    vue-router: ^3.0.3 => 3.1.3
    vue-style-loader:  4.1.2
    vue-template-compiler: ^2.6.10 => 2.6.10
    vue-template-es2015-compiler:  1.9.1
    vuex: ^3.0.1 => 3.1.1
  npmGlobalPackages:
    @vue/cli: Not Found

Steps to reproduce

run "vue ui"
create an empty project (without starter content) and "default preset"
in project manager, click on "tasks" -> "serve" -> "parameters", then click save to close.
click on "plugins" -> "add plugin"

What is expected?
I expect I'm supposed to see a list of plugins I can install.
What is actually happening?
I see the same configuration parameters that was in the tasks->serve->parameters page.  I am unable to select "search" or "files changed" tabs (no response).

This is my first <5 minute foray into vue.js, hopefully this is just a minor bug.

Related (I think), I'm getting a lot of 404's when simply running vue ui. then even more when I try to install a plugin. I think the ui is half-loading, causing a chain of other issues.


Allow search by snippet name in Insert Snippet menu

After installing a snippet pack, it's common for me to think: "I know I have a snippet for this, but I don't know the prefix." To find the prefix, I have to either Insert Snippet and scroll until I find it, or open the documentation for the relevant snippet pack.
My proposed solution is to make text typed in the Insert Snippet menu match both Snippet prefixes and descriptions. For example, to find "Add event listener," I could type add rather than having to guess that its prefix is ae:

Was rather easy to do...

👏 👏 👏


Run function and shorcut problem

hello when i use
alt + space for run command its works great but if focus in other windows (like libreoffice or chrome) to works space keyboard doesn't work its always load run search bar

@micker
this will be fixed in 0.19.1
Closing as duplicate of https://github.com/microsoft/PowerToys/issues/4578
Thanks.


v2.3.0-beta.4 - Bug background

Hello,
Here is a problem I observed using v2.3.0-beta.4:

Pop-up with image in the background do not appear in the pop-up, but in the editor the background is correct.

Here's the custom style I use:
.bubble-pop-up {
transition: transform 0.3s ease;
background: center / cover no-repeat url("/local/epellis_theme/blue.jpg") fixed;
border-radius: 12px 12px 0 0;
box-sizing: border-box;
margin-left: var(--custom-margin);
left: 7px;
z-index: 5 !important;
bottom: calc(-56px - var(--custom-height-offset-mobile));
backdrop-filter: var(--custom-popup-filter);
-webkit-backdrop-filter: var(--custom-popup-filter);
box-shadow: 0px 3px 8px 0px rgba(255, 174, 66, 0.35) !important;
border-radius: 20px;
border-style: solid;
border-width: 1px;
border-color: rgba(255, 255, 255, 0.35) !important;
}
Thank you! 🍻

Hi! Indeed I changed something in the previous release, this will be a breaking change for some users that use that kind of custom styles. I just fixed the editor to reflect the same styling, I will also add a note about it in the changelog.
Wait for the next release, there will be a new class named .bubble-pop-up-background.


New source ddtc enforcements

@pudo are there any useful topics I can add this to? Not sure violating export controls fits under crim.fin.
ok to add them to the crime collection anyway?
they're not actually all sanctioned (mostly not, it seems) and historical data is included, so I'm not sure the sanction topic or collection make sense.
is it useful to add the link to each PDF as sourceUrl to each sanction even if we can't label the links?

I'd add them to crime, and maybe even add crime.traffick to the topics for each. And definitely yes on the sourceUrl PDFs :)


Where should I put new languages files in Laravel ?

Hi !
I'm trying to add a new language file to display 'mn' instead of 'minute', etc...
I've created app/lang/altitude/date.php and tried to use Date::setLocale('altitude') or App::setLocale('altitude') but it does not seem to work.
Any idea ?

app/lang/packages/altitude/date/date.php in that case

Thanks :wink:


Bump django-crispy-forms to 2.0

Similar to #28 , but focused on the upgrade of the django-crispy-forms dependency only :slightly_smiling_face:

Add Python 3.12
Remove Django 2.2, 3.0 and 3.1
Add Django 4.1 and 4.2

CI passes: https://github.com/olivierphi/tbxforms/actions/runs/7447980089 :heavy_check_mark:

These three settings are still mentioned within the README and referenced within code:

TBXFORMS_ALLOW_HTML_LABEL
TBXFORMS_ALLOW_HTML_HELP_TEXT
TBXFORMS_ALLOW_HTML_BUTTON

Do they still have a place now that |safe has been removed?

Can you please update the Django and django-crispy-forms requirements listed in the README: https://github.com/torchbox/tbxforms#requirements

Ah, I just noticed that this was merged - sorry I couldn't address your comments, I found myself in a "client work" whirlwind these last few weeks :pensive:

No worries, a few other MRs have merged recently, so I'm working through a few changes (including those!) to prep a release :).
Thanks for everything you've done here - it's all looking mega!


Update repo to build and test against Windows App SDK

new WinUI projection
update WinUIDesktopSample to use new WinUI projection
update objectlifetime test

There's a sample app I made as part of the PR above that I will add to this.
Otherwise, this looks like stripping out all WinUI references and replacing them with Microsoft.ProjectReunion.WinUI references.
We should consider dropping the WinUI projection and the WinUIDesktopSample. And we should rename the Reunion projection to WindowsAppSDK at some point...

Requirements:
[ ] Add a new projection for WinAppSDK 1.0.0 (or whatever the latest 1.0.x is)
[ ] Update ObjectLifetimeTests to use the new projection
[ ] Update WinUIDesktopSample to use the new projection


Separate license for design assets?

I am going to shortly merge #1839. I see this as a license for all of the code examples and website source code itself. The Coding Train design assets I would prefer to live under a different license likely a creative commons one (non-commercial?). Should any other elements be licensed separately? Not sure the best way to handle this.

Not entirely sure on the legality of how this would work but you could change the LICENCE file to have two sections with two licences and say above each which file types or folders they apply to. Not sure if it would be lawyer proof but I think it would make it clear to most people what the intentions are.

Had a thought on how to go about this problem. We could move all images in to a separate repository which has the community commons licence. Then it could be nested in the assets folder on this repository. Could also add a npm post install script to clone the images repository because chances are most people wouldn't use --recursive when cloning the main website repository. The advantage is a clear distinction between images and code. The disadvantage is it could easily confuse those unfamiliar with nested repositories on github. Also would require those with post install scripts turned off to either use --recursive or manually run the script. I could work on it if you (@shiffman) think it's a good idea.

I love the thinking behind this, but I think this is a low priority issue for me (at the moment) and I'm unlikely to run into any significant issues for how the media assets are used. In the end, I think a friendly notice in the README is "good enough" for now and if we reach the point where it's important to more meaningfully separate them out we can move forward with a plan like this!
For now, I would prioritize ease and simplicity of maintenance and brining in new contributors!
Let's merge the license and add a note about licensing in the main README!


Fixed #24084 - Changed HTMLSelectElement Index Signature

This PR addresses https://github.com/Microsoft/TypeScript/issues/24084.
I changed the HTMLSelectElement index to [name: string]: HTMLOptionElement | HTMLOptGroupElement.
Thanks for considering this PR.

String indexers generally don't work, unless the target interface has no other members.

A string indexer type should be of a type that covers every property on the type.

Thanks for the comments.
I changed the signature to [name: number]: HTMLOptionElement | HTMLOptGroupElement


Collection using many different extended classes

For example, I have an Item class and a WeaponItem class that extends Item. It has a couple extra fields in it that need to be saved as well. Is it possible to store this is one collection, and return the appropriate class instance based on which type of item it is? My current thoughts is I am going to have to have an ItemModel class that has every field in it for all classes, and then will instantiate a separate object based on that. I am hoping there's a better way I can do this though.

Hi @emrys90, yes, you can store all deriveded classes in some collections. Take a look in this unit test:
https://github.com/mbdavid/LiteDB/blob/master/LiteDB.Tests/Tests/PolymorphicTest.cs

Nice! Thank you!


Bitmap function :  'CopyPixels'   is error

Describe the bug
CopyPixels function parameters has problem,the positions of the two parameters srcAddress and dstAddress are reversed.
Screenshots
If applicable, add screenshots to help explain your problem.

OS: [e.g. Windows, Mac, Linux (State distribution)]
Version [e.g. 11.0.0-rc1.1]

Please fill the form. It's there for a reason.
Important: which version of Avalonia? Can you file a minimal repo to check?

I am using version 11.0.4, but I see this problem also exists under the master branch

Ah I see, the screenshot is from Avalonia source. So I guess if you can provide a valid fix for the mentioned issue, a PR is welcome.
@Gillibald may have more knowledge if there is a specific reason for the order to be reversed, but I can't imagine any.

WAIT, it will be a breaking change. So only can be considered for 12.0.
Sorry for the confusion 😆

CopyPixels transfers pixel data out of the image into a user-provided buffer. It is working as intended. To update bitmap contents use WriteableBitmap.


WMS ID: 11452 New OAC workshop

Add here your WMS ID
General requirements

Information in the workshop is adequate and updated
Code is correct and working
Links are correct
Make sure you enter the help email link in your manifest.json
Please make sure WMS URLs are updated as needed after your PR is approved

Checklist - Refer to the QA document for the complete list
Please confirm that the following is completed before submitting your PR

[x] All filenames are lower case (including folders, images, files, etc.)
[x] Filenames are descriptive
[x] Your workshop folder structure should be similar to the one used in the sample workshop (https://github.com/oracle-livelabs/common/tree/main/sample-livelabs-templates/sample-workshop)
[x] Are you using multiple versions (desktop/, sandbox/, tenancy/)? Make sure that each of them contains a manifest.json and an index.html
[x] Image references in markdown contain an alternate text

Hi, I am seeing multiple directories:

web-semantic-modeler Part 2
web-semantic-modeler copy

Please make sure these changes are in one root directory (this lab's directory) and any unused files are cleaned up and removed

Hi Norman, I removed the two unnecessary directories. Thanks.

Rest looks good to me,
Is the desktop directory going to be used anywhere?


Use php-http/promise for the async interface

Like discussed in #86

Shouldn't actually be merged until the Promise getResponse/getException methods are removed from everywhere.

Yeah but we need this to be merged to upgrade other libs :)

We can rely on the feature branch for the time being, then magically merge everything at the same time.

All pr are done IMO


IBC: Add support for IBC details in extension

This just requires adding the WASM files to the SDK. We can do this now that the indexer is sending the hash we need.

Done and follow up tracked in #1196