Zephyr sysbuild / multi image

This is an PR for a direction of multi image support in Zephyr, as discussed in dev-review meeting Nov. 11. 2021.
Fixes: #40309
Documentation PR: #43846
~Depends on: #41301~
With #37872 the issue regarding multi image build system came up again.
This PR takes another direction, as it does not interfere with the existing Zephyr CMake build system infrastructure.
Zephyr build code and samples are not impacted with this PR.
Instead a dedicated west build project has been created in <zephyr>/sysbuild (feel free to propose alternative location).
This project is capable of building a regular Zephyr sample as well as MCUBoot.
Each sample is seen as a dedicated project.
west build has been extended to support multi image for building, as well as west flash supports flashing of all images in the multi image build.
Per default, everything builds as normal, that is a single image build IS Still build as:
west build -bnrf52840dk_nrf52840 samples/hello_world

which gives:
-- west build: making build dir /projects/github/ncs/zephyr/build pristine
-- west build: generating a build system
Loading Zephyr default modules (Zephyr base).
-- Application: /projects/github/ncs/zephyr/samples/hello_world
...

but the same sample can now also be build and flash with mcuboot (or in principle any other multi image setup), simply by enabling MCUboot and build with sysbuild option:
$ west build -bnrf52840dk_nrf52840 samples/hello_world/ --sysbuild -- -DSB_CONFIG_BOOTLOADER_MCUBOOT=y
...
   *********************************
   * Running CMake for hello_world *
   *********************************
...
-- Build files have been written to: /projects/github/ncs/zephyr/build/hello_world
--
   *****************************
   * Running CMake for mcuboot *
   *****************************
...
-- Build files have been written to: /projects/github/ncs/zephyr/build/mcuboot
-- Configuring done
-- Generating done
-- Build files have been written to: /projects/github/ncs/zephyr/build
-- west build: building application
[9/16] Performing build step for 'mcuboot'
[1/257] Preparing syscall dependency handling

[247/257] Linking C executable zephyr/zephyr_pre0.elf

[251/257] Linking C executable zephyr/zephyr_pre1.elf

[257/257] Linking C executable zephyr/zephyr.elf
Memory region         Used Size  Region Size  %age Used
           FLASH:       37836 B        48 KB     76.98%
            SRAM:       23808 B       256 KB      9.08%
        IDT_LIST:          0 GB         2 KB      0.00%
[11/16] Performing build step for 'hello_world'
[1/155] Preparing syscall dependency handling

[145/155] Linking C executable zephyr/zephyr_pre0.elf

[149/155] Linking C executable zephyr/zephyr_pre1.elf

[155/155] Linking C executable zephyr/zephyr.elf
Memory region         Used Size  Region Size  %age Used
           FLASH:       17680 B       412 KB      4.19%
            SRAM:        5440 B       256 KB      2.08%
        IDT_LIST:          0 GB         2 KB      0.00%
[16/16] Completed 'hello_world'

where it can be seen that both hello_world and mcuboot were successfully build.
Both samples can be flash in a single invocation with west flash:
$ west flash
-- west flash: rebuilding
[0/6] Performing build step for 'mcuboot'
ninja: no work to do.
[1/6] Performing build step for 'hello_world'
ninja: no work to do.
[6/6] Completed 'hello_world'
-- west flash: using runner nrfjprog
-- runners.nrfjprog: mass erase requested
Using board 683848500
-- runners.nrfjprog: Flashing file: /projects/github/ncs/zephyr/build/mcuboot/zephyr/zephyr.hex
Parsing image file.
Applying system reset.
Verified OK.
Enabling pin reset.
Applying pin reset.
-- runners.nrfjprog: Board with serial number 683848500 flashed successfully.
-- west flash: using runner nrfjprog
Using board 683848500
-- runners.nrfjprog: Flashing file: /projects/github/ncs/zephyr/build/hello_world/zephyr/zephyr.signed.hex
-- runners.nrfjprog: Flashing file: /projects/github/ncs/zephyr/build/hello_world/zephyr/zephyr.signed.hex
Parsing image file.
Applying system reset.
Verified OK.
Enabling pin reset.
Applying pin reset.
-- runners.nrfjprog: Board with serial number 683848500 flashed successfully.
-- runners.nrfjprog: Board with serial number 683848500 flashed successfully.

Resulting in following output on the terminal.
*** Booting Zephyr OS build v2.7.99-1479-g6fb2ea314f83  ***
I: Starting bootloader
I: Primary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
I: Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
I: Boot source: none
I: Swap type: none
I: Bootloader chainload address offset: 0xc000
I: Jumping to the first image slot
*** Booting Zephyr OS build v2.7.99-1479-g6fb2ea314f83  ***
Hello World! nrf52840dk_nrf52840

Running ninja -Cbuild sysbuild_menuconfig will provide a sysbuild configuration interface:

menuconfig for hello_world is started as always so that the user doesn't need to change workflow.

west build -bnrf52840dk_nrf52840 samples/hello_world -t menuconfig

menuconfig for mcuboot can be started by pre-fixing menuconfig with the name of the image.

west build -bnrf52840dk_nrf52840 samples/hello_world/ -t mcuboot_menuconfig

If users want to build using sysbuild per default, then a config build option can be set, like:
$ west config build.sysbuild true

which will build using sysbuild per default.
A user enabling sysbuild per default can still choose to build without sysbuild in specific cases by applying --no-sysbuild, like:
$ west build -bnrf52840dk_nrf52840 samples/hello_world/ ---no-sysbuild

Slides from devreview presentaion:
SYSBuild architectural overview

List of alternative name proposals instead of sysbuild:

Aeolus, proposed here

In Greek mythology, Aeolus was the keeper of the winds

FYI @shubhamkulkarni97

@tejlmand  does this support the model used in https://github.com/zephyrproject-rtos/zephyr/pull/37872, where mcuboot is not a zephyr application but a baremetal bootloader or this https://github.com/zephyrproject-rtos/zephyr/pull/37098 (secureshield)?

@tejlmand does this support the model used in #37872, where mcuboot is not a zephyr application but a baremetal bootloader

The infrastructure in this RFC supports adding extra bootloaders, either Zephyr based or bare-metal.
The exact implementation details for the Espressif mcuboot or Espressif IDF bootloader is something I will coordinate with @shubhamkulkarni97 and @sylvioalves .

or this #37098 (secureshield)?

This is one more PR which should benefit from the multi image build infrastructure in this RFC.
@YuguoWH feel free to take a look at this RFC, and let me know if you would like to see how #37098 could fit into this scheme.

dev-review (Dec 2, 2021):

Suggest to present this at TSC to get high level discussion, direction, approach feedback.

This looks great. How do you tie in specifying what partition(s) the image(s) are linked to?

Will this support a multi-image build of two or more Zephyr applications (neither of which is a bootloader)?
I have a use case where we build our Zephyr application twice, and then pack the two images together into one binary.  There is a single CONFIG_ difference between each build of our application.
This is for our security model where we store two versions of our application in the SoC flash.

Will this support a multi-image build of two or more Zephyr applications (neither of which is a bootloader)?

Yes it does.
You can see a use-case example here:
https://github.com/tejlmand/zephyr/commit/efa77ac2e8507fea7bf4be4a67bd0b42af7899a7
where the samples/subsys/ipc/openamp is build as a Zephyr sample, and then the samples/subsys/ipc/openamp/remote is built as a second Zephyr sample.
For a remote core in this case, but the principle is the same if the two Zephyr samples are build for the same core.

This looks great. How do you tie in specifying what partition(s) the image(s) are linked to?

In the demo given at the TSC, the image is placed according to the zephyr,code-partition, because MCUboot was being enabled.
This can be seen here:
https://github.com/zephyrproject-rtos/zephyr/blob/91d4b7766ceee7fabb5caa026ba181c47286e28b/Kconfig.zephyr#L563-L565
and here:
https://github.com/zephyrproject-rtos/zephyr/blob/91d4b7766ceee7fabb5caa026ba181c47286e28b/Kconfig.zephyr#L83-L84
https://github.com/zephyrproject-rtos/zephyr/blob/91d4b7766ceee7fabb5caa026ba181c47286e28b/Kconfig.zephyr#L94-L97
https://github.com/zephyrproject-rtos/zephyr/blob/91d4b7766ceee7fabb5caa026ba181c47286e28b/boards/arm/nrf52840dk_nrf52840/nrf52840dk_nrf52840.dts#L22
and MCUboot is placed at boot partition according to:
https://github.com/zephyrproject-rtos/mcuboot/blob/225b02468bf77806bdd5f503725256cb345970d2/boot/zephyr/dts.overlay#L3
		zephyr,code-partition = &boot_partition;

and as you can see, all this logic is already present on main so no changes were needed for this to work in the RFC.
If you want to add a third image, you may specify the code partition in a devicetree overlay and use that overlay when doing a sysbuild.
By having a higher level sysbuild system, which this PR / RFC presents, then it will be possible to support some default code-partition overlays depending on which images are enabled.
I have some design sketches but consider exact implementation as followup work, once the foundation has been laid down (this RFC).

Looks great, I'll take reference from that openamp example to see if I could use this to bring up ARC secureshield multi-images.

I haven't looked at the code at all, but I am still very much opposed to making west build use the sysbuild build system.
It is an important design decision that sysbuild is a layer on top of the zephyr build system, even though they may happen to share some code as an implementation detail. The west build system is the interface to the zephyr build system, not sysbuild. So I don't want west build to be able to do anything that cmake (using the zephyr build system) + ninja/make cannot do.
I am sure you are going to do an excellent job with sysbuild to meet your use cases, but I believe other people are going to have different use cases, and they are going to develop their own sysbuild-style wrappers, and they should be able to do so without worrying that west build is going to start pulling in your wrapper.

Is it possible to build multiple images and link them together with this approach? From looking at the changed files and architectural overview slides, it appears that both images are built separately, than flashed in sequence. For the use case in #41579, the primary image must load the secondary image from flash into memory. To do this, it must be aware of the location and size of the second image in flash.
The current approach used in that PR involves linking the binary file holding the secondary image into the primary core's build, but the bare minimum requirement needed to make this multiple image use case possible is a way to know the size in flash of the remote core image before building the primary core image.

Is it possible to build multiple images and link them together with this approach?

Technically you cannot link two applications, what you can link are object files / libraries.
To embed an application into another you would first need convert it, and then embed it into the other image.

From looking at the changed files and architectural overview slides, it appears that both images are built separately, than flashed in sequence.

That is correct, because on high level this will allow to build for multi core SoC that might have different architectures, and / or flashing procedures. Not to mention that not all SoC are using hex files, so it may not be possible in all cases to merge files together. As images in principle are independent of each other one can then decide to build and flash only the image that might have changes.

For the use case in #41579, the primary image must load the secondary image from flash into memory. To do this, it must be aware of the location and size of the second image in flash.

And as was discussed at dev-review, this is an important use-case. I've create a meta issue here: #41640 to track various specific use-cases.
Feel free to create a use-case with this specific information and link to it from #41640
This architecture and thus solution with this PR is extendable with custom image handling.
SYSbuild shall itself shall ensure that important build artifacts from a Zephyr based image are available as by-products, so that other images has to possibility to depend on each other (as long as they don't introduce circular dependencies)

It is an important design decision that sysbuild is a layer on top of the zephyr build system, even though they may happen to share some code as an implementation detail. The west build system is the interface to the zephyr build system, not sysbuild. So I don't want west build to be able to do anything that cmake (using the zephyr build system) + ninja/make cannot do when it comes to invoking higher layers.

My main concern is about confused users who might fail to understand when west build or west <new_command> command should be used.
For example, if one does west build samples/hello_world, and want to enable MCUboot and do a multi-image build does west build samples/hello_world -- -DCONFIG_BOOTLOADER_MCUBOOT=y, not realizing that the correct way is:
west <new_command> samples/hello_world -- -DCONFIG_SB_BOOTLOADER_MCUBOOT=y.
Or the use-case where we have multi-image behind the scenes in subsys/ipc/openamp
Users building with west build samples/subsys/ipc/openamp would have to build such samples with the west <new_command> instead, but still stick to west build for other samples like hello_world.
With the proposal in this RFC the same command is always used (west build), even when only a single image is built.
Another option we can consider is to add two new arguments to west build like this:
west build --sysbuild
west build --no-sysbuild

That way CI systems can always decide a specific type of build.
User's could select their preferred default when no argument is given, by setting west config build.sysbuild true / west config build.sysbuild false.
west build could then default to --no-sysbuild for a stabilization period, and if we decide things work well, then make --sysbuild the default in future.
Users can of course always decide their own way to build with --no-sysbuild / west config build.sysbuild false if they prefer that.
@mbolivar-nordic does that sound more acceptable to you ?

Hi @tejlmand , I've tried to use the sysbuild to build two images for ARC secureshield, I notice that the two images are build/debug/flash in sequence. Especially when I debug the program, it invokes two debuggers. In our case when debugging, the second image should be load by debugger command rather than invoke twice, so I wonder if there's a way to make debugger onle be invoke once? Thanks.

Hi @tejlmand , I've tried to use the sysbuild to build two images for ARC secureshield, I notice that the two images are build/debug/flash in sequence. Especially when I debug the program, it invokes two debuggers. In our case when debugging, the second image should be load by debugger command rather than invoke twice, so I wonder if there's a way to make debugger onle be invoke once? Thanks.

@YuguoWH Thanks for testing this out.
This observation is totally correct and is one of the reasons this PR is still in draft, as you will notice in this commit message:
https://github.com/zephyrproject-rtos/zephyr/pull/40555/commits/01a77c481a0231f291743bf9035d3047ae6cf794
Still TODO:
- Support a `--image <name>` argument to specify only a specific runner
  in an image should be invoked, for example:
  `west debug --image mcuboot` to debug mcuboot.
  Or:
  `west flash --image blinky` to (re-)flash only blinky and not all
  images.
- Support a default runner for operations where only single runner
  should be invoked, for example `debug`.
  Default should be the application image

So for now, when debugging with this PR, you can tell west directly the image to debug, for example if you build using:
$ west build -b<board> samples/hello_world/ -p -dbuild -- -DCONFIG_SB_BOOTLOADER_MCUBOOT=y

Then you decide the image to debug as:
$ west debug -dbuild/hello_world

or
$ west debug -dbuild/mcuboot

I would like to propose Aeolus as name for this feature, instead of sysbuild:

In Greek mythology, Aeolus was the keeper of the winds

https://en.wikipedia.org/wiki/Aeolus_(Odyssey)

I would like to propose Aeolus as name for this feature, instead of sysbuild:

In Greek mythology, Aeolus was the keeper of the winds

https://en.wikipedia.org/wiki/Aeolus_(Odyssey)

Maybe Eol or Aíolos. The same person but it's easier to pronounce :)

Another option we can consider is to add two new arguments to west build like this:

Discussed offline, I am fine with this approach

Support a --image <name> argument to specify only a specific runner

@tejlmand what would you say to making this --domain instead of --image? That would align with the terminology used by system devicetree, and also makes a definitive break between the downstream NCS multi-image build terminology and the upstream sysbuild terminology.

I would like to propose Aeolus as name for this feature, instead of sysbuild:

In Greek mythology, Aeolus was the keeper of the winds

https://en.wikipedia.org/wiki/Aeolus_(Odyssey)

@mbolivar-nordic @tejlmand and myself have discussed the name again. We think that sysbuild is a good candidate because:

"System" is generic enough that can cover wildly different scenarios. From a tiny M0 with MCUboot to massive multi-core and multi-image-per-core devices
It shortens nicely to "sys"
The term is currently unused in the build system
It is completely technology-neutral
It aligns with the future "System Devicetree" which targets very similar scenarios and goals

The alternative that comes close at fulfilling the above is "multi-image", but no one has ever liked this as a name for the feature. The feature in fact, results in generating multiple images, but it is more than that.

@tejlmand what would you say to making this --domain instead of --image? That would align with the terminology used by system devicetree, and also makes a definitive break between the downstream NCS multi-image build terminology and the upstream sysbuild terminology.

Sounds like a good proposal.

as before, still not a big fan of the name "sysbuild". This does not represent what this is about and the fact you have to attach "multi image" to the name confirms that sysbuild does not convey the message of this is about and might lead to confusion.
Also, do we have to have the sysbuild folder at the root of the zephyr tree?

as before, still not a big fan of the name "sysbuild". This does not represent what this is about and the fact you have to attach "multi image" to the name confirms that sysbuild does not convey the message of this is about and might lead to confusion.

As I wrote here, although not ideal, I still have not come up with a better suggestion, and this aligns with future System Devicetree work. Given that Devicetree has adopted the adjective "system" to describe "meta" or "extended", I think it's reasonable to adopt it as well.

Also, do we have to have the sysbuild folder at the root of the zephyr tree?

No, I confirmed with @tejlmand that this can be moved. Perhaps cmake/sysbuild/, although it's not only about CMake, so we need to discuss it a bit more.

@JordanYates Thanks for the comments.

This seems to cover the use-cases that I utilize at the moment. Applications with MCUboot and nRF53 boards with hci_rpmsg running on the network core. A few comments:

Great to hear.
Good to know the direction is sane.

I can't see how the build folders interact with west config build.dir-fmt. Does the format recommended here https://docs.zephyrproject.org/latest/guides/west/build-flash-debug.html#setting-source-and-build-directories make sense with sysbuild?

I don't see the west config build.dir-fmt "build/{board}/{app}" as a recommendation, more as an example.
And I do think that formatting still makes sense with SYSbuild, cause then you'll have the following structure for nrf52840dk_nrf52840, hello_world:
build/nrf52840dk_nrf52840/hello_world           <--- This being the SYSbuild build folder,
                               |                     identified with the name of the high level app you're building for
                               |--- hello_world <--- actual hello world build folder containing object, ELF, etc. files
                               |--- mcuboot     <--- actual mcuboot build folder containing object, ELF, etc. files

Now, it would be nice to be able to have a formatting string as: build/{board}/{sysbuild}/{app}" where sysbuild is only set when SYSBuild is used.
That would allow users to even be able to differentiate between sysbuild / non-sysbuild folders.
Let me know if that should be added as proposal here: https://github.com/zephyrproject-rtos/zephyr/issues/41640

It's also not clear to me how I would set a Kconfig option for the MCUboot image from the command line. --Dmcuboot_CONFIG_BOOT_BANNER=n, --DSB_CONFIG_BOOT_BANNER=n, or something else?

This is image specific setting, so that should be -Dmcuboot_CONFIG_BOOT_BANNER=n.
What is expected to go to the SB_CONFIG are high level settings that must be shared between images or needed by SYSBuild itself.
Such setting could be image signing, where keys needs to be distributed both to the bootloader (for verification) as well as application (for image signing).
Other settings that must be shared between images could be FPU (hard vs. soft)
This is one of the open areas to still work on after this PR, as we need to do more work on the Kconfig parts.
The plan for Kconfig is that if a setting is moved to SYSbuild Kconfig, it will still be present in the Kconfig for the samples (mcuboot is also a sample), but that SYSbuild will control the setting, and if user tries to manually control a setting directly in a sample, and not through SYSbuild then a proper warning should be raised.
But that infrastructure is outside the inital foundation work laid down in this PR.
Remember, all samples must still be buildable without SYSbuild.

For clarity I have a python script that builds releases with a few different options from dev images (disabling debug ports etc) that aren't placed into prj.conf (or sysbuild.conf in this case)

and that should only require small adjustments.
The main app still uses the same CONFIG_ names, and only mcuboot / other images will need adjustment to Kconfig names.

what does it do? Does it build systems?

it builds for a system.

A system may be a single image for a single device.
It may be multiple images for a single device / SoC.
It may be multiple images for a multi-core SoC.
It may even be multiple images for independent devices, see below.
Some of the systems above needs multiple images in order to function correctly.

SYSBuild can build multiple images for real different boards, and not just differently named boards due to multi core SoCs.
We can already handle a multi core SoC, like the nRF5340 or mps2_an521 (6676cd6), but one can also create a high level project with SYSBuild.
For example one could create a project utilizing SYSBuild to build an audio system consisting of a BLE audio gateway and a headset, each built and flashed to different devices.
In such a case, SYSBuild is building and flashing for a system, not just a multi-core SoC.
SYSBuild is capable of building a single image, although its advantage is building multiple images for a complete system.

multi image build

but SYSBuild can also build a single image. That's why I believe the name multi image is not proper.
It's correct that the reason for doing this work is to have the ability to build multiple images, which is also why multi image is written along with the name, SYSBuild.
But SYSBuild is not only multi image.

having sysbuild at the top will be the confusing, given this is not the standard way of building or doing things in zephyr, it is an add value to the build process, not a top level feature of the build system.

and I believe it has been stated that the location can be moved.
I just don't want to update location in this PR until it has been decided what this feature shall be named, and where it should be located.

@jfischer-no thanks for testing and providing feedback.

I played with it a little bit. I am very much in favor of calling it 'sysbuild', but then writing it consistently lowercase.

lowercase is fine by me.

What bothered me is that after west flash --domain foo, and west attach --domain foo, intuitive west build --domain foo does not work.

it's a good point, although it will not be possible to run CMake only on the domain.
CMake must be run on the sysbuild project, as that is where the knowledge of other images is known.
So west build must invoke CMake on sysbuild to generate the build files,but maybe it could be updated to only run cmake --build for the specified domain, when --domain is given. Would that be more aligned with your expectations @jfischer-no ?

Also Using CMake and ninja, cmake -B build -GNinja -DBOARD=reel_board -DAPP_DIR=samples/hello_world sysbuild, did not work for me.

because when using -D in CMake, you have no reference point in the file system.
When a user does -DVAR=file1.txt the VAR goes into the CMake cache, and once being processed inside a CMakeLists.txt there is no reference back to the directory where CMake was invoked.
So you must use absolute path in this case, like:
cmake -B build -GNinja -DBOARD=reel_board -DAPP_DIR=$PWD/samples/hello_world sysbuild`

I have problem testing this. I have checked out this branch and updated west, then I run, following the example:
west build -bnrf52840dk_nrf52840 -dsysbuild_test zephyr/samples/hello_world/ --sysbuild -- -DCONFIG_BOOTLOADER_MCUBOOT=y

with difference from example command line that I build outside of Zephyr dir to sysbuild_test dir and provide CONFIG_BOOTLOADER_MCUBOOT instead of CONFIG_WB_BOOTLOADER_MCUBOOT, because the second one cause error (non existing CONFIG).
As a result I have sysbuild_test dir with bootloader and hello_world dirs, but only hello_world gets build.
When I try to rerun the command I got:
$$ west build -bnrf52840dk_nrf52840 -dsysbuild_test zephyr/samples/hello_world/ --sysbuild -- -DCONFIG_BOOTLOADER_MCUBOOT=y
ERROR: Build directory "/repos/doer/zephyrproject-dirty/sysbuild_test" is for application "/repos/doer/zephyrproject-dirty/zephyr/sysbuild", but source directory "/repos/doer/zephyrproject-dirty/zephyr/samples/hello_world" was specified; please clean it, use --pristine, or use --build-dir to set another build directory
FATAL ERROR: refusing to proceed without --force due to above error

Now when I execute:
ninja -Csysbuild_test sysbuild_menuconfig

The menu does not show that MCUBOOT bootloader has been selected:

What bothered me is that after west flash --domain foo, and west attach --domain foo, intuitive west build --domain foo does not work.

it's a good point, although it will not be possible to run CMake only on the domain. CMake must be run on the sysbuild project, as that is where the knowledge of other images is known.
So west build must invoke CMake on sysbuild to generate the build files,but maybe it could be updated to only run cmake --build for the specified domain, when --domain is given. Would that be more aligned with your expectations @jfischer-no ?

Exact.

Also Using CMake and ninja, cmake -B build -GNinja -DBOARD=reel_board -DAPP_DIR=samples/hello_world sysbuild, did not work for me.

because when using -D in CMake, you have no reference point in the file system. When a user does -DVAR=file1.txt the VAR goes into the CMake cache, and once being processed inside a CMakeLists.txt there is no reference back to the directory where CMake was invoked.
So you must use absolute path in this case, like:
cmake -B build -GNinja -DBOARD=reel_board -DAPP_DIR=$PWD/samples/hello_world sysbuild`

Thanks, it works fine with absolute path. From then on, I found my way around.

multi image build
multi target build
advanced target build
integrated build solution
build composer (a la docker compose)
compose
build orchestrator
Fujiwhara (when two hurricanes collide)

Of all those names I still prefer sysbuild and I still think that it reflects what it does in the same way that sentences like "system-wide" do. But we're really talking about taste at this point. so I suggest we ask for opinions outside the current people who have made theirs known?

I have problem testing this. I have checked out this branch and updated west, then I run, following the example:

@de-nordic  because of name spaces you must use SB_CONFIG_BOOTLOADER_MCUBOOT=y, like this:
west build -bnrf52840dk_nrf52840 -dsysbuild_test zephyr/samples/hello_world/ --sysbuild -- -DSB_CONFIG_BOOTLOADER_MCUBOOT=y

See more here: https://github.com/zephyrproject-rtos/zephyr/blob/ba85e4cb39b1c5ba35bb0efdca5b5061e1d2a2fb/doc/guides/sysbuild/index.rst

@de-nordic have updated PR description with CONFIG_WB_BOOTLOADER_MCUBOOT --> SB_CONFIG_BOOTLOADER_MCUBOOT to reflect the changes introduced later.

So west build must invoke CMake on sysbuild to generate the build files,but maybe it could be updated to only run cmake --build for the specified domain, when --domain is given. Would that be more aligned with your expectations @jfischer-no ?

Exact.

done.

it builds for a system.

and what exactly have we been doing so far? building for a non-system?

@JordanYates @rerickson1 @galak  @keith-zephyr @danieldegrasse
Thank you for your interest and reviews so far.
We would like to get this in early in the 3.2 development cycle, so we would be very grateful for:

General reviews
Opinions on the name "sysbuild". See this and this comments for some context

Thanks!

it builds for a system.

and what exactly have we been doing so far? building for a non-system?

It's a convention, we are basically saying that a "system" is the set of firmware images required to run all the cores on a particular electronic device. Note that this is partially inspired by the upcoming System Devicetree specification.

+1 for sysbuild, it aligns with "system devicetree"

Agree, sysbuild is a fitting name.

Another vote for sysbuild.
I do agree with @nashif that I would prefer to avoid sysbuild being present in the root of the Zephyr tree. I see why there would be concerns about moving the entire folder into cmake however, and don't feel strongly opposed to the current location. Just think users may gravitate towards the folder named "sysbuild" when trying to understand Zephyr's core build structure, and become confused
Maybe we could move cmake specific parts of the build structure into the cmake folder, and place the KConfig options in appropriate files already in tree? The MCUBoot KConfig options could be placed into modules/Kconfig.mcuboot_bootutil, and I feel that the generic bootloader options could either be present at the root of the Zephyr tree, or perhaps we need a specific bootloader folder, since we already have MCUBoot specific options in the root Kconfig.zephyr folder (that I would argue should be moved elsewhere). I'm unsure if the above would be possible with the current sysbuild infrastructure, just some initial thoughts I had on possible file locations.

I do agree with @nashif that I would prefer to avoid sysbuild being present in the root of the Zephyr tree. I see why there would be concerns about moving the entire folder into cmake however, and don't feel strongly opposed to the current location. Just think users may gravitate towards the folder named "sysbuild" when trying to understand Zephyr's core build structure, and become confused

I was discussing this with @nashif as well earlier today. I personally think it's not a big issue, but since you are the second person raising it already this leads me to believe it will end up being a confusion point for some of our users. So here are the viable alternatives I can think of:

Simple one: just put everything under cmake/sysbuild. Not terribly elegant, but not horrendously dirty either
Clever one: Put everything under sys/build. This is cleverish, but might not be the best in terms of avoiding confusion
Complex one: Spread sysbuild around the tree like you suggest. I'd like to hear @tejlmand's opinion here

I do agree with @nashif that I would prefer to avoid sysbuild being present in the root of the Zephyr tree. I see why there would be concerns about moving the entire folder into cmake however, and don't feel strongly opposed to the current location.

@danieldegrasse  I'm aligned with you guys.
I don't feel the root of the repo is optimal, the problem I have is that I don't feel other options are good either.
So good suggestions with good reasoning are very welcome.
My feeling on some options:

cmake/sysbuild: The cmake folder contains CMake code used inside the Zephyr build system, but sysbuild is not just CMake code, it's technically a project which also includes Kconfig, and may even be extended with system devicetree in future.
So having this in cmake/systbuild feels wrong. It feels similar to place it under scripts/sysbuild
scripts/sysbuild: although sysbuild is similar in some ways to other tools (also complex tools like west extensions and kconfiglib) we have in Zephyr, then it's not just a script. It's technically a project of its own. Secondly, it would be first time we place cmake files in scripts. Not good option, imho.
samples/sysbuild: While sysbuild technically is a sample, then I don't think I need to explain why this is a bad and confusion option.

So good proposals with reasoning are very welcome.

Complex one: Spread sysbuild around the tree like you suggest. I'd like to hear @tejlmand's opinion here

that's a new one.
But even if code was placed various locations, then we still need an entry point for the sysbuild CMake project.
Where would that be ?
@carlescufi just curious if you have any thoughts here.

share/sysbuild is the best suggestion I've heard so far.

share/sysbuild is the best suggestion I've heard so far.

I am not completely sold on it but at this point it's the least bad of them all. +1 from me.

share/sysbuild is the best suggestion I've heard so far.

Agreed.  So far best I've seen/heard.

yeah, l am fine with share/ as well, although scripts/ would also do, because it is not only scripts we have in scripts/.
Now to the name, I am going to propose "Zephyr Compose" or compose for short (similar to Docker Compose).

it is a tool for defining and flashing multi-image Zephyr applications
system device tree should be completely orthogonal to this discussion IMO, we do not have that and this tool should be able to deal with situations that are beyond what is defined in devicetree.
It does not build any of the applications/images, this is done by the zephyr build system (cmake)
It is not the only way to build/compose an image for a system, you can build for a system only with cmake.

Tested this on smp_svr with set of commands:
west build -bnrf52840dk_nrf52840 -dsmp_svr_sysbuild/ zephyr/samples/subsys/mgmt/mcumgr/smp_svr/ --sysbuild  -- -Dsmp_svr_OVERLAY_CONFIG=${PWD}/zephyr/samples/subsys/mgmt/mcumgr/smp_svr/overlay-bt.conf -DSB_CONFIG_BOOTLOADER_MCUBOOT=y -Dmcuboot_CONFIG_ZEPHYR_TRY_MASS_ERASE=n
nrfjprog --eraseall
west flash -d smp_svr_sysbuild

Seems to work correctly:
$$ sudo ~/go/bin/mcumgr -t 1200 --conntype ble --connstring ctlr_name=hci0,peer_name='Zephyr' image list
Images:
 image=0 slot=0
    version: 0.0.0
    bootable: true
    flags: active confirmed
    hash: e612732a62b3341c05873959817654701b868c97eb687b5de828c4265aeb8e86
Split status: N/A (0)

I am just wondering whether this CMake warning shouldn't be suppressed:
...
-- Configuring done
-- Generating done
CMake Warning:                                                                                Manually-specified variables were not used by the project:

    smp_svr_OVERLAY_CONFIG

-- Build files have been written to: /repos/doer/zephyrproject-dirty/smp_svr_sysbuild/mcuboot
-- Configuring done
...

mcuboot is not going to use a variable out of it's domain, so this seems pointless.

@tejlmand - an increasingly common use case is where an MCU loads a bitstream into an FPGA. The MCU could be running Zephyr, but also a soft-core CPU in the bitstream could be running Zephyr as well. Note that the MCU and the soft core would both have distinct device trees and address spaces.
Am I understanding correctly that both images could be built from the same sysbuild invokation? Could you please elaborate somewhat or point to documentation on how that would work? It is immediately relevant for a project I have just started working on.

I am just wondering whether this CMake warning shouldn't be suppressed:
...
mcuboot is not going to use a variable out of it's domain, so this seems pointless.

@de-nordic  Thanks for spotting this.
That variable should never have been passed to mcuboot in the first placed.
I have done a minor rework on the variable passing to sub-builds and fixed this.
So the warning is now gone.

@tejlmand - an increasingly common use case is where an MCU loads a bitstream into an FPGA. The MCU could be running Zephyr, but a soft-core CPU on FPGA could be running Zephyr as well. Note that the MCU and the soft core would both have distinct device trees and address spaces, firmware images (not necessarily stored in bram), etc.

@cfriedt  Yes, this is a very common use-case.
And one that the sysbuild framework should support so that users only need to consider the actual stitching / uploading of firmware bitstream.

Am I understanding correctly that both images could be built from the same west --sysbuild invocation? Could you please elaborate somewhat or point to documentation on how that would work? It is immediately relevant for a project I have just started working on.

Yes, sysbuild can easily be used to build 2 or more Zephyr based images.
As a matter of fact, MCUboot is also a Zephyr image, so as soon as sysbuild is used to build an app with MCUboot, then two Zephyr images are actually built.
In your case, what is needed is to provide first image information about the second image, cause you need the first image to be able to upload to the FPGA.
We have three options here:

Tell the first images about the size and location of the second image.
a. First image can reserve space in the linker script and both images can be flashed independently
b. First image can reserve space in the linker script, but sysbuild can embed second image into the first image, for example using objcopy.
Second image is build and binary is converted to byte array, as header, c-source, or assembly.
First image then includes said byte array in its build.
Similar to what is done here: https://github.com/zephyrproject-rtos/zephyr/blob/main/modules/hal_rpi_pico/bootloader/CMakeLists.txt

The discussion here might also be of interest

https://github.com/zephyrproject-rtos/zephyr/pull/34835#pullrequestreview-867841580
https://github.com/zephyrproject-rtos/zephyr/pull/34835#discussion_r772425424

although I think 1. is cleanest as the Zephyr build system doesn't have to do anything, only reserve space, then I believe both ways must be possible and I believe there will be examples on how to achieve both ways.
rpi_pico is one of the use-cases for sysbuild, along with others.

Secondly, while I appreciate that a minimal framework to provide a "table of contents" for a multi-target image is helpful, I'm curious to know if there is perhaps an existing industry standard for multi-target image files that could be used. Are you aware of any?

zip files ? :laughing:
unfortunately not, we are moving into a territory where packaging firmware for a given product could contain not only MCU code, but as in your case embedded FPGA, not to mention firmware for DSPs, and even full-fledge processors (in wich case Zephyr might be just a subsystem and not center of the universe), so everyone seems to have their own way of doing things.
But it's a very good proposal, we should definitely think about packaging multi-target images, along with other important build by-products / build info.
I have added this as porposal here: https://github.com/zephyrproject-rtos/zephyr/issues/41640

I don't mean to be critical at all, just curious. I think this is a fantastic feature, so thank you for implementing it!

I value all good feedback, so thank you.

"Zephyr compose" is exactly wrong in my opinion.
The purpose of docker compose is to run multiple docker containers at once, with some extra metadata about how to hook them together. Note that in this tool, everything is a docker container.
This is different from this tool, which is used to produce build systems for a larger firmware system and tie them together. Some of them may be Zephyr, some not.
System devicetree is not orthogonal to this, either.

I have no problems with share/sysbuild. The only new idea I came up with is a new top level folder which build related functionality could be placed in. The best name I've come up with is build_system, as build is a non-starter given its current use.
build_system
    - cmake (formerly zephyr/cmake)
    - dts (formerly zephyr/scripts/dts)
    - kconfig (formerly zephyr/scripts/kconfig)
    - scripts (formerly zephyr/scripts/build)
    - sysbuild (or whatever the final name is)

Just throwing out another option, I do like that it keeps related functionality together.

Hi, I would like to know if it is possible to do a multi image build for custom boards?https://docs.zephyrproject.org/latest/develop/application/index.html#custom-board-definition
Please let me know what would be the configuration changes for the same.
I do not want to create a new directory zephyr/boards//my_custom_board. Please let me know if there are any efficient methods?

@tejlmand please go ahead and make the relevant changes.

Done.

@danieldegrasse @JordanYates @galak @carlocaione @pdunaj @sylvioalves @de-nordic @nvlsianpu @gmarull @rerickson1 @keith-zephyr @jfischer-no @nordicjm reviews welcome!

Some comments:
When configuring through cmake using cmake -GNinja -DBOARD=<board> /<path>/zephyr/share/sysbuild/ -DCONFIG_USE_SEGGER_RTT=n -DSB_CONFIG_BOOTLOADER_MCUBOOT=y -DAPP_DIR=/<path>/zephyr/samples/subsys/mgmt/mcumgr/smp_svr/ the following warning is emitted: ```
CMake Warning:
Ignoring extra path from command line:
"//zephyr/samples/subsys/mgmt/mcumgr/smp_svr/_build/y"

Building for `nrf5340dk_nrf5340_cpuapp_ns` gives the following failure (not sure if building for _ns should be supported, building for the secure core does work):

Memory region         Used Size  Region Size  %age Used
FLASH:       46292 B        64 KB     70.64%
SRAM:       22600 B       192 KB     11.49%
IDT_LIST:          0 GB         2 KB      0.00%
sign the payload
sign the payload
Traceback (most recent call last):
File "//zephyr/scripts/build/mergehex.py", line 28, in merge_hex_files
ih.merge(to_merge, overlap=overlap)
File "/home/jamie/.local/lib/python3.10/site-packages/intelhex/init.py", line 875, in merge
raise AddressOverlapError(
intelhex.AddressOverlapError: Data overlapped at address 0x0
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "//zephyr/scripts/build/mergehex.py", line 56, in
main()
File "//zephyr/scripts/build/mergehex.py", line 52, in main
merge_hex_files(args.output, args.input_files, args.overlap)
File "//zephyr/scripts/build/mergehex.py", line 30, in merge_hex_files
raise AddressOverlapError("{} has merge issues".format(hex_file_path))
intelhex.AddressOverlapError: //zephyr/samples/subsys/mgmt/mcumgr/smp_svr/_DEL/mcuboot/zephyr_ns_signed.hex has merge issues
ninja: build stopped: subcommand failed.
[13/16] Completed 'hello_world'

@greg-leach Unrelated to this change, seems the bl5340_dvk_cpuapp_ns board is broken as per:

In file included from //zephyr/modules/hal_nordic/nrfx/./nrfx_config.h:10,
from //modules/hal/nordic/nrfx/nrfx.h:37,
from //modules/hal/nordic/nrfx/drivers/include/nrfx_wdt.h:37,
from //bootloader/mcuboot/boot/zephyr/include/mcuboot_config/mcuboot_config.h:236,
from //bootloader/mcuboot/boot/bootutil/include/bootutil/fault_injection_hardening.h:57,
from //bootloader/mcuboot/boot/bootutil/include/bootutil/bootutil.h:32,
from //bootloader/mcuboot/boot/bootutil/src/swap_move.c:24:
//bootloader/mcuboot/boot/bootutil/src/swap_move.c: In function 'swap_status_source':
//zephyr/include/zephyr/devicetree/fixed-partitions.h:53:16: error: 'DT_COMPAT_fixed_partitions_LABEL_image_1_PARTITION_ID' undeclared (first use in this function); did you mean 'DT_COMPAT_fixed_partitions_LABEL_image_0_EXISTS'?
53 |         DT_CAT(DT_COMPAT_fixed_partitions_LABEL_, label)

@nordicjm thanks for testing this

When configuring through cmake using cmake -GNinja -DBOARD=<board> /<path>/zephyr/share/sysbuild/ -DCONFIG_USE_SEGGER_RTT=n -DSB_CONFIG_BOOTLOADER_MCUBOOT=y -DAPP_DIR=/<path>/zephyr/samples/subsys/mgmt/mcumgr/smp_svr/ the following warning is emitted:

I don't see this CMake error.
Are you sure you're not trying to re-run CMake for sysbuild in an old build folder ?
This is my output:
$ cmake -GNinja -DBOARD=nrf5340dk_nrf5340_cpuapp_ns share/sysbuild/ -DCONFIG_USE_SEGGER_RTT=n -DSB_CONFIG_BOOTLOADER_MCUBOOT=y -DAPP_DIR=(pwd)/samples/subsys/mgmt/mcumgr/smp_svr -Bbuild
Loading Zephyr module(s) (Zephyr repository): extensions sysbuild_extensions python west root zephyr_module boards shields sysbuild_kconfig
-- Found Python3: /usr/bin/python3.8 (found suitable exact version "3.8.10") found components: Interpreter
-- Cache files will be written to: /home/tora/.cache/zephyr
-- Found west (found suitable version "0.13.1", minimum required is "0.7.1")
-- Board: nrf5340dk_nrf5340_cpuapp_ns
Parsing /projects/github/ncs/zephyr/share/sysbuild/Kconfig
Loaded configuration '/projects/github/ncs/zephyr/build/empty.conf'
Merged configuration '/projects/github/ncs/zephyr/build/empty.conf'
Merged configuration '/projects/github/ncs/zephyr/build/zephyr/misc/generated/extra_kconfig_options.conf'
Merged configuration '/projects/github/ncs/zephyr/build/empty.conf'
Configuration saved to '/projects/github/ncs/zephyr/build/zephyr/.config'
Kconfig header saved to '/projects/github/ncs/zephyr/build/autoconf.h'
--
   *****************************
   * Running CMake for smp_svr *
   *****************************

Loading Zephyr default modules (Zephyr repository).
-- Application: /projects/github/ncs/zephyr/samples/subsys/mgmt/mcumgr/smp_svr
-- Found Python3: /usr/bin/python3.8 (found suitable exact version "3.8.10") found components: Interpreter
-- Cache files will be written to: /home/tora/.cache/zephyr
-- Zephyr version: 3.1.99 (/projects/github/ncs/zephyr)
-- Found west (found suitable version "0.13.1", minimum required is "0.7.1")
-- Board: nrf5340dk_nrf5340_cpuapp_ns
-- Found toolchain: gnuarmemb (/opt/gcc-arm-none-eabi-7-2018-q2-update)
-- Found BOARD.dts: /projects/github/ncs/zephyr/boards/arm/nrf5340dk_nrf5340/nrf5340dk_nrf5340_cpuapp_ns.dts
-- Generated zephyr.dts: /projects/github/ncs/zephyr/build/smp_svr/zephyr/zephyr.dts
-- Generated devicetree_unfixed.h: /projects/github/ncs/zephyr/build/smp_svr/zephyr/include/generated/devicetree_unfixed.h
-- Generated device_extern.h: /projects/github/ncs/zephyr/build/smp_svr/zephyr/include/generated/device_extern.h
-- Including generated dts.cmake file: /projects/github/ncs/zephyr/build/smp_svr/zephyr/dts.cmake
Parsing /projects/github/ncs/zephyr/Kconfig
Loaded configuration '/projects/github/ncs/zephyr/boards/arm/nrf5340dk_nrf5340/nrf5340dk_nrf5340_cpuapp_ns_defconfig'
Merged configuration '/projects/github/ncs/zephyr/samples/subsys/mgmt/mcumgr/smp_svr/prj.conf'
Merged configuration '/projects/github/ncs/zephyr/build/smp_svr/zephyr/misc/generated/extra_kconfig_options.conf'
Configuration saved to '/projects/github/ncs/zephyr/build/smp_svr/zephyr/.config'
Kconfig header saved to '/projects/github/ncs/zephyr/build/smp_svr/zephyr/include/generated/autoconf.h'
-- The C compiler identification is GNU 7.3.1
-- The CXX compiler identification is GNU 7.3.1
-- The ASM compiler identification is GNU
-- Found assembler: /opt/gcc-arm-none-eabi-7-2018-q2-update/bin/arm-none-eabi-gcc
-- Configuring done
-- Generating done
-- Build files have been written to: /projects/github/ncs/zephyr/build/smp_svr
--
   *****************************
   * Running CMake for mcuboot *
   *****************************

Loading Zephyr default modules (Zephyr workspace).
-- Application: /projects/github/ncs/bootloader/mcuboot/boot/zephyr
-- Found Python3: /usr/bin/python3.8 (found suitable exact version "3.8.10") found components: Interpreter
-- Cache files will be written to: /home/tora/.cache/zephyr
-- Zephyr version: 3.1.99 (/projects/github/ncs/zephyr)
-- Found west (found suitable version "0.13.1", minimum required is "0.7.1")
-- Board: nrf5340dk_nrf5340_cpuapp_ns
-- Found toolchain: gnuarmemb (/opt/gcc-arm-none-eabi-7-2018-q2-update)
-- Found BOARD.dts: /projects/github/ncs/zephyr/boards/arm/nrf5340dk_nrf5340/nrf5340dk_nrf5340_cpuapp_ns.dts
-- Found devicetree overlay: /projects/github/ncs/bootloader/mcuboot/boot/zephyr/dts.overlay
-- Generated zephyr.dts: /projects/github/ncs/zephyr/build/mcuboot/zephyr/zephyr.dts
-- Generated devicetree_unfixed.h: /projects/github/ncs/zephyr/build/mcuboot/zephyr/include/generated/devicetree_unfixed.h
-- Generated device_extern.h: /projects/github/ncs/zephyr/build/mcuboot/zephyr/include/generated/device_extern.h
-- Including generated dts.cmake file: /projects/github/ncs/zephyr/build/mcuboot/zephyr/dts.cmake
Parsing /projects/github/ncs/bootloader/mcuboot/boot/zephyr/Kconfig
Loaded configuration '/projects/github/ncs/zephyr/boards/arm/nrf5340dk_nrf5340/nrf5340dk_nrf5340_cpuapp_ns_defconfig'
Merged configuration '/projects/github/ncs/bootloader/mcuboot/boot/zephyr/prj.conf'
Merged configuration '/projects/github/ncs/zephyr/build/mcuboot/zephyr/misc/generated/extra_kconfig_options.conf'
Configuration saved to '/projects/github/ncs/zephyr/build/mcuboot/zephyr/.config'
Kconfig header saved to '/projects/github/ncs/zephyr/build/mcuboot/zephyr/include/generated/autoconf.h'
-- The C compiler identification is GNU 7.3.1
-- The CXX compiler identification is GNU 7.3.1
-- The ASM compiler identification is GNU
-- Found assembler: /opt/gcc-arm-none-eabi-7-2018-q2-update/bin/arm-none-eabi-gcc
MCUBoot bootloader key file: /projects/github/ncs/bootloader/mcuboot/root-rsa-2048.pem
-- Configuring done
-- Generating done
-- Build files have been written to: /projects/github/ncs/zephyr/build/mcuboot
-- Configuring done
-- Generating done
-- Build files have been written to: /projects/github/ncs/zephyr/build

Building smp_svr for nrf5340dk_nrf5340_cpuapp_ns gives the following failure (not sure if building for _ns should be supported, building for the secure core does work):

However, building hello_world for the board without mcuboot works fine.

Thanks for informing, will take a look.

Are you sure you're not trying to re-run CMake for sysbuild in an old build folder ?

Deleted the old build folders and made a new one and I still see it:
mkdir _new
cd _new/
cmake -GNinja -DBOARD=nrf52840dk_nrf52840 /tmp/bb/zephyr/share/sysbuild/ -DSB_CONFIG_BOOTLOADER_MCUBOOT=y -DAPP_DIR=/tmp/bb/zephyr/samples/subsys/mgmt/mcumgr/smp_svr/
Loading Zephyr module(s) (Zephyr base): extensions sysbuild_extensions python west root zephyr_module boards shields sysbuild_kconfig
-- Found Python3: /usr/bin/python3.10 (found suitable exact version "3.10.5") found components: Interpreter
-- Cache files will be written to: /home/jamie/.cache/zephyr
-- Found west (found suitable version "0.13.1", minimum required is "0.7.1")
-- Board: nrf52840dk_nrf52840
Parsing /tmp/bb/zephyr/share/sysbuild/Kconfig
Loaded configuration '/tmp/bb/zephyr/samples/subsys/mgmt/mcumgr/smp_svr/_new/empty.conf'
Merged configuration '/tmp/bb/zephyr/samples/subsys/mgmt/mcumgr/smp_svr/_new/empty.conf'
Merged configuration '/tmp/bb/zephyr/samples/subsys/mgmt/mcumgr/smp_svr/_new/zephyr/misc/generated/extra_kconfig_options.conf'
Merged configuration '/tmp/bb/zephyr/samples/subsys/mgmt/mcumgr/smp_svr/_new/empty.conf'
Configuration saved to '/tmp/bb/zephyr/samples/subsys/mgmt/mcumgr/smp_svr/_new/zephyr/.config'
Kconfig header saved to '/tmp/bb/zephyr/samples/subsys/mgmt/mcumgr/smp_svr/_new/autoconf.h'
--
   *****************************
   * Running CMake for smp_svr *
   *****************************

CMake Warning:
  Ignoring extra path from command line:

   "/tmp/bb/zephyr/samples/subsys/mgmt/mcumgr/smp_svr/_new/y"

Seems to be at share/sysbuild/cmake/modules/sysbuild_extensions.cmake line 142, debug output shows this: COMMAND /usr/bin/cmake      -GNinja      -DCONFIG_MCUBOOT_SIGNATURE_KEY_FILE:STRING="/tmp/bb/bootloader/mcuboot/root-rsa-2048.pem";y      -B/tmp/bb/zephyr/samples/subsys/mgmt/mcumgr/smp_svr/_new/build/smp_svr      -S/tmp/bb/zephyr/samples/subsys/mgmt/mcumgr/smp_svr    RESULT_VARIABLE   return_val    WORKING_DIRECTORY /tmp/bb/zephyr/samples/subsys/mgmt/mcumgr/smp_svr/_new/build
So it appears to be coming from the mcuboot signing key value?

Please send a follow-up commit adding --domain to the synopsis (not going to -1 so you don't have to get your +1s again)

Done: #48678

Hi, I still have a question on "sysbuild with custom board."
When I build application with "west build -p -b <custum_board>" the build goes thru without any issues. But when I try to build "west build -p -b <custum_board> --sysbuild -- -DSB_CONFIG_BOOTLOADER_MCUBOOT=n" the build is not able to find the custom board. Please let me know what are the configuration changes that I am missing for using sysbuild for custom board. Note: I do not want to add the custom board under zephyr/board/arm but I would like to maintain the architecture as shown in the attached image.

do not want to add the custom board under zephyr/board/arm but I would like to maintain the architecture as shown in the attached image.

@ram1kor are you perhaps setting a board root in your sample's CMakeLists.txt file like this:
set(BOARD_ROOT ${CMAKE_CURRENT_LIST_DIR})

If you are, then sysbuild will not be able to pick up your board root cause it doesn't parse the CMakeLists.txt file of the sample.
What you can do is telling sysbuild about your custom board root like this:
west build --sysbuild -b <your_board> <your_sample> -- -DBOARD_ROOT=<path-to-your-sample>

another option is to turn your repo into a Zephyr module and specify the boards root at the module.yml using board_root: <path>.
That will have a second benefit, as that will allow any of your samples or Zephyt test sample to be built against your custom board.
See more here: https://docs.zephyrproject.org/latest/develop/modules.html#build-settings

I've tried to build hello_world fot nucleo_f746zg board according to the instruction using the following command:
west build -b nucleo_f746zg samples/hello_world/ --sysbuild -- -DSB_CONFIG_BOOTLOADER_MCUBOOT=y
However the build complains about #include <st/f7/stm32f746zgtx-pinctrl.dtsi> not being found:
zephyr/boards/arm/nucleo_f746zg/nucleo_f746zg.dts:9:10: fatal error: st/f7/stm32f746zgtx-pinctrl.dtsi: No such file or directory
pinctrl.dtsi files comes with stm32-hal. It seems that the build system misses the include path pointing to stm32-hal.
In the very same time the traditional build (eg without --sysbuild option) works correctly.

@gkujawsk Please run west update


Finesse llmaindex feature

To create the best search, Postgresql vectordatabase, Azure AI search and llamaindex is to be explore. Currently llamaindex is ready to be deployed on Finesse and the expectation is to be able to swapped out between all three searches in debugging version of Finesse.
Tasks

[x] #35
[ ] Deploy llamaindex searching

Acceptance Criteria

llamaindex search functions and operates as expected

Well documented

This is an epic, moving this to finesse-frontend


path conflict bug

path/conflict/folder/. happens even when there are no folders with duplicate names?

Instead of trying to create directories before putting stuff in them I've just ended up uploading files since I found out it creates the directories automatically


[Question] This comes at the cost of load times, right?

Basically the title, but I have another question: does the answer change if I have lazy dfu installed?

sure, but there other info you should know, smooth boot (reloaded) by its default configuration its makes the game smoother while increasing loading time speed, but you can change the configuration about your weldments, increasing threads count and priority will improve the loading time speed, for more info about the configuration file, press here.
I hope you have a good day!


why not work with this situation

[ x] I have looked at the documentation here first?
[x ] I have looked at the examples provided that may showcase my question here?

Package version eg. v9, v10:
v10
Issue, Question or Enhancement:
[SignUpParam]         work      {"name":"q1mi","age":18,"email":"123.com"}
[Array]        not work    {"params": [{"name":"q1mi","age":18,"email":"123.com"}]}
Code sample, to showcase or reproduce:
type Array struct {
	Params []SignUpParam `json:"params" binding:""`
}

type SignUpParam struct {
	Age        uint8  `json:"age" binding:"gte=1,lte=130"`
	Name       string `json:"name" binding:"required"`
	Email      string `json:"email" binding:"required,email"`
	Password   string `json:"password" binding:"required"`
	RePassword string `json:"re_password" binding:"required,eqfield=Password"`
}

Validator does not automatically traverse arrays or maps. You want to use the dive Tag to do that.
please take a look at the document.

thanks, bro


Heatmap height too small in docs

https://github.com/frappe/charts/blob/4bb44c0914125e4b511592623870946bdac3f0a8/docs/index.html#L218
It’s hiding the seventh day each week. Should be 115px.

Send in a PR!

@jakob-stoeck The PR for this small change would be as easy as editing the file on GitHub itself :)

I came here to mention this, the 115px height fixed it.
On a possibly related note, I don't appear to see today, or any days since the 27th, it appears to be the same on https://frappe.github.io/charts/

Also, it'd be good to see some actual data behind the heatmap on https://frappe.github.io/charts/ as it's the only chart that has no data, despite the heatmap_data variable being populated in your index.js file. I'm struggling to get it to show data, but it seems this way on your site too?


Disable multiple instances

Hi,
thanks for this awesome app!
I'd just like to suggest that you add an option to allow only one instance of the application.

Implemented by @MatheusOliveira-dev and merged in #44.


Store state to merkle trie and set hash root back to block header

Tree: use AVL trie (other option: Merkle Patricia trie used by Ethereum)
Codec: MsgPack. This is for quick (and somewhat dirty), non-type encoding. Other options: Google's protobuf3, Tendermint's Amino, Mongodb's BSON, Ethereum's RLP

Will use LevelDB.


Push reader functionality to callback.

Currently callback and reader handle much the same functionality but need to be used much differently. Ideally it should not matter whether you have access to a callback or a Reader, any viz or analysis should work exactly the same.

[ ] New log file format, opt for more compressed(eg obj) with ability to selectively read in sections eg just results and only pull specific pieces from info when needed. Log files need to be easily and safely shareable.
[ ] Make way to process results from multiple callbacks same way reader does log files.
[ ] Make reader template ExperimentCallback and both have exactly the same interface.
[ ] Allow callback to store info on file so all info dicts dont need be in memory.
[ ] - For future log file just being pickled callback, I really could easily do this via the pickle customizations w/ getstate and setstate

Closed until further development of training platform, main focus is now snn framework.


Schema compare of two objects showing more differences than expected

Steps to Reproduce:

Create a project with one table
Create an empty project
Schema compare the two projects in ADS

expected: One difference of the table
actual: 7 differences
Comparing the same projects in SSDT shows only the table difference, as expected

Could be related to the customer SQL types dll used for the .net core DacFx since it doesn't target .net core. There's recently been a version released targeting net standard, so we can try using that to test (not an official release). https://www.nuget.org/packages/Microsoft.SqlServer.Types/160.600.9-ctp2p0

only happens when comparing to an empty project. When comparing to the dacpac built from the empty project, the comparison is as expected:
one table project compared to empty project's dacpac

one table project compared to empty project


Move registering Tepp to function file

Redesign so that each function can register its own Tepp in its own file, rather than having to update everything in one central location.

Added function to do the registering
Implemented in Get-DbaBackupHistory as an example

Sorry, but in order to make that happen I had to reshuffle the loading order in the psm1

Thank you!


fix: #254 correct property and fix documentation

Codecov Report

Merging #258 (03e9d41) into master (6abc1f1) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #258   +/-   ##
=========================================
  Coverage     48.75%   48.75%
  Complexity      461      461
=========================================
  Files            46       46
  Lines          2322     2322
  Branches        290      290
=========================================
  Hits           1132     1132
  Misses         1009     1009
  Partials        181      181

Impacted Files
Coverage Δ

...nity/rest/config/FeignErrorDecoderConfiguration.kt
60.00% <ø> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Resources not removed when changing/deleting from manifest

I noticed that when the OU is changes for an SCP or a CFN resource, that SCP or CFN resource is added to the 'new' OU, but not removed from the old one.
Also, when deleting an SCP or CFN resource, that does not appear to delete it.

@dustnic The current version does not support the deletion feature at the stack set level. If an OU, account or region is removed from the CFN resource the pipeline should delete the stack instances.

hi Lalit. point taken re: not supporting deletion feature at stack set level. I tested by changing the list of OUs a resource was deployed to from (A,B) to (A,C) the resource was deployed to C, but not removed from B

So what I do is comment out the OU or account and that removes the stack. However, there has to be at least one left in the section and thus, it can't be deleted.
@groverlalit  I suggest using the "deploy_method = remove" in a future version. That would be great. Thoughts?

Is this on the roadmap?

I also believe it should be optional to select retain policy in manifest for each stack. I causes some misconception that a manual deletion procedure is neccesary when removing stacks from manifest file. In my understanding it goes against the IaC idea when you have to make sure the stacks  is in sync with the Stacksets.

I also believe it should be optional to select retain policy in manifest for each stack. I causes some misconception that a manual deletion procedure is neccesary when removing stacks from manifest file. In my understanding it goes against the IaC idea when you have to make sure the stacks  is in sync with the Stacksets.

Hi all - This feature is on our roadmap and will be evaluated a future release. Thanks!

I agree this issue is very problematic, counter-intuitive and against general IaC principals. If you add an account or OU to the manifest file and an SCP or CFN template is applied to it, removing the account or OU from the manifest file should cause those resources to be removed. This is how, for example, Terraform or Chef handle things and maintain idempotency: the diff from the existing state is always applied. This issue forces us to manually go in and remove resources that were previously applied with this stack, and at scale this can be extremely time-consuming and error-prone.

It has been almost two years since this was reported.  Do you have any updates on the progress?  It's a rudimentary feature without which this solution isn't very usable beyond the initial set up.

Yeah this is really bad. Every time we remove an account from our Control Tower the pipeline subsequently fails and we have to go in and manually remove the Stack Set instances for the account that no longer exists. In retrospect we should have used Terraform from the beginning.

agreed we have the same problem.  its frustrating, I even created a script
to remove them from all our stacksets just for this, but I should not have
had to do that.
Randy Spainhower
Cloud Core Team (Team : @.***)
801.641.3572
On Wed, Mar 9, 2022 at 12:36 PM Speculor @.***> wrote:

Yeah this is really bad. Every time we remove an account from our Control
Tower the pipeline subsequently fails and we have to go in and manually
remove the Stack Set instances for the account that no longer exists. In
retrospect we should have used Terraform from the beginning.
—
Reply to this email directly, view it on GitHub
https://github.com/aws-solutions/aws-control-tower-customizations/issues/24#issuecomment-1063286888,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ASH5MXJJFVOEU67JMHLRYTLU7D4R7ANCNFSM4P4XHG7Q
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you are subscribed to this thread.Message
ID: <aws-solutions/aws-control-tower-customizations/issues/24/1063286888@
github.com>

Is this on the roadmap? This is a huge limitation and should be mentioned in the docs.

Also a targeted time frame would be good. Some other partners have already built their own customization solutions because deleting objects cleanly is not supported.

It has been 2 years since this was created. This is fundamental functionality.

Hi @groverlalit
Any plans to implement this anytime soon?

Thanks for the feedback, we understand the pain point here. This is on our roadmap for medium-term implementation, we will circle back with an update as we get closer to deploying a solution.

It has been almost two years since this was reported. Do you have any updates on the progress? It's a rudimentary feature without which this solution isn't very usable beyond the initial set up.

Check out my method:  https://github.com/aws-solutions/aws-control-tower-customizations/issues/24#issuecomment-695317733

We've just published v2.5.0 containing an opt-in flag to enable Stack Set resource deletion.

MANY THANKS GUYS!!!
YOU'RE THE BEST!!!

My apology for the late update on this.
Tested on version v2.5.1 and flag enable_stack_set_deletion: true works as described for the StackSet resources which is great!
This config parameter name also suggests that it was only meant for StackSets and according to my testing Service Control Policies (SCP) type of resources are still missing similar feature. That would be great if similar feature would be there for SCPs as well!
But anyways, many thanks to you guys from University of New South Wales for getting it done!

What about SCPs? Any plans to update that?


Removing PSPs until 1.8

This change will certainly disable the PSPs but will also leave some other cruft from enabling them in the first place.  I think the cleaner path is to remove the PSP by default commit altogether, esp if we re-able later.


https是否过期？

hi all.
https是否过期?
Fetching source index from https://ruby.taobao.org/
Retrying fetcher due to error (2/4): Bundler::Fetcher::CertificateFailureError Could not verify the SSL certificate for https://ruby.taobao.org/.
There is a chance you are experiencing a man-in-the-middle attack, but most likely your system doesn't have the CA certificates needed for verification. For information about OpenSSL certificates, see http://bit.ly/ruby-ssl. To connect without using SSL, edit your Gemfile sources and change 'https' to 'http'.
Retrying fetcher due to error (3/4): Bundler::Fetcher::CertificateFailureError Could not verify the SSL certificate for https://ruby.taobao.org/.
There is a chance you are experiencing a man-in-the-middle attack, but most likely your system doesn't have the CA certificates needed for verification. For information about OpenSSL certificates, see http://bit.ly/ruby-ssl. To connect without using SSL, edit your Gemfile sources and change 'https' to 'http'.
Retrying fetcher due to error (4/4): Bundler::Fetcher::CertificateFailureError Could not verify the SSL certificate for https://ruby.taobao.org/.
There is a chance you are experiencing a man-in-the-middle attack, but most likely your system doesn't have the CA certificates needed for verification. For information about OpenSSL certificates, see http://bit.ly/ruby-ssl. To connect without using SSL, edit your Gemfile sources and change 'https' to 'http'.
Could not verify the SSL certificate for https://ruby.taobao.org/.
There is a chance you are experiencing a man-in-the-middle attack, but most likely your system doesn't have the CA certificates needed for
verification. For information about OpenSSL certificates, see http://bit.ly/ruby-ssl. To connect without using SSL, edit your Gemfile
sources and change 'https' to 'http'.

相同的问题，证书已经过期了没法更新

相同问题，求尽快解决

问题已经解决

Thanks! cool.

应该是同样的原因

网上已经有人碰到与楼上同样的问题了。

还是存在证书问题，更新openssl证书也没解决。

现在好像还没解决

修改下 Bundler 的 Gem 源代码镜像，执行如下命令即可
$ bundle config mirror.https://rubygems.org https://gems.ruby-china.org

RubyGems 镜像的管理已经交由 Ruby China 负责了，需要切换下原先的 source
$ gem sources --add https://gems.ruby-china.org/ --remove https://rubygems.org/

执行完，校验下 source 是否切换过来了，就可以了
$ gem sources -l
*** CURRENT SOURCES ***

https://gems.ruby-china.org

当前应该使用
bundle config mirror.https://rubygems.org https://gems.ruby-china.com


chore: build process to export in commonjs

Does this make it only commonjs?  are we still supporting both?

No, this does not make it only CommonJS. By configuring tsc to output CommonJS, the library still supports both module systems. The PR simplifies the build process by removing Webpack and relying solely on tsc to generate a CommonJS package, ensuring compatibility. This change also avoids issues with dependencies like async_hooks, which rely on require and can cause errors when bundled for ESM. We have tested the library using npm link in both CommonJS and ESM projects, with and without TypeScript.


Dark Mode

This PR add button in navbar that can change to dark mode using inverted filter CSS
resolve #6124

Hello, your PR should be targeting Defectdojo's dev branch instead

@salvianf We only take PRs against the dev branch - see here
Also, there's something not right with this PR since its touches 300+ files.  It's likely better to just close it and resubmit only your changes against the dev branch.

Closing due to #7212


ValueError: BitVects must be same length (rdkit)

I am calculating the structure similarity profile between 2 moles using rdkit. When I am running the program in google colab (rdkit=2020.09.2 python=3.7) the program is working fine.
I am getting an error when I am running on my PC (rdkit=2021.03.2 python=3.8.5). The error is a bit strange. The dataframe contains 500 rows and the code is working only for the first 10 rows (0-9) and for later rows I am getting an error
 s = DataStructs.BulkTanimotoSimilarity(fps_2[n], fps_2[n+1:])
    ValueError: BitVects must be same length

The block of code is given below
  data = pd.read_csv(os.path.join(os.path.join(os.getcwd(), "dataset"), "test_ssp.csv"), index_col=None)

  #Proff and make a list of Smiles and id
  c_smiles = []
  count = 0
  for index, row in data.iterrows():
    try:
      cs = Chem.CanonSmiles(row['SMILES'])
      c_smiles.append([row['ID_Name'], cs])
    except:
      count = count + 1
      print('Count Invalid SMILES:', count, row['ID_Name'], row['SMILES'])

  # make a list of id, smiles, and mols
  ms = []
  df = DataFrame(c_smiles,columns=['ID_Name','SMILES'])
  for index, row in df.iterrows():
    mol = Chem.MolFromSmiles(row['SMILES'])
    ms.append([row['ID_Name'], row['SMILES'], mol])

  # make a list of id, smiles, mols, and fingerprints (fp)
  fps = []
  df_fps = DataFrame(ms,columns=['ID_Name','SMILES', 'mol'])
  df_fps.head

  for index, row in df_fps.iterrows():
    fps_cal = FingerprintMols.FingerprintMol(row['mol'])
    fps.append([row['ID_Name'], fps_cal])

  # the list for the dataframe
  qu, ta, sim = [], [], []

  fps_2 = DataFrame(fps,columns=['ID_Name','fps'])
  fps_2 = fps_2[fps_2.columns[1]]
  fps_2 = fps_2.values.tolist()

  c_smiles2 = DataFrame(fps,columns=['ID_Name','fps'])
  c_smiles2 = c_smiles2[c_smiles2.columns[0]]
  c_smiles2 = c_smiles2.values.tolist()

  # compare all fp pairwise without duplicates
  for n in range(len(fps_2)):
      s = DataStructs.BulkTanimotoSimilarity(fps_2[n], fps_2[n+1:])
      for m in range(len(s)):
          qu.append(c_smiles2[n])
          ta.append(c_smiles2[n+1:][m])
          sim.append(s[m])
  print(len(qu))
  print(len(ta))
  print(len(sim))

Can you tell me why I am getting this error on my PC while the code is working fine in Google Colab? How can I solve the issue? Is there anyway to install rdkit=2020.09.2?
Reproducible Data
DB00607	[H][C@]12SC(C)(C)[C@@H](N1C(=O)[C@H]2NC(=O)C1=C(OCC)C=CC2=CC=CC=C12)C(O)=O
DB01059	CCN1C=C(C(O)=O)C(=O)C2=CC(F)=C(C=C12)N1CCNCC1
DB09128	O=C1NC2=CC(OCCCCN3CCN(CC3)C3=C4C=CSC4=CC=C3)=CC=C2C=C1
DB04908	FC(F)(F)C1=CC(=CC=C1)N1CCN(CCN2C(=O)NC3=CC=CC=C23)CC1
DB09083	COC1=C(OC)C=C2[C@@H](CN(C)CCCN3CCC4=CC(OC)=C(OC)C=C4CC3=O)CC2=C1
DB08820	CC(C)(C)C1=CC(=C(O)C=C1NC(=O)C1=CNC2=CC=CC=C2C1=O)C(C)(C)C
DB08815	[H][C@@]12[C@H]3CC[C@H](C3)[C@]1([H])C(=O)N(C[C@@H]1CCCC[C@H]1CN1CCN(CC1)C1=NSC3=CC=CC=C13)C2=O
DB09143	[H][C@]1(C)CN(C[C@@]([H])(C)O1)C1=CC=C(NC(=O)C2=CC=CC(=C2C)C2=CC=C(OC(F)(F)F)C=C2)C=N1
DB06237	COC1=C(Cl)C=C(CNC2=C(C=NC(=N2)N2CCC[C@H]2CO)C(=O)NCC2=NC=CC=N2)C=C1
DB01166	O=C1CCC2=C(N1)C=CC(OCCCCC1=NN=NN1C1CCCCC1)=C2
DB00813	CCC(=O)N(C1CCN(CCC2=CC=CC=C2)CC1)C1=CC=CC=C1

Configuration (please complete the following information):
RDKit version: 2021.03.2
OS (not relevant): Ubuntu 20.04
Python version (not relevant):3.8.5
Are you using conda? No / pip install rdkit-pypi from this link

I mentioned a temporary fix here. Apart from that it definitely looks like a bug. A shorter way to reproduce the error is
from rdkit.Chem import MolFromSmiles
from rdkit.Chem.Fingerprints.FingerprintMols import FingerprintMol

mol1 = MolFromSmiles("CCC(=O)N(C1CCN(CCC2=CC=CC=C2)CC1)C1=CC=CC=C1")
fp1 = FingerprintMol(mol1).GetNumBits()

mol2 = MolFromSmiles("O=C1CCC2=C(N1)C=CC(OCCCCC1=NN=NN1C1CCCCC1)=C2")
fp2 = FingerprintMol(mol2).GetNumBits()

print(fp1, fp2)
# 1024, 2048

This is the intended behavior. FingerprintMol is poorly documented but, by default, folds the fingerprints it calculates to a target density of 0.3
Since I guess that's not what you want, I would suggest using the fingerprinting function you want to use directly.
There's documentation about doing that here:
https://rdkit.org/docs/GettingStartedInPython.html#fingerprinting-and-molecular-similarity

Ohh okay. But then isn't it breaking the BulkTanimotoSimilarity function like in above case? It works fine in the older versions.

BulkTanimotoSimilarity assumes that the fingerprints are the same length. So, yes, the output of FingerprintMol won't necessarily work there. The fact that it worked for you in an earlier version is an accident - no fingeprints ended up being folded.
As I said: if you don't want FingerprintMols defaults behavior, you should either change the arguments you're passing so that it does what you want (unfortunately you need to read the code to figure out how to do this) or just call the fingerprinting functions directly. I recommend the second approach.


Added a quick fix for vertical demo.

Reverted to EMMO version  1.0.0-alpha2.
The demo needs a throughout cleanup - this is just a quick fix such that the vertical demo runs...
Relates to issue #11 and #66

Hmm, the pytest fails. Would like to fix that before merge.


Player's volume level won't reflect the actual level

What version of Music Assistant has the issue?
2.0.0b133
What version of the Home Assistant Integration have you got installed?
n/a
Have you tried everything in the Troubleshooting FAQ and reviewed the Open and Closed Issues and Discussions to resolve this yourself?

[X] Yes

The problem
If volume levels are changed from within snapweb or any other interface, Music assistant does not show the new value causing confusion, this also happens from within Musica assistant and changing the volume level of any snapcast group player.
How to reproduce
Change the volume leve of any group player or change the volume level from within snapweb .
Music Providers
Spotify
Player Providers
Snapcast
Full log output
nothing in the log relevant.
Additional information
No response
What version of Home Assistant Core are your running
n/a
What type of installation are you running?
Home Assistant OS
On what type of hardware are you running?
Alternative

Again you are not providing enough information. Provide a step by step of what you are doing to get this behaviour. Its not clear if you are talking about grouped players, individual players or both? I have tried a grouped player and there are some problems there but we need you to be specific about the problem that you are observing.

In the video it can be seen how after interacting with the group player volume bar does not update the players that are linked to it and viceversa.  this also happens when interacting with any other inteface (Snapweb) . If the volume is to any of the snapclients in snapweb, music assistant won't display the new levels unless the web interface is refreshed.
https://github.com/music-assistant/hass-music-assistant/assets/9822111/5569bc43-76e1-4f03-abca-17a721d4b8fa

This is hopefully fixed in the next update. Please advise if not.

@ronluna we will close this soon assuming fixed


Change undefined function error message

Purpose
$subject for undefined langlib function and record function pointer field
Fixes #16321
Check List

[ ] Read the Contributing Guide
[ ] Updated Change Log
[ ] Checked Tooling Support (#)
[x] Added necessary tests

[x] Unit Tests
[ ] Spec Conformance Tests
[ ] Integration Tests
[ ] Ballerina By Example Tests

[ ] Increased Test Coverage
[ ] Added necessary documentation

[ ] API documentation
[ ] Module documentation in Module.md files
[ ] Ballerina By Examples

Codecov Report

Merging #22188 into master will not change coverage by %.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #22188   +/-   ##
=======================================
  Coverage   14.59%   14.59%
=======================================
  Files          51       51
  Lines        1411     1411
  Branches      219      219
=======================================
  Hits          206      206
  Misses       1189     1189
  Partials       16       16

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6ec986b...eccb79d. Read the comment docs.

Tried the following program:
public function main() {
    foo();

    Foo f = {};
    f.recFoo();

    Bar b = new;
    b.objBar();

    Baz bz = 100;
    bz.baz();
}

type Foo record {|
|};

type Bar object {
};

type Baz int|string;

Gives the following errors:
error: .::16321.bal:2:5: undefined function 'foo'
error: .::16321.bal:5:7: undefined function 'recFoo' in record 'Foo'
error: .::16321.bal:8:7: undefined function 'objBar' in object 'Bar'
error: .::16321.bal:11:8: undefined function 'baz' in type '(int|string)'

Didn't we agree to change the error message for records in https://ballerina-platform.slack.com/archives/CAXS256RX/p1585129408018000?thread_ts=1585122318.012200&cid=CAXS256RX


uv build --wheel doesn't seem to be installing dev-dependencies

https://github.com/pyca/cryptography/pull/11558 shows trying to take advantage of uv build, however CI doesn't seem happy: https://github.com/pyca/cryptography/actions/runs/10748158349/job/29811544879?pr=11558
Notably, when running uv build -v --wheel --require-hashes --build-constraint=$BUILD_REQUIREMENTS_PATH cryptography*.tar.gz $PY_LIMITED_API -o wheelhouse/:
DEBUG uv 0.4.7
DEBUG Searching for Python interpreter in managed installations, system path, or `py` launcher
DEBUG Searching for managed installations at `C:\Users\runneradmin\AppData\Roaming\uv\python`
DEBUG Found `cpython-3.11.9-windows-x86_64-none` at `C:\hostedtoolcache\windows\Python\3.11.9\x64\python3.exe` (search path)
DEBUG Using request timeout of 30s
Building wheel from source distribution...
DEBUG Calling `maturin.build_wheel("D:\\a\\cryptography\\cryptography\\wheelhouse\\.tmpNob9dO", {"build-args":"--features=pyo3/abi3-py39"}, None)`
Traceback (most recent call last):
  File "<string>", line 8, in <module>
ModuleNotFoundError: No module named 'maturin'
error: Build backend failed to build wheel through `build_wheel()` with exit code: 1

For whatever reason, uv doesn't seem to actually install maturin.
When running this same command locally (on macOS), I see output like the following:
(cryptography) ~/p/cryptography ❯❯❯ uv build -v --wheel --require-hashes --build-constraint=.github/requirements/build-requirements.txt dist/cryptography-44.0.0.dev1.tar.gz -o wheelhouse
DEBUG uv 0.4.6 (Homebrew 2024-09-05)
DEBUG Found workspace root: `/Users/alex_gaynor/projects/cryptography`
DEBUG Adding current workspace member: `/Users/alex_gaynor/projects/cryptography`
DEBUG Searching for Python >=3.7 in managed installations or system path
DEBUG Found `cpython-3.12.5-macos-aarch64-none` at `/Users/alex_gaynor/.virtualenvs/cryptography/bin/python3` (active virtual environment)
DEBUG Using request timeout of 30s
Building wheel from source distribution...
DEBUG Ignoring empty directory
DEBUG Solving with installed Python version: 3.12.5
DEBUG Solving with target Python version: >=3.12.5
DEBUG Adding direct dependency: maturin>=1, <2
DEBUG Adding direct dependency: maturin==1.7.1
DEBUG Adding direct dependency: cffi{platform_python_implementation != 'PyPy'}>=1.12
DEBUG Adding direct dependency: cffi{platform_python_implementation != 'PyPy'}==1.17.1
DEBUG Adding direct dependency: setuptools<74.0.0 | >74.0.0, <74.1.0 | >74.1.0, <74.1.1 | >74.1.1, <74.1.2 | >74.1.2
DEBUG Adding direct dependency: setuptools==73.0.1
[...]

Happy to test anything that'd be useful.

Very strange, let me think about what could be happening here...

Do you have --no-build-isolation set anywhere, like in a pyproject.toml somewhere? (You could run uv build --show-settings to verify.)

(Poking around here: https://github.com/charliermarsh/cryptography/pull/1)

Definitely not intentionally disabled! Thanks for taking a look, let me know how I can help debug this.

It looks like the builder thinks build isolation is enabled (despite --show-settings not indicating as such) so something's going wrong...

AHHHHH, --no-build-isolation is in the PY_LIMITED_API variable. I assume that's causing this.

No problem at all! Glad it's not a bug at least. Though confused why --show-settings didn't make this clear... Maybe a bug there? I'll add some more logging for this regardless (and close this issue with that PR).

Oh sorry... Dumb of me. I'm running --show-settings in the first command which doesn't have the env var attached. I will add more logging though.


V2 : Negotiation should consider 2 countires mutual relationship - for negatives

Implemented https://github.com/patsonluk/airline/commit/eda15a580071ca6229c7c06780b755b9d6fb4634

Implemented https://github.com/patsonluk/airline/commit/eda15a580071ca6229c7c06780b755b9d6fb4634


'Tier' filter include results of deleted assets

Affected module
UI
Describe the bug
When using 'tier' filter, the number shows the results+deleted results, which is confusing for the user
To Reproduce

Expected behavior
The filter should count the amount on valid assets only, without deleted ones.
Version:

OpenMetadata version: 1.3.2

Additional context
Add any other context about the problem here.

@karanh37 this should be a UI fix to pass deleted=false cc @sonika-shah


[SPARK-23938][SQL] Add map_zip_with function

What changes were proposed in this pull request?
This PR adds a new SQL function called map_zip_with. It merges the two given maps into a single map by applying function to the pair of values with the same key.
How was this patch tested?
Added new tests into:

DataFrameFunctionsSuite.scala
HigherOrderFunctionsSuite.scala

cc @ueshin @mgaido91 @hvanhovell

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Test build #94325 has started for PR 22017 at commit ef56011.

Test build #94325 has finished for PR 22017 at commit ef56011.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ArrayDataMerger(elementType: DataType)
case class ArrayUnion(left: Expression, right: Expression) extends ArraySetLike
abstract class GetMapValueUtil extends BinaryExpression with ImplicitCastInputTypes
case class MapZipWith(left: Expression, right: Expression, function: Expression)

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94325/
Test PASSed.

Test build #94362 has started for PR 22017 at commit ec583eb.

Test build #94362 has finished for PR 22017 at commit ec583eb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ArrayDataUnion(elementType: DataType) extends ((ArrayData, ArrayData) => ArrayData)
case class ArrayUnion(left: Expression, right: Expression) extends ArraySetLike

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94362/
Test FAILed.

Test build #94393 has started for PR 22017 at commit 89a3da4.

Test build #94394 has started for PR 22017 at commit 12ad8b2.

Test build #94394 has finished for PR 22017 at commit 12ad8b2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94394/
Test PASSed.

Test build #94393 has finished for PR 22017 at commit 89a3da4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94393/
Test PASSed.

Test build #94458 has started for PR 22017 at commit 38ce4e7.

Test build #94458 has finished for PR 22017 at commit 38ce4e7.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94458/
Test FAILed.

Test build #94508 has started for PR 22017 at commit 5d2a78e.

Test build #94508 has finished for PR 22017 at commit 5d2a78e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94508/
Test PASSed.

Test build #94526 has finished for PR 22017 at commit 3c849cb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94526/
Test PASSed.

LGTM.
@mgaido91 Do you have any other comments on this?

Test build #94570 has started for PR 22017 at commit 595161f.

Test build #94570 has finished for PR 22017 at commit 595161f.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94570/
Test FAILed.

retest this please

Test build #94585 has started for PR 22017 at commit 595161f.

Test build #94585 has finished for PR 22017 at commit 595161f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94585/
Test FAILed.

retest this please

Test build #94685 has started for PR 22017 at commit 595161f.

Test build #94685 has finished for PR 22017 at commit 595161f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94685/
Test PASSed.

Test build #94712 has started for PR 22017 at commit 2b7e9e5.

Test build #94712 has finished for PR 22017 at commit 2b7e9e5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94712/
Test PASSed.

Test build #94737 has started for PR 22017 at commit bcd4e0f.

Test build #94737 has finished for PR 22017 at commit bcd4e0f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94737/
Test PASSed.

Thanks! merging to master.


Fix getStats hyphen shim regression in pre-48 FF.

Description
Fix TypeError: "type" is read-only error in pre-48 Firefox (regression from https://github.com/webrtc/adapter/pull/392).
Purpose
Make getStats work again in pre-48 Firefox.

Was this caught by tests?

the tests for ESR are in a miserable state... due to marionette stuff :-/

aha right....

LGTM


OpenVR latency is worse than OSC

This is a companion issue to SlimeVR/SlimeVR-OpenVR-Driver#34, which tracks how we can improve latency in the openvr driver.

OpenVR: messages are sent to a queue which is polled at 200-ish hz from another thread

(I linked to the windows named pipes version, but the unix socket version works the same way)
This probably accounts for some of the latency I measured using https://github.com/kitlith/slimevr-latency (which ranged from 6ms to 12ms on my machine, though I should set something up to aggregate and do statistics to it.)

OSC: messages are sent immediately, barring any complications inside the OSC library that I've not investigated.

This probably causes at least a few ms (and possibly mildly inconsistent) latency for openvr that is not present for OSC. We might be able to improve this by removing the thread, and try to use non-blocking IO to do all the network/IPC in parallel. Otherwise, we could schedule the thread to wake when we push data to the queue instead of having it poll at a fixed 200-ish hz. (the latter is probably a quicker immediate fix, the former may require changing server architecture. shrug)

I have confirmed that the thread polling at 200hz is responsible for some extra latency by replacing the Thread.sleep(5) with a Thread.yield() locally. This reduces latency down to 0.9-3 ms in most cases, with the occasional outlier of >10ms (probably due to thread scheduling?)
I tried setting it up so that instead of using Thread.sleep(5), we block on the outgoing message queue for up to 5ms. Unfortunately, that seems to either do nothing, or increase the latency we experience.

I've updated my latency measuring tool to also listen to OSC. Can confirm, currently OSC consistently has 3-4ms less latency than sending to the driver.
Replacing Thread.sleep(5) with Thread.yield() brings OSC and OpenVR to have almost identical latency of 1-4ms (it's random which one recieves first)

the thread is at 200Hz because the driver should update at hmd update rate which is usually 144Hz or lower. we can set the sleep to 1ms it should give more consistent result than THread.yield() with less CPU load.

we can set the sleep to 1ms it should give more consistent result than Thread.yield() with less CPU load.

Well, it consistently has about 3ms more latency than OSC if you put any time in Thead.sleep() other than 0, which is equivalent to Thread.yield()
I worked for a few hours on a version of the bridge that uses overlapped IO instead of polling, with the idea being "maybe get the best of both worlds." I can push that somewhere if other people are also interested in working on it. I need another break from it though. I feel like there's got to be a library somewhere that wraps all this up into a nice async interface that i'm missing.

Okay, question. Would it make sense the way that the server is currently built to turn this thread into one that pulls information from the skeleton instead of having data pushed to it? That way, this thread could be simplified into: (pseudocode)
while True:
    hmd_info = read_hmd_packet() # blocking
    copy_skeleton() # or make a copy or something, so it isn't being concurrently modified
    update_skeleton(hmd_info) # only does the minimum computations necessary to account for current hmd position
    send_trackers_to_driver() # blocking

driver would then go: (pseudocode)
def driver_frame():
    send_hmd_info() # blocking
    recv_trackers() # blocking

we could extend this idea to also include the trackers from feeder app with some extra work. (feeder would no longer feed positions, just role information directly to driver. something something I finally cave to making feeder a thread or subprocess of driver)
This would kinda kill two birds with one stone. This issue, since we're always just waiting on the OS, and the issue on the driver side, because we're always sending up to date information.

#607 added polling to the Windows pipes, remaining is Linux :sob:

More specifically, that PR moves from sleeping to waiting for an incoming message with timeout. It prioritizes listening to the driver over sending updates from the server. (I'd like to build something that wake up immediately in both cases, but this is probably a step in the right direction.)

I think this can be closed now. The latency is good enough to where I now prefer steamvr over osc.

This was only implemented on windows

Have you tested it yourself to know if the latency is even a problem on linux? I would open a new issue for linux, as this issue blocks #577 and linux does not need to block 1.0.

Have you tested it yourself to know if the latency is even a problem on linux? I would open a new issue for linux, as this issue is discussion about windows named pipes and blocks #577. Linux does not need to block 1.0.

aside from you specifying this is windows, I dont see any place that speaks about windows-specific code :/

Literally all of the linked performance research is about windows named pipes, the discussion that prompted this issue was due to performance observed on windows, and the original motivation for linking 1.0 is to ensure the experience for reviewers is good, which means windows.
I appreciate the effort to improve linux, but that simply is not the focus of this discussion. If you want to open an issue for that, then you need to actually run it on linux and demonstrate that there is a problem with latency that can be improved.

If you want to open an issue for that, then you need to actually run it on linux and demonstrate that there is a problem with latency that can be improved.

This was, in fact, an issue on linux, which will be fixed in #629, which also fixes an bug in the previous PR that left the server submitting data a frame behind on windows. However, I had brought up that bug directly on the older PR, not in this issue.
I'm not going to bother re-opening since that PR should fix it soon.


Contract display/selection on timesheet#new

Currently, timesheet#new only allows selecting the month (and year). The contract that the timesheet will we attached to is part of the url, e.g. /contracts/2/time_sheets/new.

[x] If a HiWi only has one contract, show the contract (as simple text + link) that the timesheet is being added to.
[x] There should be the possibility for a HiWi with multiple contracts to choose one (dropdown or radio buttons)

Is it possible in the current system for a HiWi to have two simultaneous contracts? Tests?

Currently always shows the dropdown, even if user has only a single contract (see https://github.com/hpi-swt2/wimi-portal/commit/2e6ad3d01e48d38f15cc1b6d74aedcad548e4136#diff-bcbdf7d5188a8ef3ac4fdff8087eee7eR167)


Update Seek test cases

Bugs discovered in the PL/I code and fixes implemented in the CLI version have cascaded into changes in the PCyl-Web test cases.

Seek6 is a real challenge !  Pushed to remote master.  Closing.


Target region with print execution on device(0) host fails with undefined reference for numba_gil_ensure

Reporting a bug

[x] I have tried using the latest released version of Numba (most recent is
visible in the change log (https://github.com/numba/numba/blob/master/CHANGE_LOG).
[x] I have included a self contained code sample to reproduce the problem.
i.e. it's possible to run as 'python bug.py'.

Running An OpenMP target region that uses print and executes on device(0), that this using the host as a target, fails with undefined reference errors on the compled shared library, missing the symbol numba_gil_ensure

from numba.openmp import openmp_context as openmp
from numba.openmp import omp_get_thread_num
from numba.openmp import omp_get_num_threads
import numpy as np

@njit
def hello():
    with openmp('target device(0)'):
        print('Hello from target')

hello()

Fixed by PR #32