RFC: Mark all safety relevant files with a SPDX tag

Introduction
IEC 61508 safety certification is planned for the Zepyhr OS core.
Zephyr OS as a eco system is too big to be certified as a whole. The effort would be too high to finish in reasonalbe time.
Therefore a safety scope was defined in the past (Zephyr-Overview-2022Q1-Public.pdf ) and will be re-evaluated with the currently running safety certification process in the Safety Committee.
In order to clearly distinguish between non-safe (the far bigger part) and the safe code, we need a mechansim to cluster both, resp. separate / mark at least the safe code parts.
The mechanism may or should serve other such puposes (e.g. security, bluetooth, Profinet etc.).
Please find the closed PR with the with the initial discussion here: #47029.
Problem description
Certified code (also in tools) needs special attention concering modifications. Typically the certification is lost and requires re-certification, which normally implies:

Updating specification: requirements specifications and test specifications
Re-testing: in the simplest case just running a existing test suite again, for safety update and run self written test cases
Updating documentation: manuals
Updating certification inputs: misra reports, cve reports, coverage reports, test reports
Re-qualification of tools: a bit of all the points above but in a smaller extent

Also how modifications are done, at least concering safety, may be critical (e.g. using dynamic memory allocation is typically avoided in safe code) and render a module nearly impossilbe for re-certification.
Proposed change
All files being in a dedicated scope (e.g. safety scope) which will undergo any kind of certification shall be marked with a SPDX file comment tag.
SPDX-FileComment: <cert-scope1>[, <cert-scope2>]
For the safety scope it shall be:
"SPDX-FileComment: IEC-61508-SIL3" for Zephyr OS source code
"SPDX-FileComment: IEC-61508-T2" for T2 class tools
"SPDX-FileComment: IEC-61508-T3" for T3 class tools
That change will be done on 2.7-auditable-branch first, and be merged to main before LTS 3.
Detailed RFC
Motion 1:
We use the SPDX-FileComment tag together with a to-be-defined certification scope label to mark source code files of any programming language, belonging to the Zephyr OS source tree, belonging to a tool or the firmware built when that source code belongs to that certification scope.
Motion 2:
For the to-be-defined safety scope the certification scope label shall be:

IEC-61508-SIL3
IEC-61508-T2
IEC-61508-T3

Motion 3:
The spdx tagging mechanism as described above shall be implemented and tested on 2.7-auditable first before finally being merged to main. The resulting PRs will again be subject to approvals.
Proposed change (Detailed)
See Proposed change plus:
Third-party such as HAL will currently not be included in the safety scope primarily due to its extent.
The HAL providers would have to also provide the certifcation for their HALs or the Zepyhr OS integrator would have to achieve the certification for a particular HAL.
Depending on a tag a file has, the respective lead would have to be involved in planned changes (added as reviewer in a PR or similar).
How such a process could look like, is to be defined.
Dependencies
Dependency to peoples roles:
From @gregshue out of #47029:

As an integrator, it most clearly indicates what level of certification each file is intended to support.
As a contributor, it most clearly indicates the level of rigor that I must apply to this file.
As a downstream module developer, it helps me distinguish which tools and processes I may reuse for my own modules and executables.
As a participant in the Security WG, it establishes a reasonable pattern for indicating the set of security standards to be achieved. This is particularly important when looking to integrate multiple downstream modules where each was designed to meet a different security standard.

Concerns and Unresolved Questions

How to handle third-party code if a certification scope expands to also include third-party (HALs)?
west spdx does not yet include the headers used in a binary (tested by Anas). It seams only the command_comile.json listed files are currently used

I (romkell) consider that a short coming: If a header by chance is GPL licensed, that would mean that I have to opensource all the code im my binary and can not protect my IP, if I need to.
Fix: most compiler know a --dependencies, -MD, -FD -depend or similar preprocessor option, which outputs the implementation files dependencies etc.  typically in a makefile format. Those files can be generated either during a ordinary build or by re-invoking the compiler by west spdx to generate them. The dependency files then can be used as input to generate an input file list to west spdx.
zephyr.lst, unluckily, does not provide such information.

The tags meaning need to be clearly defined, a "safety" chapter should be started on 2.7-auditable and successively be merged (PRs) to main.

Alternatives

Process wise

Repeat SPDX tagging on every new <x>.<x>-auditable-branch as difference from the last <y>.<y>-auditable-tag

pros:

no impact (dependencies) to "main" developers

cons:

no control over certification in-conform changes
possible changes in a way, which will prevent certification

Technical wise

use a "database" e.g. yaml file, to mark the file outside of the file

pros:

easy to tag any file anywhere without modifiying it

cons:

some new tooling required similar to SPDX
outdate faster or "harder" to maintain, since in different location
no awareness of the certification status to people dealing with the file

Thoughts

Safety and innovation are somewhat contradicting, unless one is willing to take all the specification, testing etc. effort in its full extent over and over again.
Safety code should be mature and not be fundamentaly changed anymore.
Safety can only work if all parts involved (Zephyr OS firmware sources and tools) fulfill the cert criterias. Sparing out some of it, will not work.
For safety a "do's and dont's" or best practice can be written in a safety section of the docu.

Annex
Safety application development implications in short (and simplified)
This section provides a very brief overview of the implications of developing a (SIL3) safety application according to IEC 61508.

IEC61508 defines four safety integrity levels: SIL1, SIL2, SIL3 and SIL4

Safe applications may fail, but in a safe manner (at the cost of availability)
The higher the SIL number the lower the probability of a uncontrolled dangerous failure is accepted [1/h]

SIL1: ≥ 10 E–6 to < 10 E–5
SIL2: ≥ 10 E–7 to < 10 E–6
SIL3: ≥ 10 E–8 to < 10 E–7
SIL4: ≥ 10 E–9 to < 10 E–8

That applies to hardware and software, for the software aspect:

All code needs to be written against a specification on typically three levels

System / software requirements specification (SysRS or SRS)
Software architecture and interface specification (SAIS)
Software module  design specification (SMDS)

All test code needs to be written against a test specification on typically three levels

System / software test specification (STS) - securing that the SRS was implemented
Software integration test specification (SITS) - securing that the SAIS was implemented
Software module test specification (SMTS) - securing that the SMDS was implemented

All code involved needs to be tested with

System/Software Tests - showing that the System / software specification is fulfilled - not more, not less
Integration Tests - showing that the Software integration specification is fulfilled
Module (Unit) Tests - showing that the Software module specification is fulfilled

that 100% entry point coverage was reached (SIL1, 2, 3, 4)
that 100% statement coverage was reached (SIL2, 3, 4)
that 100% branch coverage was reached (SIL3, 4)
that 100% condition (MC/DC) coverage was reached (SIL4)
(exceptions to be justified, e.g. covered by code review as substitution)

Traceability (linkage) must be given from the top level specifications down into (test) code: V-Model

SRS <------------- verifies ------------- STS

SAIS <-------- verifies --------- SITS

SMDS <--- verifies --- SMTS

----- Code -----

SRS <-- refines -- SAIS <-- refines -- SMDS

STS <-- refines -- SITS <-- refines -- SMTS

STS <-- implements -- (Software test code)

SITS <-- implements -- (Software integration test code)

SMTS <-- implements -- (Software Module test code)

Tools used must be qualified for safety development

Three categories exist as defined by ISO 61508-4 3.2.11: T1, T2 and T3

T1: generates no outputs which can directly or indirectly contribute to the executable code
(e.g. text editor, design describing tool, ...) - No qualification required
T2: errors in the tool can fail to reveal defects but cannot directly create errors in the executable software - Qualification required
(e.g. test generator, test coverage tool, static analysis tool, ...)
T3: can directly or indirectly contribute to the executable code - Qualification required
(e.g. code generator, optimizing compiler, linker, ...)

Non-safety applications running on the same system may not influence the safety application in any way - evidence must be given that this is the case; safe and non-safe applications on the same MCU without MPU/MMU are basically impossible and even with, had to prove that the MPU/MMU guarantees a fail-safe reaction!

Therefore and as consequence of the above, mixing safe and non-safe code, in a eco system / framework where both is present, must be prevented - evidence must be given

Zephyr OS as a framework can, strictly spoken, not be certified, it is the applications (HW and SW) build on and using Zephyr OS that will be certified.

Zephyr OS can be, lets call it pre-certified, so that someone who builds a safe application using Zephyr OS does not have to prove safety fitness of source code and tools anymore but can treat it similar to a safety certified library with all the papers (safety collaterals) available.

"Why tagging files?" or "The Use Cases"

The project aims to certify the core part of the code as IEC 61508 SIL3
Having a clear indication in the code base what is safety relevant and what not

provides freedom to the developer on the non-safety side and
creates awareness to the developer on the safety side at the beginning of a development not on PR review (avoiding hard to impossible to certify implementations)
creates the security for the certifying assessor/body (TÜV/exida) that what is non-safe and safe is distinguishable

Allows to create a (tool supported) process to involve the safety architect early.
Allows to create automated configurations for tools treating the safety and non-safety side differently (e.g. static code analyzer)
Enables a safety user using the spdx tooling to verify and prove to TÜV/exida that only safety certified code was used
Marking the source files will very likely outdate less than a "external database" e.g. yaml file, since it is part of the PR review process, updating an external database gets forgotten easily

Tag handing "Auditable branches" vs "main"
I see two or three options how to work

Do it on both immediately

Advantages

Have the opportunity to handle most use cases as above immediately
Avoid safety contradicting developments in "main"
Have "main" closer to "safety ready" on tagging next LTS and auditable

Disadvantages

Safety Committee will have to take care about both
"main" developer will get confronted with safety topics

Only do safety on "Auditable Branches" and never on "main"

Advantages

Safety Committee can exclusively focus on auditables
"main" developer do not need to care about safety topics

Disadvantages

Over the "main" releases, there is a growing gap between the last and the next "auditable" branch - this gap will have to be closed (cleaned up) every time again
There is hardly any control over "safety un-friendly" implementation in the safety scope

Do it on "1. Auditable Branch" first, later, when finished forward merge to "Main"

Advantages and Disadvantages are at first those of 2. and later of 1.

SPDX-FileComment: [, ]

Putting multiple identifiers on the same file line has already shown to be problematic for enumerations.  We should use a separate comment line per cert-scope.

Safety and innovation are somewhat contradicting

I think you mean "orthogonal" rather than "contradicting".  Innovation is largely about concept. Safety (and security) are largely about process.

Safety code should be mature and not be fundamentaly changed anymore.

Verified that it fully meets the requirements and does nothing else - absolutely.  Not be recently fundamentally changed - impractical as change will happen.

How to handle third-party code if a certification scope expands to also include third-party (HALs)?

Not "if" but "when". We really need a common strategy and mechanism that can also identify files needing to meet security cert compliance (e.g., IEC-62443-SL2).

A couple of top-level concerns/questions.

Who will be responsible for determining the "to-be-defined safety scope" in motion 2?
What constitutes the T2 and T3 class tools in this scope? (I see this partially addressed in the now-closed PR, would be good to see this copy/pasted into the issue for the record, since we're voting on the issue in the TSC, not the PR)

Finally:
I know you're asking us not to consider the particular files, but you also seem convinced that the devicetree scripts should be in the safety scope.
I want to ask whether you consider that a library like pyelftools, which we rely on in a variety of scripts that are just as critical as the devicetree package (gen_kobject_list.py, gen_kobject_placeholders.py, gen_handles.py, ..., and the list goes on), should be part of the safety scope.
If yes, how do you plan on certifying it?
If no, why should the devicetree scripts get treated any differently?

SPDX-FileComment: [, ]

Putting multiple identifiers on the same file line has already shown to be problematic for enumerations. We should use a separate comment line per cert-scope.

I am fine with any format, as long as its is suitable and conformant to SPDX and west spdx (which I would assume it is, since also licenses are listed in separate lines).

Safety and innovation are somewhat contradicting

I think you mean "orthogonal" rather than "contradicting". Innovation is largely about concept. Safety (and security) are largely about process.

About processes that request output which have to be created with typically a lot more effort that a lean change.
In my experience companies tend to re-think changing concepts fundamentally twice, when the price tag is double or more.

Safety code should be mature and not be fundamentally changed anymore.

Verified that it fully meets the requirements and does nothing else - absolutely. Not be recently fundamentally changed - impractical as change will happen.

The gist here was, that a planned change should be a conscious step taking into account the consequences of any kind of certification effort. It was not meant absolute, otherwise we could stop working on any certified code.

How to handle third-party code if a certification scope expands to also include third-party (HALs)?

Not "if" but "when". We really need a common strategy and mechanism that can also identify files needing to meet security cert compliance (e.g., IEC-62443-SL2).

"When" is fine.
Nevertheless it leads back to the discussion in the PR if the mechanism is suitable also to cover non-Zephyr code.

A couple of top-level concerns/questions.
1. Who will be responsible for determining the "to-be-defined safety scope" in motion 2?

I guess it would be the respective Committee, for safety the Safety Committee. The maintainers participation is highly appreciated, I guess even needed, since (almost) no one from the Safety Committee knows the implementation details in all the code / tool code and would be able identify the relevant code part / files without spending weeks.
But in the end there is not a lot room for bargaining - if a tools generates code, modifies binaries etc. of a safe application, it is 61508-T3. If it is used for testing in any way (static code analysis, unit test frameworks etc.) it is 61508-T2

2. What constitutes the T2 and T3 class tools in this scope? (I see this partially addressed in the now-closed PR, would be good to see this copy/pasted into the issue for the record, since we're voting on the issue in the TSC, not the PR)

I will copy it from the PR.

Finally:
I know you're asking us not to consider the particular files, but you also seem convinced that the devicetree scripts should be in the safety scope.

Since it creates C structures present in the binary, yes I do not see a way around.

I want to ask whether you consider that a library like pyelftools, which we rely on in a variety of scripts that are just as critical as the devicetree package (gen_kobject_list.py, gen_kobject_placeholders.py, gen_handles.py, ..., and the list goes on), should be part of the safety scope.

If "pyelftools" participates in "code generation" in any way, it is in.

If yes, how do you plan on certifying it?

I respectively we (the Safety Committee) does not yet have a plan yet for all the tools used.
The advantage of an external tools I see is, that we use a fixed version (hopefully not changing all the time, since it is a mature tool - otherwise we have to re-qualify each new version).
Typically you prepare some tests around your use cases, which prove that the tools is fit for the purpose. And then do the paper work around those tests (docu). Nicole Pappler ( our safety manager) or @simhein would have to give more details.
@evgeniy-paltsev was preparing the tools list, where we have to indicate the tools purpose and criticality - this is work in progress.

If no, why should the devicetree scripts get treated any differently?

It will not be treated differently.
Btw. Zephyr OS is rather special in the way of generating code and post processing the binary.
Not something that makes it easier for safety.
For safety application I personally would prefer a simple constellation:
Tested and reviewed static source code -> the certified compiler -> the certified linker -> the safe binary
A fixed set of tools, at least for one release, ideally for several releases - new tool version, hence re-qualification of the tool (unless bought with certification).

Copied last comment from #47029 to here to continue:

We need to

@gregshue I believe I have seen you begin several sentences this way.
I think it is likely to cause confusion for people when you speak this way. It may falsely give people the impression that you are a maintainer or collaborator in the zephyr project, and that you are speaking in that capacity.

I understood that as "we" "the community" in what ever role.
Concerning out-of-tree code:
@gregshue takes a Zepyhr OS eco system end user position, the same position as Baumer is in. When building a safe application, as a end user I would want that the west SPDX output tells me, that I have used only certified code. Zephyr OS applications without peripherals drivers  and a HAL are limited to the host emulation or qemu (did I forget something?).
Hence as a end user I need to be able to create safe applications running on a board including HAL.
From that perspective and that I would like to get a report out of SPDX that I only used certified artifact, out-of-tree sources would have to be included and to carry that tag too.
But it is not in the control of the project but of the HAL vendor.
As long as this is a Zephyr OS proprietary solution, it will be difficult that it spreads elsewhere.
I understood @kestewart that she takes the approach to the SPDX standardization board (what ever the organizations name is) and tries to create an official solution (e.g. SPDX-Certification) instead of (mis-)using the SPDX-FileComment tag for that. But that might take a while to become officially released.
But It is a perspective. Maybe @kestewart can comment on that.
Finally, it is the out-of-tree source code providers and maintainers decision, whether or not he uses that mechanism. No way to force them. The more official, the more the likelihood that it will be adopted.
Using it for the inside-tree code is a start. Getting a SPDX report on that for the inside-tree is a start and already helpful to the end user.
Get a full coverage report for all files compiled in is deluxe, but might become true in the future.
Concerning commitment:
I feel that there should be some kind of commitment connected. Otherwise it is just useless tags.

As an example for safety

developers /maintainers take special care that changes are done in a manner, that do not hinder the respective re-certification.
If developers / maintainers, do not know what the implications are, that advice is seeked from a dedicated role (safety architect, security architect, some committee etc.) or/and documentation (safety do's and dont's)
That the roles listed above are invited to review and are given the authority to ask for a changed implementation (once the kernel is full of mallocs and frees where you can hardly prove that there is no constellation where you run out of heap, safety certification is history)
That certain to be defined special processes are followed for that code (as indicated above with reviewer invitation)

I would appreciate that further conversations continue in the issue #49829

But it is not in the control of the project but of the HAL vendor.

This actually depends on where the HAL lives.  If the HAL is provided through the zephyrproject-rtos then it has an identified module owner who is:

"required to submit the corresponding changes that are required in module repositories, to ensure that the Zephyr CI on th e pull request with the original changes, as well as the module integration testing are successful"
"required to fix issues in the module's codebase that have not been caught by Zephyr pull request CI runs"
So, the Zephyr Project does have some control over each repository provided through zephyrproject-rtos, and the identified module owner has the freedom (and responsibility) to update relevant files with the appropriate safety markings.

I agree that there is no way to force the usage in modules provided outside of zephyrproject-rtos.

A couple of top-level concerns/questions.

Who will be responsible for determining the "to-be-defined safety scope" in motion 2?
What constitutes the T2 and T3 class tools in this scope? (I see this partially addressed in the now-closed PR, would be good to see this copy/pasted into the issue for the record, since we're voting on the issue in the TSC, not the PR)

It is worth mentioning that whether this method is picked (SPDX) or some other method (A spreadsheet for example) the scope will still be defined and tools will be part of the certification/qualification process. So the method picked does not have any impact on the scope itself.

I agree that there is no way to force the usage in modules provided outside of zephyrproject-rtos.

There is also no way to force modules within the zephyr project to follow this. The key point is that we will not engage in certifying any third party code and code we do not own or control and I do not understand why we will do this. Actually, if someone decided to do so on their own and introduce this type of tagging into their HAL or module, it will just be confusing and you might end up getting code in a binary that marked for safety that is actually not covered by what the project is trying to do.
Thats where something like SPDX-FileComment: <cert-scope1>[, <cert-scope2>] might not be enough, it will need to have identifier of the scope, i.e. SPDX-FileComment: zephyr:<cert-scope1>[, <cert-scope2>] to make sure things do not get mixed up.

Not "if" but "when". We really need a common strategy and mechanism that can also identify files needing to meet security cert compliance (e.g., IEC-62443-SL2).

You keep bringing this but I have not heard anything, not from the security WG nor from the Security Architect that this is planned, being discussed or needed. Last time I checked, @ceolin  is the authority from the security WG/Committee and he should be putting this on the table and making the requirements on strategies and mechanisms for any safety certification, which again, I am not aware of.

The Security WG (which I am participating in) is not ready to discuss security standards yet.  The agenda for the next meeting is to review a Requirements Syntax training deck.  I expect we will begin to discuss OT/IoT security standards and certifications in the next few meetings.
Regarding need: files to be kept compliant with any standard(s) need to be marked with each a tag for each standard. For standards like IEC-62443 the tag needs to indicate the security level also.
More generally, an end user may reasonably need to identify files in their own workspaces with their own tags for similar purposes.  Limiting the tag detection tool and build hooks to only look for Zephyr-defined values would be inconsistent with the user need for extensibility.  Once the system can support a user-defined tag value it is trivial to add another Zephyr-defined value whenever a new tag is identified.

The Security WG (which I am participating in) is not ready to discuss security standards yet. The agenda for the next meeting is to review a Requirements Syntax training deck. I expect we will begin to discuss OT/IoT security standards and certifications in the next few meetings.
Regarding need: files to be kept compliant with any standard(s) need to be marked with each a tag for each standard. For standards like IEC-62443 the tag needs to indicate the security level also.
More generally, an end user may reasonably need to identify files in their own workspaces with their own tags for similar purposes. Limiting the tag detection tool and build hooks to only look for Zephyr-defined values would be inconsistent with the user need for extensibility. Once the system can support a user-defined tag value it is trivial to add another Zephyr-defined value whenever a new tag is identified.

A couple of top-level concerns/questions.
1. Who will be responsible for determining the "to-be-defined safety scope" in motion 2?

I guess it would be the respective Committee, for safety the Safety Committee. The maintainers participation is highly appreciated, I guess even needed, since (almost) no one from the Safety Committee knows the implementation details in all the code / tool code and would be able identify the relevant code part / files without spending weeks.
But in the end there is not a lot room for bargaining - if a tools generates code, modifies binaries etc. of a safe application, it is 61508-T3. If it is used for testing in any way (static code analysis, unit test frameworks etc.) it is 61508-T2

2. What constitutes the T2 and T3 class tools in this scope? (I see this partially addressed in the now-closed PR, would be good to see this copy/pasted into the issue for the record, since we're voting on the issue in the TSC, not the PR)

I will copy it from the PR.

Finally:
I know you're asking us not to consider the particular files, but you also seem convinced that the devicetree scripts should be in the safety scope.

Since it creates C structures present in the binary, yes I do not see a way around.

I want to ask whether you consider that a library like pyelftools, which we rely on in a variety of scripts that are just as critical as the devicetree package (gen_kobject_list.py, gen_kobject_placeholders.py, gen_handles.py, ..., and the list goes on), should be part of the safety scope.

If "pyelftools" participates in "code generation" in any way, it is in.

If yes, how do you plan on certifying it?

Ok, my two cents. Even if it is generating code, it is still part of a tool. We should consider it a tool, otherwise one could use the same argument for the python interpreter or other libraries. Of course, the generated code should be tagged and go over all certification process, but all these scripts should be considered tools (classified and possible qualified).

I respectively we (the Safety Committee) does not yet have a plan yet for all the tools used. The advantage of an external tools I see is, that we use a fixed version (hopefully not changing all the time, since it is a mature tool - otherwise we have to re-qualify each new version). Typically you prepare some tests around your use cases, which prove that the tools is fit for the purpose. And then do the paper work around those tests (docu). Nicole Pappler ( our safety manager) or @simhein would have to give more details. @evgeniy-paltsev was preparing the tools list, where we have to indicate the tools purpose and criticality - this is work in progress.

If no, why should the devicetree scripts get treated any differently?

It will not be treated differently.
Btw. Zephyr OS is rather special in the way of generating code and post processing the binary. Not something that makes it easier for safety. For safety application I personally would prefer a simple constellation: Tested and reviewed static source code -> the certified compiler -> the certified linker -> the safe binary A fixed set of tools, at least for one release, ideally for several releases - new tool version, hence re-qualification of the tool (unless bought with certification).

The Security WG (which I am participating in) is not ready to discuss security standards yet. The agenda for the next meeting is to review a Requirements Syntax training deck. I expect we will begin to discuss OT/IoT security standards and certifications in the next few meetings.

It is not about the WG be ready to discuss security standards, simply there is no demand from the project to pursue any certification, so it is definitely not a goal to group at the moment.  Regarding the next meeting, it is about how to capture requirements. If you remember, I asked to keep it as simple as possible. I really want the working group focusing in security enhancements and innovations, there are so many things that we know would be good to have, like, security storage, protocol fuzzing, syscall fuzzing, better samples integrating all these bits in the right way, threat models, crypto hw acceleration, .... but we lack human resources to do all these activities so we are talking about how to document (requirements) them for when someone can work in these items.
Now, if we start to put so many blockers no one will be interesting to work on them unless they are paid for ...

Regarding need: files to be kept compliant with any standard(s) need to be marked with each a tag for each standard. For standards like IEC-62443 the tag needs to indicate the security level also.
So, basically we have to mark all files in a build (consequently in the project) ? And attribute to them a security level ?

How this is manageable in an open source project ? Basically anytime someone change something we should assess and evaluate the risks again ? Even if it is just a high level assessment based on functionality and exposure, it is still almost unbearable.

More generally, an end user may reasonably need to identify files in their own workspaces with their own tags for similar purposes. Limiting the tag detection tool and build hooks to only look for Zephyr-defined values would be inconsistent with the user need for extensibility. Once the system can support a user-defined tag value it is trivial to add another Zephyr-defined value whenever a new tag is identified.

If it is about tool, the tool should be able to get whatever tag you put there and spill tag:files dictionary. But the problem for the project is how to identify them ...

simply there is no demand from the project to pursue any certification,

Of course not. The Zephyr Project repositories cannot get certified. That must be pursued by the end users.  Potential end users will evaluate Zephyr Project processes, tools, and code for suitability for their secure products.

I really want the working group focusing in security enhancements and innovations,

I really want the working group to answer what processes and artifacts I can regenerate to support a claim of "secure" code, and to which files and functions that applies.

If you remember, I asked to keep it as simple as possible.

I did, and even posted the training deck early so people could go through it off line.

so we are talking about how to document (requirements) them for when someone can work in these items

Good.

if we start to put so many blockers no one will be interesting to work on them unless they are paid for ...

The vision and mission of the project is firmware for safe and secure resource-constrained devices.  If the processes needed for that are not interesting to them, then they probably won't be satisfied working in these domains.  These processes only need to be applied to the functionality claimed to be secure.  At this point I don't know how much of the code base is targeted for that label.

How this is manageable in an open source project ? Basically anytime someone change something we should assess and evaluate the risks again ? Even if it is just a high level assessment based on functionality and exposure, it is still almost unbearable.

Great questions!  I see these as the innovative area that the Zephyr Project is chartered to solve. Though it may be burdensome for developers it gives the Zephyr Project great value to end users.

If it is about tool, the tool should be able to get whatever tag you put there and spill tag:files dictionary. But the problem for the project is how to identify them ...

The charter refers to functionality being brought into the auditable branch, so it starts with identifying functionality and then identifying all source associated with each distinct piece identified.

if we start to put so many blockers no one will be interesting to work on them unless they are paid for ...

Several events have happened in the world over the last 3-4 years that are fundamentally changing how general software development needs to be done. Security events and new regulations are driving many domains toward secure software.  Regulations, customer demand, and competition are  driving software developers toward standards that require no unused/dead code, defense in depth, and all external input validated.  Proving something doesn't exist either requires proving that it can't exist, or knowing everything about the executable.  For a non-trivial executable that will demand a thorough process. Software developers in most non-regulated domains will probably need to adjust anyway.

Not sure why this was moved to a new issue but my https://github.com/zephyrproject-rtos/zephyr/pull/47029#issuecomment-1175973912 still stands.

But it is not in the control of the project but of the HAL vendor.

This actually depends on where the HAL lives. If the HAL is provided through the zephyrproject-rtos then it has an identified module owner who is:
* "required to submit the corresponding changes that are required in module repositories, to ensure that the Zephyr CI on th e pull request with the original changes, as well as the module integration testing are successful"

* "required to fix issues in the module's codebase that have not been caught by Zephyr pull request CI runs"
  So, the Zephyr Project does have some control over each repository provided through zephyrproject-rtos, and the identified module owner has the freedom (and responsibility) to update relevant files with the appropriate safety markings.

I agree that there is no way to force the usage in modules provided outside of zephyrproject-rtos.

I quickly looked at the stm32 HAL. Only stm32cube is some 3000+ files. Assuming Zepyhr OS uses 1/3, hence 1k, mainly the low level driver as far as I know.
The HALs may be no real downstream repos from a main where you have a fixed linkage and simply update to a tag or what so ever.
As for stm32cube there is a maintainer who, when updating to new version, rearranges the folder structure a bit from the original stm32cobe and then updates to the new files.
In general I would avoid modifiying third-party code a much as possible (also those SPDX tags), since when you update, you will have to redo your modifications over and over again.
And if such SPDX tags should stand for "this is IEC61508-SIL3 ready code" on a tagged x.y-auditable-branch, the Zephyr OS project would have to run the certification of all those HALs and boards.
That is, effort-wise, impossible.
Staying with the stm32 example, I rather see it the way, that, STM will have to provide certification prove for a particular version of their HAL to the end user (or adopiting such a mechanism as SPDX tagging for their HAL, and for that it needs to be a official open standard at least), or the end user having to do that effort for that particular hardware he uses.

if we start to put so many blockers no one will be interesting to work on them unless they are paid for ...

Several events have happened in the world over the last 3-4 years that are fundamentally changing how general software development needs to be done. Security events and new regulations are driving many domains toward secure software. Regulations, customer demand, and competition are driving software developers toward standards that require no unused/dead code, defense in depth, and all external input validated. Proving something doesn't exist either requires proving that it can't exist, or knowing everything about the executable. For a non-trivial executable that will demand a thorough process. Software developers in most non-regulated domains will probably need to adjust anyway.

I can understand that a regulated environment (such as safety and I guess security - which is less my experience - might put off developers. To my experience there is a difference between regulated and non-regulated.
The more important I see it, that the safety scope is clearly defined and marked.

Cryptosecurity of IoT devices has been regulated in California since 1/1/2020.  Cryptosecurity of consumer wireless devices with an address (e.g., BT) has been regulated, whether or not it contains an IP stack, in Oregon since 1/1/2020. More laws are under way (see https://www.nabto.com/us-and-california-iot-security-laws-guide/).
Even without regulations, customers in application domains being targeted by the Zephyr Project have already adopted IEC 62443 Security for Industrial Automation and Control Systems (see https://press.siemens.com/global/en/pressrelease/siemens-process-control-system-first-product-iec-62443-security-certification, dated 2016).  Certificate of compliance can be awarded for a specific device.  Separately, ETSI EN 303 645 Cybersecurity Standard for Consumer IoT Devices is a globally acceptable standard including 33 mandatory requirements and 35 recommendations.

The more important I see it, that the safety scope is clearly defined and marked.

I agree that the safety scope is clearly defined and marked, and the security scope is not.  As I look at application domains, defining a security scope and level that meets a broad set of

if we start to put so many blockers no one will be interesting to work on them unless they are paid for ...

Several events have happened in the world over the last 3-4 years that are fundamentally changing how general software development needs to be done. Security events and new regulations are driving many domains toward secure software. Regulations, customer demand, and competition are driving software developers toward standards that require no unused/dead code, defense in depth, and all external input validated. Proving something doesn't exist either requires proving that it can't exist, or knowing everything about the executable. For a non-trivial executable that will demand a thorough process. Software developers in most non-regulated domains will probably need to adjust anyway.

I can understand that a regulated environment (such as safety and I guess security - which is less my domain) might put off developers. To my experience there is a difference between regulated and non-regulated. Processes in non-regulated environments are typically leaner.
For that, see it even more important, that the safety scope is clearly defined and marked.

Ok, my two cents. Even if it is generating code, it is still part of a tool. We should consider it a tool, otherwise one could use the same argument for the python interpreter or other libraries. Of course, the generated code should be tagged and go over all certification process, but all these scripts should be considered tools (classified and possible qualified).

Not fully clear to me what the notion of the said is. I do not think that we required to qualify the python interpreter as a tool, but we will have to qualify a version of the code generating script together with a particular version of the python interpreter, showing that the combination of both does what it is meant to.
Your comment leads me to a thought that for the source code generating tools, the generated code parts could be treated and tested similar to hand coded files (unit testing, integration testing, software testing), possibly re-leaving the tool from qualification. Best, the test code then should be written flexible enough to cope with the different tool outputs.
It is all options, not yet defined and not yet accepted by TÜV/exida.
At this point we are simply looking for a mechanism to mark scopes.

Not sure why this was moved to a new issue but my #47029 (comment) still stands.

This was decided in the last TSC meeting to get the focus on the mechanism instead of the scope with its 800+ files.

I don't like the idea of statically tagging source files with meta-information that typical developers cannot take decisions on.

A short follow up on your comment @henrikbrixandersen . If we decide on another mechanism to mark the safety scope ( e.g spreadsheet or a yml file ) isn't this also adding meta-information to a file with a detour? And the developers cannot take decisions on that too but the only difference I guess would be the decision is not present in the developers file directly.

Cryptosecurity of IoT devices has been regulated in California since 1/1/2020. Cryptosecurity of consumer wireless devices with an address (e.g., BT) has been regulated, whether or not it contains an IP stack, in Oregon since 1/1/2020. More laws are under way (see https://www.nabto.com/us-and-california-iot-security-laws-guide/).
Even without regulations, customers in application domains being targeted by the Zephyr Project have already adopted IEC 62443 Security for Industrial Automation and Control Systems (see https://press.siemens.com/global/en/pressrelease/siemens-process-control-system-first-product-iec-62443-security-certification, dated 2016). Certificate of compliance can be awarded for a specific device. Separately, ETSI EN 303 645 Cybersecurity Standard for Consumer IoT Devices is a globally acceptable standard including 33 mandatory requirements and 35 recommendations.

Can we all focus on the topic of this RFC please? Which is not the security standard or the focus of the security WG.

The more important I see it, that the safety scope is clearly defined and marked.

I agree that the safety scope is clearly defined and marked, and the security scope is not. As I look at application domains, defining a security scope and level that meets a broad set of customer domains is more important to the success of Zephyr than the safety scope. There are FAR more end users needing a secure solution than a safe one.

I think we all agree on the point that the safety scope needs to be clearly defined, but I would appreciate if we could avoid the
Debat on principles of the project whether security or safety is more important. This needs to be discussed in another forum.

Ok, my two cents. Even if it is generating code, it is still part of a tool. We should consider it a tool, otherwise one could use the same argument for the python interpreter or other libraries. Of course, the generated code should be tagged and go over all certification process, but all these scripts should be considered tools (classified and possible qualified).

Not fully clear to me what the notion of the said is. I do not think that we required to qualify the python interpreter as a tool, but we will have to qualify a version of the code generating script together with a particular version of the python interpreter, showing that the combination of both does what it is meant to. Your comment leads me to a thought that for the source code generating tools, the generated code parts could be treated and tested similar to hand coded files (unit testing, integration testing, software testing), possibly re-leaving the tool from qualification. Best, the test code then should be written flexible enough to cope with the different tool outputs.
It is all options, not yet defined and not yet accepted by TÜV/exida.
At this point we are simply looking for a mechanism to mark scopes.

I think the notion here is that scripts shall be treated as own tools instead of code that generates c code and the generated c code needs to be treated as "hand" written code with all the safety aspect and methods in mind. I fully agree with that.
If we don't do this someone could come up with the argument "hey you have python code which generates c code and the tool you are using is python" and than the python interpreter would come to the table. ( Is that what you wanted to point out @ceolin ?)
About your thought @romkell It wouldn't release the tool from qualification but it can be used for it to minimize the effort.
But the discussion what needs to be defined as tool and what not is another one and with this mechanism we would have a least a tool ( pun intended ) to do such things.

A short follow up on your comment @henrikbrixandersen . If we decide on another mechanism to mark the safety scope ( e.g spreadsheet or a yml file ) isn't this also adding meta-information to a file with a detour? And the developers cannot take decisions on that too but the only difference I guess would be the decision is not present in the developers file directly.

Sure, at some point - before doing a recertification - the list needs to be evaluated again. But that would be in the scope of the recertification, not in the day-to-day changes done by developers without insights into how to handle the safety scope.

This was decided in the last TSC meeting to get the focus on the mechanism instead of the scope with its 800+ files.

Fair enough, although that's not what I understood from the meeting. I don't believe we can fully decouple the two (scope vs. mechanism).
Say, if this was tagging a few files (e.g. the base kernel files) in the tree with a safety tag, I don't think many people would object. But tagging 800+ files all over the tree makes this relevant for a lot of developers with little or no insight in the processes around safety certification.

A short follow up on your comment @henrikbrixandersen . If we decide on another mechanism to mark the safety scope ( e.g spreadsheet or a yml file ) isn't this also adding meta-information to a file with a detour? And the developers cannot take decisions on that too but the only difference I guess would be the decision is not present in the developers file directly.

Sure, at some point - before doing a recertification - the list needs to be evaluated again. But that would be in the scope of the recertification, not in the day-to-day changes done by developers without insights into how to handle the safety scope.

This is about working together and having mechanisms in the safety scope which support development towards a safety mindset. Is is about defensive programming (do's and dont's with a focus on safety, which I feel is not very far from do's and dont's for security).
Measures to achieve that can be:

In the safety scope have a safety architect review code changes
write guidelines how to program for safety (what to do and what to better leave)
train developers concerning safety, if required

Your options with leaving the day-to-day changes done by developers just happen, might end up with a practically in-re-certifiable code base (e.g. mallocs / frees all over the place, loop boundaries re-calculated in the loop, etc.).
It is a bit like "lets develop and see if we get it re-certified again in 2 years with the next LTS". A risky strategy.

In the safety scope have a safety architect review code changes

This goes back to the question of the scope for this. Are we talking day-to-day changes of 800+ files requiring review from one dedicated person (the safety architect) - or are we talking 10 files? 20 files?

write guidelines how to program for safety (what to do and what to better leave)
train developers concerning safety, if required

While this sounds easy as a one-line, how would you do this on an open source project such as Zephyr in practice? We have 1000+ developers so far.

Your options with leaving the day-to-day changes done by developers just happen, might end up with a practically in-re-certifiable code base (e.g. mallocs / frees all over the place, loop boundaries re-calculated in the loop, etc.). It is a bit like "lets develop and see if we get it re-certified again in 2 years with the next LTS". A risky strategy.

Now you are just making things up ;-) We don't see "mallocs / frees all over the place" or "loop boundaries re-calculated in the loop" as it is today as these are bad practice, safety scope or not.

In the safety scope have a safety architect review code changes

This goes back to the question of the scope for this. Are we talking day-to-day changes of 800+ files requiring review from one dedicated person (the safety architect) - or are we talking 10 files? 20 files?

At first place we need to distinguish between the tool files and the source code files which where mixed up in the 800+ files for the PR. I'm actually working on refining the scope for the source files and it seems like that have a number around 230 files under scope but I need to double check it.

write guidelines how to program for safety (what to do and what to better leave)
train developers concerning safety, if required

While this sounds easy as a one-line, how would you do this on an open source project such as Zephyr in practice? We have 1000+ developers so far.

Okay as I read your argument you are right, it it is maybe impossible for the project in a whole to introduce such things, but this is more like a thought or a idea what could be maybe done with such a marking mechanism. What also needs to be pointed out if we agree on this mechanism is that it should not happen that the safety or whatever committee/WG will introduce new processes or implementations based on this mechanism without the approval of the TSC/process WG and the community.

Your options with leaving the day-to-day changes done by developers just happen, might end up with a practically in-re-certifiable code base (e.g. mallocs / frees all over the place, loop boundaries re-calculated in the loop, etc.). It is a bit like "lets develop and see if we get it re-certified again in 2 years with the next LTS". A risky strategy.

Now you are just making things up ;-) We don't see "mallocs / frees all over the place" or "loop boundaries re-calculated in the loop" as it is today as these are bad practice, safety scope or not.

Well that is a good thing to hear :) I see the point of @romkell more like from the architectural perspective, when someone wants to introduce  for example a new implementation or a design for something very dynamic, it would be possible that the project or committee could act on it like we do now with the normal Architecture WG meeting ( correct me if I'm wrong ).

This was decided in the last TSC meeting to get the focus on the mechanism instead of the scope with its 800+ files.

Agreed.  It was also noted by @kestewart that the mechanism is expected to work for identifying the security scope.

Can we all focus on the topic of this RFC please? Which is not the security standard or the focus of the security WG.

Agreed.  As I see it, all of the discussion around security regulations and standards pretty clearly shows the scope of the mechanism needs to include all the places where security tags are expected to appear. This brings up a very relevant question:

Does Zephyr Project need to support identifying in the main branch which source is "secure" and to what level?

This is about working together and having mechanisms in the safety scope which support development towards a safety mindset. ...

There are analogs in the security scope that this mechanism is also expected to support.
Have we even created a list of the requirements a mechanism must meet?

Not fully clear to me what the notion of the said is. I do not think that we required to qualify the python interpreter as a tool, but we will have to qualify a version of the code generating script together with a particular version of the python interpreter, showing that the combination of both does what it is meant to. Your comment leads me to a thought that for the source code generating tools, the generated code parts could be treated and tested similar to hand coded files (unit testing, integration testing, software testing), possibly re-leaving the tool from qualification. Best, the test code then should be written flexible enough to cope with the different tool outputs.
It is all options, not yet defined and not yet accepted by TÜV/exida.
At this point we are simply looking for a mechanism to mark scopes.

I think the notion here is that scripts shall be treated as own tools instead of code that generates c code and the generated c code needs to be treated as "hand" written code with all the safety aspect and methods in mind. I fully agree with that. If we don't do this someone could come up with the argument "hey you have python code which generates c code and the tool you are using is python" and than the python interpreter would come to the table. ( Is that what you wanted to point out @ceolin ?)

Yes !

About your thought @romkell It wouldn't release the tool from qualification but it can be used for it to minimize the effort.
But the discussion what needs to be defined as tool and what not is another one and with this mechanism we would have a least a tool ( pun intended ) to do such things.

From https://github.com/zephyrproject-rtos/zephyr/pull/47029#issuecomment-1170212688

We need to have a mechanism to indicate what is in and what out.
What are other options:

parsing zephyr.lst is fine but to verify it again what?

some kind of database telling what is in the safety scope and what not? Who would maintain that? That database would rot even faster, I guess, because it is detached from the code completely.
Use KConfig #ifdef CONFIG_SAFETY or similar
Mark all safety related functions with /* @in_safety_scope */ or similar so those information can be copied when splitting files. Quite some micro management, but easier to handle for those moving these things around

Another option would be to leave "main" untouched from such mechanisms and only do and re-do it on each x.y-auditable branch which will be certified.

A few thoughts on the options above:

The purpose of the security discussion was to show the last options will be impractical very soon.
In order to support development of FW images using either a workspace application model or freestanding application model what is in/out of each scope must track and follow each module. A centralized "database" doesn't scale to our current needs.
A per-function or per-file marking is more granular than the "features to move from the long term support (“LTS”)
code base to the auditable code base" described in the Zephyr Charter.

It seems "features" are scoped and each activation controlled by Kconfigs.  How cleanly are the "features" partitioned into files?

No conclusion on this specific RFC for almost one year,  also other options are in consideration in the safety committee/WG.
I will close this RFC and it can be reopened if needed.


Arabic Characters not display properly in IE and un selectable with Firefox and Chrome

My PDF has the arabic characters,  It is showing with Firefox and Chrome as like image. I am not able to select the text at all...
In case of  ie 11, the characters not at all visible it displaying nothing.

How to fix the ie arabic character issue...
How to make text are select-able?

Do i required to installed the arabic fonts with me or any other workaround need to do here?
Here is the screen shot,
ie 11 display :-1:

Firefox / chrome display:-

Please help me to sort this.

When reporting an issue, please make sure that you read: https://github.com/mozilla/pdf.js/blob/master/CONTRIBUTING.md.
At the very least, you must provide a link to a PDF file that can be used to reproduce this issue.

I am sorry, here is the link for pdf file. https://dl.dropboxusercontent.com/u/6257603/aspire2.pdf

The file renders identical to Adobe Reader for me, in both Firefox 39 and IE 11, using PDF.js 1.1.11.
Please note that the Arabic "text" isn't actually text, but rather a series of images. This thus explain why text selection isn't possible even in Adobe Reader.

I am using the version of "PDFJS.version = '1.0.1149';" , I am using pdf.js simpleviewer from the components. just a couple of day before I integrated using node. in case if I required to update my version of pdf.js can you give me the link of latest please?

Yes, I agree all this Arabic text are series of images. since i am getting from server i am not realized this page. but still the images is not shows the correct result like other browser do. Is there a way to fix this? or avoiding series of images to "select-able" text may work out?

Closing IE11/Edge (non-Chromium-based) issues in response to #11211. Please note that, as outlined in https://github.com/mozilla/pdf.js/wiki/Frequently-Asked-Questions#faq-support, only bugs which completely prevent the library and/or the default viewer from running will be accepted.


Configurable prefix for Pool-Managed Simulators

Adds an additional property to FBSimulatorControlConfiguration to allow for prefixes other than E2E. A prefix must be non-empty for the meantime, until it is possible to work this out with a separately managed SimDeviceSet.

lgtm


Use array field with currency ids instead individual bool fields

That we have currently:
{
...
"rub": true,
"usd": true,
"eur": false,
...
}

Expected:
{
...
"currency": [643, 840],
...
}

See https://en.wikipedia.org/wiki/ISO_4217#Active_codes
RUB, USD, EUR are enough for near future.

Resolved in #51 by @CloudedCat


Roll engine c635d70c7266..414ad3866251 (3 commits)

git@github.com:flutter/engine.git/compare/c635d70c7266...414ad3866251
git log c635d70c7266..414ad3866251 --no-merges --oneline
2019-10-08<EMAIL_ADDRESS>Missing link flag (flutter/engine#13001)
2019-10-08<EMAIL_ADDRESS>Started setting our debug background task id to invalid after completion. (flutter/engine#12999)
2019-10-08<EMAIL_ADDRESS>Add onUnregistered callback in 'Texture' and 'FlutterTexture' (flutter/engine#12695)
If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Please CC<EMAIL_ADDRESS>on the revert to ensure that a human
is aware of the problem.
To report a problem with the AutoRoller itself, please file a bug:
https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

PullRequest is not longer mergeable. Closing it.

PullRequest is not longer mergeable. Closing it.

Commit queue failed; closing this roll.

PullRequest is not longer mergeable. Closing it.


feat(Lezer grammar): Add nested pipeline

This PR adds the token NestedPipeline to the list of expressions in order to get the included test to pass.
It also removes a redundant set of parenthesis in the CallExpression token, and makes the escape sequence matching more strict.

Nice — good change, great test!


[Feature Request]: Dynamically removing tools during agent conversation when the tools are not needed or not useful.

Is your feature request related to a problem? Please describe.
To save tokens or improve GPT responses, it makes sense to remove certain tools when they are no longer needed.
Describe the solution you'd like
autogen.agentchat.unregister_function(
hello_world,
caller=picker,
executor=picker
)
Additional context
No response

Can you please provide an example scenario where this will be helpful?

Sure, I use this in my other chatbot all the time. Let's say you have diferent tools available for the Agent to choose from, but when a tool fails, because it's impossible to execute, you might want to remove it, save tokens, and let the Agent use other tools instead to complete the job.
Another example is simply removing tools to save tokens. Very often you can assume that certain tools will not be needed when other tools have been already picked. For example, if the Agent picked a weather_info tool, then you can assume that the stock_info tool might be not needed. So you remove it.

Some observations:

there is method update_tool_signature which even has "is_remove" flag. But it seems removing the function only from llm_config["tools"] but from function_map

I am not sure this is an issue for execution..

I was able to acomplish this feature myself with this code:
def remove_tool(agent, tool_name):
    """Remove tool from the tool list"""
    tools = agent.llm_config.get("tools")
    tools = [tool for tool in tools if tool.get("function", {}).get("name") != tool_name]
    agent.llm_config["tools"] = tools
    map_ =  agent._function_map
    del map_[tool_name]

+1 for this feature request
In my use case, each function correspond to an action that the agent could perform.
As the agent interact with the world,

new actions become available -> register_function
old actions could be no longer possible -> unregister_function (but no such thing in autogen)

Example:
LLM agent was put to play a text based adventure game
Agent is first at the pub:
functions it can call:

walk
talk
drink

Agent now walks to forest:
functions it can call:

walk
~talk~
~drink~
chop

Since the LLM APIs only support a fixed max number of tools (ex: 128 for OpenAI), it is necessary to be able to provide only the most relevant tools. In order to do this, the user should have the possibility to implement its own "tool pool" with an associated selection strategy.
Two ways to do this:

Simple: add AssistantAgent::remove_tools(self, tools: List) and AssistantAgent::add_tools(self, tools: List) .
Advanced: implement a ToolPool interface/class.

If that's ok with the maintainer I'll start by implementing 1. then we can design 2.

@JMLX42 happy to feature your extension if you would like to pursue an experimentation of this idea.

"tool pool" aka "workbench" design proposal issue: https://github.com/microsoft/autogen/issues/4721


zenergy: init at a3e1244

Description of changes
Adds a new kernel module, zenergy, this is necessary for accessing CPU power draw data without root permission, specifically MangoHud requires it to be able to show the user their CPU power draw.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[X] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[X] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

I apologize if I missed something in advance, first time contributing to the repository! I am a little confused though with the testing and release node markdown, for the testing I couldn't find an example of a kernel module being tested, if it's even possible with the suite available; whereas the markdown file, it implies that not all additions will be covered, so would this PR be applicable still? Not quite sure.

I am a little confused though with the testing and release note markdown, for the testing I couldn't find an example of a kernel module being tested, if it's even possible with the suite available; whereas the markdown file, it implies that not all additions will be covered, so would this PR be applicable still? Not quite sure.

If you are doing the change on top of nixos-unstable, you can just rebuild your system with it and try to use the kernel module like normally done.
We don't need a release note entry here.

I've been using that kernel module locally since the PR has been open, so I can attest that it's in working condition!


[WIP][SPARK-44937][CORE] Add SSL/TLS support for RPC and Shuffle communications

What changes were proposed in this pull request?
This PR adds support for SSL/TLS based communication for Spark RPCs and block transfers - providing an alternative to the existing encryption / authentication implementation documented at https://spark.apache.org/docs/latest/security.html#spark-rpc-communication-protocol-between-spark-processes
This is based on an existing PR from 2015: https://github.com/apache/spark/pull/9416 - with some refactoring and a number of updates to make it work with changes to Spark since.
I understand this is a large PR, so I've broken this down by a high level summary of changes and a suggested review order:

Add a dependency on netty-tcnative-boringssl-static which provides support for faster TLS communication
Add documentation for the new functionality in docs/security.md which describe the new flags and configuration.
Extend SSLOptions and TransportConf to support the new flags for this feature
Extend TransportContext to optionally add an SSL based handler if configured, and make similar changes in the transport client and server factories
Add a new API to ManagedBuffer to convert objects to a Netty object for SSL encoding (since we can't use zero-copy transfers, we can't use convertToNetty() directly)
Add some helper classes for communication:

EncryptedMessageWithHeader and SSLMessageEncoder are quite similar to the existing variants but just different enough that it was hard to consolidate
SSLFactory to create the JDK / Netty SSL handlers as appropriate. This handles configuration of the protocols, ciphers, etc
ReloadingX509TrustManager to support trust store reloading.

Change SecurityManager to disable the existing authentication/encryption mechanisms if this new feature is enabled
Update SparkConf and CommandUtils to pass passwords via env variables if needed, to preserve security guarantees (similar to the existing SSL password propagation)
Update almost all constructor callsites of SparkTransportConf to propagate SSL options
Add tests:

Add test keys + certificates in a bunch of places
Add an SSLSampleConfigs class for sample configurations used in tests
Add tests for a bunch of the new classes + features
Add new tests for almost all modules that create a TransportContext which rerun the same tests but with this new feature enabled (this caught a bunch of bugs). This will ensure features are compatible with SSL going forward

Why are the changes needed?
Spark currently does not support TLS/SSL for RPC and block transfers. It is helpful to have this as an alternative encryption method for users which may need to use more standard encryption mechanisms instead of one that is more internal to spark.
Does this PR introduce any user-facing change?
Yes, this includes new configuration options and updates associated documentation. Besides that, though, all other aspects of Spark should remain unchanged (modulo performance differences).
How was this patch tested?
Added a bunch of unit tests that pass.
I also ran some queries locally to ensure they still work.
I verified traffic was encrypted using TLS using two mechanisms:

Enabled trace level logging for Netty and JDK SSL and saw logs confirming TLS handshakes were happening
I ran wireshark on my machine and snooped on traffic while sending queries shuffling a fixed string. Without any encryption, I could find that string in the network traffic. With this encryption enabled, that string did not show up, and wireshark logs confirmed a TLS handshake was happening.

Was this patch authored or co-authored using generative AI tooling?
No

cc: @JoshRosen, @mridulm, @vanzin and @turp1twin as potential reviewers to look at this PR based on blame. Happy to discuss/share more context, I understand this is a big PR. I'm looking to gather high level feedback so I can address any concerns and work towards getting this merged. Thank you in advance!

@hasnain-db Wow, this a blast from the past.. Happy to review... Cheers!

I wonder if the failing test is flaky, it consistently passes for me locally

Thanks for working on this @hasnain-db , this is a very nice adding to spark !
Given the size of the PR, can we split this up to make it easier to review ?

Thanks @mridulm ! Happy to do that. I can think of a nice split in line with the bullet points listed in the summary here.
Just to confirm (since I'm not sure this repo has support for stacked PRs - if there is, please link me to an example) - you're proposing I put up one PR, get it approved and merged, then put up the second PR, and so on, right (since most changes depend on each other).

+CC @otterc

Just to confirm (since I'm not sure this repo has support for stacked PRs - if there is, please link me to an example) - you're proposing I put up one PR, get it approved and merged, then put up the second PR, and so on, right (since most changes depend on each other).

If there are independent PR's, those can be done in parallel.
If there is a interdependency, they would end up being serial, agree

The overall feature has been merged.


GetAllAccountsAsync is nto working

Hi..
I would like to use your API...
but your testing example is not working...
what are you usign as signature ?
could you check it your example is working ?
please see me test code bellow...
Many thanks
Tomas

using GDAXClient.Authentication;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Security.Cryptography;
using System.Text;
using System.Threading.Tasks;
namespace test
{
class Program
{
static string apiSecret = "xxx";
    private static async void xxx()
    {
        string phrase = "xxx";
        string apiKey = "xxx";

        string signature = "";

        string ts = GetNonce();
        string method = "/accounts";
        string sig = GetSignature(ts, "GET", method, string.Empty);

        signature = sig;

        GDAXClient.Authentication.Authenticator aut = new GDAXClient.Authentication.Authenticator(apiKey, signature, phrase);
        GDAXClient.GDAXClient client = new GDAXClient.GDAXClient(aut);
        var a = await client.AccountsService.GetAllAccountsAsync();

    }

    static void Main(string[] args)
    {
        xxx();

    }

    private static string GetNonce()
    {
        String a = (DateTime.Now.ToFileTimeUtc() / 1000).ToString();
        String b = (DateTime.UtcNow - new DateTime(1970, 1, 1)).TotalSeconds.ToString();
        return b;
    }

    private static string GetSignature(string nonce, string method, string url, string body)
    {
        string message = string.Concat(nonce, method.ToUpper(), url, body);
        var encoding = new ASCIIEncoding();
        byte[] keyByte = Convert.FromBase64String(apiSecret);
        byte[] messageBytes = encoding.GetBytes(message);
        using (var hmacsha256 = new HMACSHA256(keyByte))
        {
            byte[] hashmessage = hmacsha256.ComputeHash(messageBytes);
            return Convert.ToBase64String(hashmessage);
        }
    }

}

}

@moski0 its even easier than that. the client will do all of the signing for you, all you need to do is just pass in your api key, signature (unsigned), and passphrase
so from your example.... it is just this:
        GDAXClient.Authentication.Authenticator aut = new GDAXClient.Authentication.Authenticator(<your-apiKey>, <your-unsignedsignature>, <your-phrase>);
        GDAXClient.GDAXClient client = new GDAXClient.GDAXClient(aut);
        var a = await client.AccountsService.GetAllAccountsAsync();

let me know if something still doesnt work 😄

I have still exception during running GetAllAccountsAsync...
"invalid timestamp"

May be related to https://github.com/dougdellolio/gdax-csharp/issues/26. The issue should have been fixed with v1.0.5 (from this pr https://github.com/dougdellolio/gdax-csharp/pull/27). Make sure you have the latest version from Nuget. If it still doesn't work then it may still be a bug that I will fix.
Let me know!

Hi..
yes I have last version 1.0.5 .. (I just tried remoev adn download it again) .. and problem is still the same...
![x](https://user-images.githubusercontent.com/34658970/34207692-d19ab826-e58b-11e7-8cd2-
974fe82e63f2.png)

here is me version...

I checked your fix.. for  #26 .. it does not seems to be related .... ( dot is instead of comma)
see debug..

Another user that opened the issue I mentioned above had another fix that seemed to work for him if you would like to try that out. I will be able to make the change tonight.

I found it.. now it works .. :-)
next change from ToString() to ToString(CultureInfo.InvariantCulture) was needed ...
.. see code bellow ...
private void AddHeaders(
HttpRequestMessage httpRequestMessage,
IAuthenticator authenticator,
string signedSignature,
double timeStamp)
{
httpRequestMessage.Headers.Add("User-Agent", "GDAXClient");
httpRequestMessage.Headers.Add("CB-ACCESS-KEY", authenticator.ApiKey);
httpRequestMessage.Headers.Add("CB-ACCESS-TIMESTAMP", timeStamp.ToString(CultureInfo.InvariantCulture));
httpRequestMessage.Headers.Add("CB-ACCESS-SIGN", signedSignature);
httpRequestMessage.Headers.Add("CB-ACCESS-PASSPHRASE", authenticator.Passphrase);
}

i have added the fix here: https://github.com/dougdellolio/gdax-csharp/pull/32 if you agree, I will release in the next version 😄
thanks so much!

yes..
thanks very much :-)


change of direction

Scenario:
Navigate to the page: "https://satt-token.com/"
Click on the "sign in" button.
Obtained result:
Redirected to the "sign in" page.
Expected result:
Redirected to the "welcome page."
satt.token.webm

the button now redirect to the right page ! , the link was changed for the two languages


Is Libmagic required?

@danielknell this pip library relies on libmagic it seems. I'm trying to decide if this is wise? What is this for? Can we work without it? Is it easy to install on every OS (including windows)?
If we do need it we should look at making this dependency clear in the docs.

libmagic is the mimemagic binding, its required for guessing mime types, the ruby gem also depends on the same library.

@danielknell the ruby gem relies on the mimemagic gem but it does not rely on any other dependencies (including libmagic).
Could you have a look and see if there's any viable option that does not use libmagic? And if not, can we add a note to the docs pointing out this dependency?

@danielknell @cbetta  Guys,
It might be overkill to use a third-party MIME detection library for the general case. Can we assume that the developer knows the media type he's providing?
Also, for those devs who don't know what their media type is (e.g. with those cryptic ones like application/vnd.openxmlformats-officedocument.presentationml.presentation), we have a handy extension-to-media-type conversion tool at https://rest.interfax.net/outbound/help/mediatype.
And for those who don't even know their file extensions, e.g. if they're receiving a buffer, we might point them to a third-party library.
@eyalnevo

@danielknell @if-adam I am trying to find the right balance between:

Ease of install (minimal dependencies)
Ease of use (not having to specify mime type explicitly)
Reliability (it always picking the right mime type for each file)

I'd be happy to consider any option, I just want to make sure @danielknell has considered all.

filename based mime lookup seems fragile enough to be a non starter, and mime types tend to be poorly understood from what i have seen amongst more junior developers, so making it implicit has a bunch of benefits, but the library needs to be installed manually on windows though (its there by default on *nix and in homebrew for mac), which is also  something to balance...
https://github.com/ahupp/python-magic there are some straightforward instructions to get it working on the projects homepage.
iin time should a pure python alternative emerge it would be worth using that, as it stands the options seem to be:

rely on libmagic (possible slight complications on windows)
rely on the path based mime lookup in the python standard library (prone to errors when dealing with temporary files)
have the user provide mime types for every file, requiring a substantial reworking of the api and a deviation from the ruby and node apis.

if no changes are made then the docs probably need an extra paragraph in the installation field to inform people of the dependency.

@danielknell agreed that just using filenames is a no-go. So the only 2 options are:

Use a pure python alternative
Use libmagic

I'd like to make sure we searched for a library for options 1 (I assume you did now and it's not there). Therefore happy to go with option 2.
Please go ahead and add the extra install instructions and notes.

experimented with puremagic, a pure python alternative to libmagic, results were disappointing...

Ah that's sad to hear.


Add integration tests to cover trinity attach web3 interaction

What is wrong?
We already have some form of integration tests that spin up a real Trinity node in the same way a user would, and then scan the output for various issues.
This has proven to be extremely valuable as a last line of defense. For instance, before we had this, we sometimes had bugs slipping through where booting trinity with certain parameters would just crash (e.g. trinity --ropsten --light).
With these tests in place, bugs like this get caught in CI.
It would be extremely valuable to go one step further and have tests that spin up one Trinity node and then run trinity attach in as second command to test out some basic interaction. E.g. attaching to a trinity instance and then running w3.net.version should print out 1.
In fact, I started to write such tests a while back but got stuck at the point where I needed to feed in STDIN.
My attempt can be found here: https://github.com/ethereum/py-evm/pull/1054
How can it be fixed

Figure out how to feed in the commands
Look at https://github.com/ethereum/py-evm/pull/1054 as a starting point
Add a basic test that tests the interaction of trinity attach using something as simple as w3.net.version
Profit

This issue will get a bounty assigned soon. Notice that whoever wants to work on this needs to be prepared to just "figure it out". The most important skill for this issue should be solid Python debugging skills :)

Apology for the late update. There are a few issues that I have to workaround to setup the dev env on my mac and run the test. It's requires more time than I initially expected so I'll release this bounty for now for other people who'd like to take a look at the issue. I'll come back to this when I have more downtime.
For future reference, here are issues I encountered and the workarounds I did although they require proper fix:
I use OSX 10.14.1.
When running pip install -e .[dev], I encountered the following error: error: command 'clang' failed with exit status 1 due to clang not being able to locate leveldb.h even though I installed it via homebrew. To solve this issue, I had to install plyvel which depends on leveldb, separately with the following command where I overwrote the paths: pip install --global-option=build_ext --global-option="-I/usr/local/include" --global-option="-L/usr/local/lib" plyvel.
When running pip install -e .[dev], I encountered another error where it complains that there is a version conflict with pluggy. Some requires a version less than 0.8.0 but 0.8.0 got installed via package dependencies. I resolved it by manually installing pluggy==0.7.1 first.
When running pytest -s tests/trinity/integration/test_trinity_cli.py -k test_web3, I encountered the following error: OSError: AF_UNIX path too long. It's due to OSX using longer prefix path name for $TMPDIR than the test file expected as noted in here.  If you have docker installed, maybe working in an Ubuntu environment through docker would make this simpler.

Hi @cburgdorf @kn I noticed that the bounty went back to open. I applied yesterday but somehow GitCoin bot did not notify it here. I had a look today, rebased the branch against latest master and the approach that I used works more or less out of the box:
{code}
(envpyevm) 04:07:33 {feature/trinity-attach} /freelance/py-evm$ time tox -e py37-trinity-integration
/freelance/py-evm/.tox/py37-trinity-integration/lib/python3.7/site-packages/setuptools/dist.py:470: UserWarning: Normalizing '0.2.0-alpha.34' to '0.2.0a34'
normalized_version,
py37-trinity-integration develop-inst-noop: /freelance/py-evm
py37-trinity-integration installed: apipkg==1.5,argh==0.26.2,asn1crypto==0.24.0,async-generator==1.10,async-lru==0.1.0,asyncio-cancel-token==0.1.0a2,atomicwrites==1.2.1,attrdict==2.0.0,attrs==18.2.0,backcall==0.1.0,bloom-filter==1.3,cachetools==2.1.0,certifi==2018.10.15,cffi==1.11.5,chardet==3.0.4,coincurve==8.0.2,colorama==0.4.0,coverage==4.5.2,cryptography==2.4.2,cytoolz==<IP_ADDRESS>,decorator==4.3.0,docopt==0.6.2,eth-abi==1.2.2,eth-account==0.2.3,eth-bloom==1.0.1,eth-hash==0.2.0,eth-keyfile==0.5.1,eth-keys==0.2.0b3,eth-rlp==0.1.2,eth-typing==2.0.0,eth-utils==1.3.0,execnet==1.5.0,hexbytes==0.1.0,hypothesis==3.69.5,idna==2.7,ifaddr==0.1.4,ipaddress==1.0.22,ipython==6.5.0,ipython-genutils==0.2.0,jedi==0.13.1,jsonschema==2.6.0,lahja==0.9.0,lru-dict==1.1.6,lxml==4.2.5,more-itertools==4.3.0,mypy-extensions==0.4.1,netdisco==2.2.0,netifaces==0.10.7,parsimonious==0.8.1,parso==0.3.1,pathtools==0.1.2,pexpect==4.6.0,pickleshare==0.7.5,pluggy==0.7.1,plyvel==1.0.5,prompt-toolkit==1.0.15,ptyprocess==0.6.0,py==1.7.0,py-ecc==1.4.3,-e git+https://github.com/cryptomental/py-evm.git@2ed32a5d16d0e287add72faed4e84d1c6ebbef4b#egg=py_evm,pycparser==2.19,pycryptodome==3.7.0,pyethash==0.1.27,Pygments==2.2.0,pysha3==1.0.2,pytest==3.6.4,pytest-asyncio==0.9.0,pytest-asyncio-network-simulator==0.1.0a2,pytest-cov==2.5.1,pytest-watch==4.2.0,pytest-xdist==1.18.1,python-dateutil==2.7.5,PyYAML==3.13,requests==2.20.1,rlp==1.0.3,simplegeneric==0.8.1,six==1.11.0,termcolor==1.1.0,toolz==0.9.0,traitlets==4.3.2,trie==1.3.8,-e git+https://github.com/cryptomental/py-evm.git@2ed32a5d16d0e287add72faed4e84d1c6ebbef4b#egg=trinity_peer_count_reporter_plugin&subdirectory=trinity-external-plugins/examples/peer_count_reporter,uPnPClient==0.0.8,urllib3==1.24.1,uvloop==0.11.2,watchdog==0.9.0,wcwidth==0.1.7,web3==4.4.1,websockets==5.0.1,zeroconf==0.21.3
py37-trinity-integration run-test-pre: PYTHONHASHSEED='1940916303'
py37-trinity-integration runtests: commands[0] | pip install -e /freelance/py-evm/trinity-external-plugins/examples/peer_count_reporter
Obtaining file:///freelance/py-evm/trinity-external-plugins/examples/peer_count_reporter
Installing collected packages: trinity-peer-count-reporter-plugin
Found existing installation: trinity-peer-count-reporter-plugin 0.0.0
Uninstalling trinity-peer-count-reporter-plugin-0.0.0:
Successfully uninstalled trinity-peer-count-reporter-plugin-0.0.0
Running setup.py develop for trinity-peer-count-reporter-plugin
Successfully installed trinity-peer-count-reporter-plugin
py37-trinity-integration runtests: commands[1] | pytest -n 1 tests/trinity/integration/ -k 'not lightchain_integration' -k test_web3
============================================================================= test session starts ==============================================================================
platform linux -- Python 3.7.1, pytest-3.6.4, py-1.7.0, pluggy-0.7.1
rootdir: /freelance/py-evm, inifile: pytest.ini
plugins: xdist-1.18.1, cov-2.5.1, asyncio-0.9.0, asyncio-network-simulator-0.1.0a2, hypothesis-3.69.5
gw0 [1]
scheduling tests via LoadScheduling
.Task was destroyed but it is pending!
task: <Task pending coro=<AsyncProcessRunner.kill_after_timeout() done, defined at /freelance/py-evm/trinity/tools/async_process_runner.py:61> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f175b501ca8>()]>>
========================================================================== 1 passed in 13.21 seconds ===========================================================================
___________________________________________________________________________________ summary ____________________________________________________________________________________
py37-trinity-integration: commands succeeded
congratulations :)
{code}
Please let me know if I could continue on this, if so I can squash, rebase, and submit a PR today.

I was fed up staring at ETH price charts and decided to do sth useful instead.
https://github.com/ethereum/py-evm/pull/1516
The solution is to simply use async_process_runner to bootstrap Trinity with dependencies and use pexpect to attach to session and interact with console. Adding new console interaction tests using this approach should be pretty straightforward.

@ceresstation This bounty can be paid out to @cryptomental who has done a great job working out a solution in #1516

@ceresstation I guess you need first to approve me on this one, then I will be able to submit via gitcoin.

Hi @ceresstation , submitted -)

Thanks a lot!


Matrix[] passed to Effect seems to not send updated values

I have a Matrix[] to send to shader
When i first set values it works, but if i edit values of the array, then updated array is never sent
                for (int i = 0; i < n; i++)
                {
                    var invTransform = part.invBoneBindTransforms.values[i];
                    var node = part.invBoneBindTransforms.keys[i];

                    part.bones[i] = invTransform * node.globalTransform;
                }

(...)

effect.Parameters["u_bones"].SetValue(nodePart.bones);

But if i create new instance of the array, then it works and send the updated values
Seems like cached value issue

Here https://github.com/tlgkccampbell/ultraviolet/blob/develop/Source/Ultraviolet.OpenGL/Shared/Graphics/OpenGLEffectParameterData.cs#L94
It compare the reference instead of the value of the array

Yes, I see the problem, here. I can probably add some logic to forcibly invalidate the cache in this situation.
I'm going to be doing some work on Ultraviolet today, so this will be fixed in the next release.

Fix has been merged to develop. Moving forward, calling SetValue() with any array type will always cause the array to be re-uploaded to the graphics device, regardless of it content.


Version retrieval failing since saturday

Decoder is failing due to being fed:

Too many connections
Connection refused

curl -v https://api.biorxiv.org/pub/2017-08-21/2017-08-28/30
*   Trying <IP_ADDRESS>:443...
* Connected to api.biorxiv.org (<IP_ADDRESS>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=api.biorxiv.org
*  start date: Jan 28 02:10:51 2021 GMT
*  expire date: Apr 28 02:10:51 2021 GMT
*  subjectAltName: host "api.biorxiv.org" matched cert's "api.biorxiv.org"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
> GET /pub/2017-08-21/2017-08-28/30 HTTP/1.1
> Host: api.biorxiv.org
> User-Agent: curl/7.75.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Mon, 15 Feb 2021 09:23:16 GMT
< Server: Apache/2.2.15 (Red Hat)
< X-Powered-By: PHP/5.4.45
< Transfer-Encoding: chunked
< Content-Type: application/json
<
* Connection #0 to host api.biorxiv.org left intact
Error : (2002) Connection refused

The whole API is down (but returning 200 OK and an error message in the body).

Contacted the email address provided by @BlueReZZ with him and Sciety team in cc.


RPIP-4 voting weight formula

The formula is stated as:
weight = min( (registrationDate - currentDate) / 100, 1)
First, currentDate is greater than registrationDate, so registrationDate - currentDate is negative. Shouldn't it be currentDate - registrationDate?
I also don't see the unit specified, is it in days, i.e. maximum weight reached 100 days after registration?

Thanks for your review! You are correct that that the weight calculation is reversed. I've submitted a PR to fix this and add clarification for the date variables. Based on this edit, these variables will be integer level precision, which I think is sufficient for this purpose.

Corrected documentation via PR#7


Related Products

As a store manager
I need to associate products with related items
So that customers can discover complementary products and increase the likelihood of cross-selling.
Details and Assumptions

Related products are determined based on category, purchase history, or manual curation.
Each product can have a list of related products.
Related products should be easy to update and manage in the database.

Acceptance Criteria
Given a product's SKU
When I associate a list of related product SKUs in the system
Then these related products should be displayed when viewing the main product

For recommendations team


Can't find an example for use of user_provider_custom_retrieve_method

Can anyone guide me on how to implement the custom method?
if i understand it right:

Create a class that implements: Illuminate\Contracts\Auth\UserProvider
overwrite the retrieveByCredentials(array $credentials) inside the class to match own needs
now customize the Keycloak-Driver to somehow use the new class

Am i right? am i missing something? WOuld be nice to get some help, maybe someone could point me to a relevant tutorial or something. Thanks.

You've probably moved on, but I worked this out the other day.
You don't overwrite the retriveByCredentials method if using user_provider_custom_retrieve_method, that variable defines the name of the additional function you add to your custom UserProvider class. Can name it anything, just supply the name. That function will take two parameters, $token & $credentials.
However there is currently a bug with the package, and it fails to load the user_provider_custom_retrieve_method value, I've created a PR to fix the issue, and add a small additional explanation to make it clearer how you're supposed to use it.


Documentation: update docs for cutting release to include building cilium operator image

Do not forget about the dockerfile location


Alpha releases are not advancing

The first build after changing to use semver 2.0 was alpha.1, which is correct. But the second one was also alpha.1, which is not.

Reopened this as it's only partially working.
In order for the alphas to advance - alpha.1, alpha.2, etc. - we need to apply the new tag to the new commit. We currently do that only at the point where we make a github release. BUT the plan is to not release bunches of alphas on github, only on myget. SO we needed a new way to do that.
This turns out to be a can of worms. I'm using the Cake.Git addin to allow me to tag the latest commit on the build machine. That works fine. But for it to persist, I need to do the equivalent of git push --tags. When I do that, a new build is kicked off recursively, which is not what I want. I can prevent that happening in the workflow file, but I'm not happy with that solution as it requires both the workflow and the script itself to know that alphas are handled specially. Maybe I need an entirely separate script for alphas, with different triggers.
@nunit/engine-team Suggestions welcome!

UPDATE
As a workaround, I have changed the script so that the new tag is pushed to origin only for alpha builds. In this case, the extra build that is triggered is not needed but does no harm. The myget package is simply replaced by an identical package. For beta and full releases on GitHub, the extra build would pose a problem.
I'm continuing to look at this and would still appreciate suggestions from the @nunit/engine-team. My current thoughts are...

Fork this repository so I can experiment in making releases without harm
Split the workflow scripts into two parts: build+test+package and publish+release. There might be two versions of the second script, one for alphas and one for beta, rc and full releases.
The cake script does everything, so it would need to be called by each workflow using appropriate targets.

In the (possibly far) future, I can see moving away entirely from cake and relying on dotnet CLI directly, but that seems like an awfully big step for the moment.

:tada: This issue has been resolved in version 1.3.0 :tada:
The release is available on:

GitHub Release
NuGet Package


Cannot register 2 metrics with the same name: /tensorflow/cc/saved_model/load_attempt_count

This template is for miscellaneous issues not covered by the other issue categories.
For questions on how to work with TensorFlow, or support for problems that are not verified bugs in TensorFlow, please go to StackOverflow.
If you are reporting a vulnerability, please use the dedicated reporting process.
For high-level discussions about TensorFlow, please post to<EMAIL_ADDRESS>for questions about the development or internal workings of TensorFlow, or if you would like to know how to contribute to TensorFlow, please post to<EMAIL_ADDRESS>

1


Implement Job Openings block

Fix #67
Test URLs:

Before: https://main--netcentric--hlxsites.hlx.page/jobs
After: https://jobs--netcentric--hlxsites.hlx.page/jobs

After changing the value in the filter, for me clicking "show more" is always required for any jobs to be shown (otherwise, the list is empty after changing the filter).
Also, after changing the filter, the job count ("showing x of y jobs") didn't reset.

@hannessolo fixed