Execution time improvements

I'm not sure how exactly Github Actions are executed on remote server. I suppose, currently it will guix pull every time it runs, which is a lot of redundant work to do. But there is probably a way to speed up this process. Do you have any idea about that?

I don’t think we can solve this fundamental problem since Guix works best if you have a permanent store, whereas GitHub’s workers are designed to be ephemeral. If you want fast Guix-based building you’d better use Cuirass or a long-running machine with guix installed and just while true; do guix build …; sleep 1m; done.
I tried using the worker cache to store /gnu/store, but it does not work properly and usually it’s too big to be cached.

Hmm, was not aware of Cuirass, will take a look. Thanks!

Maybe we could cache merely the closure of the pulled Guix, or store a pack of Guix and unpack that instead of pulling every time?

I’m not sure how you’d cache a closure of the pulled Guix. Would guix gc -R $(which guix) be enough?
Since we’re installing the lastest guix tarball from cuirass we could also offer to disable pulling completely for those, who are fine with a “reasonably recent” Guix. Afaik it uses the latest Guix package.

I, as original author of this issue, recommend you to move to Guix-powered CI 🙂

@tribals Realistically, how would that work?  We don't have a hosted cuirass service.  The appeal of using Github workflow is that we don't need to take care of the resources and infrastructure needed to deploy CI.

@PromyLOPh not running "guix pull" each time is indeed what I'm thinking about.  I don't think we can avoid the initial installation of Guix -- we still need to create build users, launch the daemon, etc.  Starting from a more recent Guix could be enough to reduce setup time.
The Github cache lets us store up to 10GB.  We cannot snapshot /gnu/store completely, but we might be able to store substitutes corresponding to our environment there, so instead of fetching them from ci.guix.gnu.org we'd import them from the cache.  This fails gracefully as we would fall back to substitutes from ci.guix.gnu.org on a cache miss.

Realistically, how would that work? We don't have a hosted cuirass service.

That is exactly the issue.
Bacisally, we have two options:

hack aroud GitHub CI in order it to better support Guix
use other project hosting service where Guix already supported, so integrating Cuirass (or something other) would be trivial.

I don't want to spent my time to make GitHub more Guix-friendly. So, only option is second option?
I don't think either.
Although, there is project hosting services who support Guix natively (https://capsul.org/, https://sr.ht/), only support is "you get that Guix machine, you do what you want to it". That means you need setup everything yourself - choose CI, install it, mantain it. (Funny thing is that with help of Guix, it will be simplier, but still). This leads us to ultimate reason of the problem: in the end you need some resources to run your CI. Clouds make you think they are "just there", but this is not the case. In the end, you need to run that executable, no matter where.
I prefer to turn it other way around. First, I set up CI on my localhost and make it "just work". So-mentioned Cuirass fits there nicely. And only then, I start to think how to move that setup to some "project hosting provider". At this time, I already have CI working, mostly with help of Guix. Replicating it to other machine is a matter of wrintig "operating system definiton". This way, when I start to looking for "publicly available project hosting provider", all I need is just "server runnig Guix system".

I think this falls outside the scope of a project that provides a Github action to use Guix as part of a Github workflow.

Exactly. But what you want? Running CI, or running Github CI?

Moving hosters is not an option for the project in question, so I do want Github actions.  The alternative is not moving to another hoster but not using Github actions.  (Our packages are already part of Guix, so we are using ci.guix.gnu.org, just not for day to day development.)

@rekado This action is already using the most recent Guix available.
How could we “cache” substitutes? Set up an nginx reverse proxy for ci.guix.gnu.org, which caches file to disk and put that directory into GitHub’s cache?

@PromyLOPh oh, I missed that.  I guess when using the default channels we could do without running "guix pull", thus reducing setup time significantly.
Substitute caching does seem a little gnarly.  Perhaps it would be easier to use "guix archive" to export an environment and cache/import that on subsequent uses.

Another issue with caching is that composite actions cannot depend on other actions it seems. So every user of this action would have to implement his own caching (as already suggested by this action’s workflow).

Anyway, I added an option to disable pulling with v1.3.

Thank you, this is very useful!

@PromyLOPh this may be of interest wrt caching /gnu/store: https://github.com/rikhuijzer/cache-install


Iphone Safari Auto-zooms on Input selection

When you select an input (such as a text input) in Safari on the Iphone, the viewport zooms in and does not return to normal size when editing has been completed.  Two double taps are required to return to the initial viewport size.
Normal size, no zoom:

After input selected, zoomed:

Here are some possible solutions to this (http://stackoverflow.com/questions/2989263/disable-auto-zoom-in-input-text-tag-safari-on-iphone):

Disable zoom in meta tag
Use js events to change meta tag dynamically
Set font-size for inputs to 16px

Changing the font size (#3) seems like the easiest and most reliable way to do this.

How about not optimizing things for mobile webkit?

Since this is default behavior for Mobile Safari, I'm inclined not to mess with it.

The easiest way is to set "tabindex=0" for div at top of page. It works for me.

I know this is long closed, but maybe can we have the default font size of inputs 16px or larger?

I know this is long closed, but maybe can we have the default font size of inputs 16px or larger?

@Haroenv Please file a new ticket instead of necroposting. Thanks.

@Haroenv Please file a new ticket instead of necroposting. Thanks.


[WIP] Adding e2e tests for kube-scheduler-simulator

What type of PR is this?
/kind feature

What this PR does / why we need it:
As part of this change, we are adding e2e testing for kube-scheduler-simulator
Which issue(s) this PR fixes:

Fixes #52
Special notes for your reviewer:
/label tide/merge-method-squash

/remove-lifecycle rotten

/reopen


ENH: Environment Analysis - Animate Wind Gust Distribution

Pull request type
Please check the type of change your PR introduces:

[x] Code base additions (bugfix, features)
[ ] Code maintenance (refactoring, formatting, renaming, tests)
[ ] ReadMe, Docs and GitHub maintenance
[ ] Other (please describe):

Pull request checklist
Please check if your PR fulfills the following requirements, depending on the type of PR:

Code base additions (for bug fixes / features):

[ ] Tests for the changes have been added
[ ] Docs have been reviewed and added / updated if needed
[x] Lint (black rocketpy) has passed locally and any fixes were made
[x] All tests (pytest --runslow) have passed locally

What is the new behavior?

Method EnvironmentAnalysis.animate_wind_gust_distribution_over_average_day is now implemented.
Does this introduce a breaking change?

[ ] Yes
[x] No

@PatrickSampaioUSP, Stano and I performed a first implementation and it is ready for review. Here is a preview of the outcome I got:

When executing animate_wind_gust_distribution_over_average_day i'm receiving the following error:
AttributeError: 'list' object has no attribute 'set_animated'

@MateusStano my review is ready, waiting for your response

When executing animate_wind_gust_distribution_over_average_day i'm receiving the following error:
AttributeError: 'list' object has no attribute 'set_animated'

I am not able to reproduce this error here. Would you mind sending the entire traceback showing which line is triggering this error?
Since I can't reproduce it here, I would guess it has to do with different versions of matplotlib. I am currently running the latest version. I will try older ones to check. If you could also detail which version you are using, that would help.

Going back to Matplotlib version 3.2, which is from March 4, 2020: AttributeError: 'NoneType' object has no attribute '_send_event'

@PatrickSampaioUSP, can you confirm that with and updated version of matplotlib you were able to run the code?

Sorry for the delay, I tried to update the matplotlib version but I'm still receiving an error

Sorry for the delay, I tried to update the matplotlib version but I'm still receiving an error

The stack trace error
stack.txt

Pretty sure I have fixed the error now! After I was inspired by @MateusStano, I was able to reproduce the exact same AttributeError: 'list' object has no attribute 'set_animated' on my PC by running a script instead of a notebook.
It turns out I was missing a * operator in two lines of code... Now that I see this, I don't even know how was it working before on notebooks! Anyways...
@PatrickSampaioUSP and @MateusStano, could you give this PR one more chance, run it and check if there is any problem?
Here is a sample code to make things easier:
from rocketpy.EnvironmentAnalysis import EnvironmentAnalysis
from datetime import datetime

test_analysis = EnvironmentAnalysis(
    start_date=datetime(2010, 6, 15),
    end_date=datetime(2010, 6, 30),
    latitude=33,
    longitude=-107,
    elevation=1400,
    pressureLevelDataFile="data\weather\spaceport_america_pressure_levels_reanalysis_2002_2021.nc",
    surfaceDataFile="data\weather\spaceport_america_single_level_reanalysis_2002_2021.nc",
)

test_analysis.animate_wind_gust_distribution_over_average_day()


Implement UNICODE function

First contribution, thought I'd start with something simple. Let me know if I missed anything.

This is great!!
For what I can see you are just missing a test workbook. You just need to create a workbook with Excel called UNICODE.xlsx and place it here:
https://github.com/ironcalc/IronCalc/tree/main/xlsx/tests/calc_tests
The heavy lifting of the tests should be in that file

Thanks, I completely missed the xlsx tests but they are very useful! Caught a bug writing to the wrong Excel formatting and missing the _xlfn variant of the function. The branch is updated

Hi @ExtraBB , don't worry about the test coverage.
I will get around doing a proper review on my evening (Berlin time)

Awesome, sorry for the late reply. The test function is great but doesn't contain error cases:

Also can you color those cells that have a formula:

Then it is easy to see for instance that A20 has a formula (as opposed to be empty)

You also need to add UNICODE here:
https://github.com/ironcalc/IronCalc/blob/main/base/src/functions/mod.rs#L452-L481

@nhatcher updated with error cases.

Approved and merged!


Add rule type change detection in profile parse

We went through migration cases for the new KubeletConfig changes. We decided to handle the TailoredProfile migration on the operator instead.
This fixes future rule checkType change migration issues as well as rule/variables deprecation issues.
To spot deprecated variables and rules, we'll use annotations derived from the content:

Variable/Rule deprecation: compliance.openshift.io/deprecated

During the profileParser upgrade, we'll detect rule TypeChanges by comparing existing rule types against the new updates. If there's a change, the annotation compliance.openshift.io/rule-last-check-type will be added to the rule.
We will issues event and show warning in the TailoredProfile Status, if there are any deprecated or migrated rules/variables in the TailoredProfile, and user can also choose to set following annotation on the tailoredProfile: compliance.openshift.io/prune-outdated-references, so that the tailoredProfile controller will removal of all deprecated rules and variables along with migrated rules references from that specific tailoredProfile object.
To bypass migration and deprecation validation on specific TailoredProfile object, add the tag: compliance.openshift.io/disable-outdated-reference-validation.
Testing Profiles:

Initial Setup:

Install CO prior to the release of the KubeletConfig patch.
Wait for all the profile bundles to reach a 'ready' state.

Deploying TailoredProfile (TP):

Deploy the TP. This should lead to an 'all ready' state for TP.

Modifying ProfileBundle Deployment:

Remove the profile bundle deployment. Important: Ensure that the profile bundle object remains intact and isn't deleted.

Re-deploying CO with the Patch:

Make sure the operator pod has restarted and ProfileBundle deployment is being created

Validation:

Verify that the TP displays the appropriate states.

Check comment for testing profiles

Testing profiles
---
apiVersion: compliance.openshift.io/v1alpha1
kind: TailoredProfile
metadata:
  name: cis-infra-tp-noextend-disableonly-rules
  namespace: openshift-compliance
spec:
  description: Test
  disableRules:
  - name: ocp4-kubelet-anonymous-auth
    rationale: test
  title: My modified nist profile with a custom value
---
apiVersion: compliance.openshift.io/v1alpha1
kind: TailoredProfile
metadata:
  name: cis-infra-tp-noextend-enableonly-rules
  namespace: openshift-compliance
spec:
  description: Test
  enableRules:
  - name: ocp4-kubelet-anonymous-auth
    rationale: test
  title: My modified nist profile with a custom value
---
apiVersion: compliance.openshift.io/v1alpha1
kind: TailoredProfile
metadata:
  name: cis-infra-tp-noextend-disablemix-rules
  namespace: openshift-compliance
spec:
  description: Test
  disableRules:
  - name: ocp4-kubelet-anonymous-auth
    rationale: test
  - name: ocp4-api-server-insecure-port
    rationale: test
  - name: ocp4-api-server-tls-cert
    rationale: test
  title: My modified nist profile with a custom value
---
apiVersion: compliance.openshift.io/v1alpha1
kind: TailoredProfile
metadata:
  name: cis-infra-tp-noextend-disablemix-enableonly-rules
  namespace: openshift-compliance
spec:
  description: Test
  disableRules:
  - name: ocp4-kubelet-anonymous-auth
    rationale: test
  - name: ocp4-api-server-insecure-port
    rationale: test
  - name: ocp4-api-server-tls-cert
    rationale: test
  enableRules:
  - name: ocp4-kubelet-enable-streaming-connections
    rationale: test
  title: My modified nist profile with a custom value
---
apiVersion: compliance.openshift.io/v1alpha1
kind: TailoredProfile
metadata:
  name: cis-infra-tp-noextend-enablemix-rules
  namespace: openshift-compliance
spec:
  description: Test
  enableRules:
  - name: ocp4-kubelet-anonymous-auth
    rationale: test
  - name: ocp4-api-server-insecure-port
    rationale: test
  - name: ocp4-api-server-tls-cert
    rationale: test
  title: My modified nist profile with a custom value
---
apiVersion: compliance.openshift.io/v1alpha1
kind: TailoredProfile
metadata:
  name: cis-infra-tp-noextend-enablemix-disableonly-rules
  namespace: openshift-compliance
spec:
  description: Test
  enableRules:
  - name: ocp4-kubelet-anonymous-auth
    rationale: test
  - name: ocp4-api-server-insecure-port
    rationale: test
  - name: ocp4-api-server-tls-cert
    rationale: test
  disableRules:
  - name: ocp4-kubelet-enable-streaming-connections
    rationale: test
  title: My modified nist profile with a custom value
---
apiVersion: compliance.openshift.io/v1alpha1
kind: TailoredProfile
metadata:
  name: cis-infra-tp-noextend-enablemix-disablemix-rules
  namespace: openshift-compliance
spec:
  description: Test
  enableRules:
  - name: ocp4-kubelet-anonymous-auth
    rationale: test
  - name: ocp4-api-server-insecure-port
    rationale: test
  - name: ocp4-api-server-tls-cert
    rationale: test
  disableRules:
  - name: ocp4-kubelet-enable-streaming-connections
    rationale: test
  - name: ocp4-kubelet-enable-server-cert-rotation
    rationale: test
  - name: ocp4-idp-is-configured
    rationale: test
  title: My modified nist profile with a custom value
---
apiVersion: compliance.openshift.io/v1alpha1
kind: TailoredProfile
metadata:
  name: cis-infra-tp-noextend-disablemixmany-rules
  namespace: openshift-compliance
spec:
  description: Test
  enableRules:
  - name: ocp4-kubelet-anonymous-auth
    rationale: test
  - name: ocp4-kubelet-enable-streaming-connections
    rationale: test
  - name: ocp4-kubelet-enable-server-cert-rotation
    rationale: test
  - name: ocp4-api-server-insecure-port
    rationale: test
  - name: ocp4-api-server-tls-cert
    rationale: test
  - name: ocp4-idp-is-configured
    rationale: test
  title: My modified nist profile with a custom value
---
apiVersion: compliance.openshift.io/v1alpha1
kind: TailoredProfile
metadata:
  name: cis-infra-tp-disableonly-rules
  namespace: openshift-compliance
spec:
  description: Test
  disableRules:
  - name: ocp4-kubelet-anonymous-auth
    rationale: test
  extends: ocp4-cis
  title: My modified nist profile with a custom value
---
apiVersion: compliance.openshift.io/v1alpha1
kind: TailoredProfile
metadata:
  name: cis-infra-tp-enable-only-rules
  namespace: openshift-compliance
spec:
  description: Test
  enableRules:
  - name: ocp4-kubelet-anonymous-auth
    rationale: test
  extends: ocp4-cis
  title: My modified nist profile with a custom value
---
apiVersion: compliance.openshift.io/v1alpha1
kind: TailoredProfile
metadata:
  name: cis-infra-tp-disablemix-rules
  namespace: openshift-compliance
spec:
  description: Test
  disableRules:
  - name: ocp4-kubelet-anonymous-auth
    rationale: test
  - name: ocp4-api-server-insecure-port
    rationale: test
  - name: ocp4-api-server-tls-cert
    rationale: test
  extends: ocp4-cis
  title: My modified nist profile with a custom value
---
apiVersion: compliance.openshift.io/v1alpha1
kind: TailoredProfile
metadata:
  name: cis-infra-tp-disablemix-enableonly-rules
  namespace: openshift-compliance
spec:
  description: Test
  disableRules:
  - name: ocp4-kubelet-anonymous-auth
    rationale: test
  - name: ocp4-api-server-insecure-port
    rationale: test
  - name: ocp4-api-server-tls-cert
    rationale: test
  enableRules:
  - name: ocp4-kubelet-enable-streaming-connections
    rationale: test
  extends: ocp4-cis
  title: My modified nist profile with a custom value
---
apiVersion: compliance.openshift.io/v1alpha1
kind: TailoredProfile
metadata:
  name: cis-infra-tp-enablemix-rules
  namespace: openshift-compliance
spec:
  description: Test
  enableRules:
  - name: ocp4-kubelet-anonymous-auth
    rationale: test
  - name: ocp4-api-server-insecure-port
    rationale: test
  - name: ocp4-api-server-tls-cert
    rationale: test
  extends: ocp4-cis
  title: My modified nist profile with a custom value
---
apiVersion: compliance.openshift.io/v1alpha1
kind: TailoredProfile
metadata:
  name: cis-infra-tp-enablemix-disableonly-rules
  namespace: openshift-compliance
spec:
  description: Test
  enableRules:
  - name: ocp4-kubelet-anonymous-auth
    rationale: test
  - name: ocp4-api-server-insecure-port
    rationale: test
  - name: ocp4-api-server-tls-cert
    rationale: test
  disableRules:
  - name: ocp4-kubelet-enable-streaming-connections
    rationale: test
  extends: ocp4-cis
  title: My modified nist profile with a custom value
---
apiVersion: compliance.openshift.io/v1alpha1
kind: TailoredProfile
metadata:
  name: cis-infra-tp-enablemix-disablemix-rules
  namespace: openshift-compliance
spec:
  description: Test
  enableRules:
  - name: ocp4-kubelet-anonymous-auth
    rationale: test
  - name: ocp4-api-server-insecure-port
    rationale: test
  - name: ocp4-api-server-tls-cert
    rationale: test
  disableRules:
  - name: ocp4-kubelet-enable-streaming-connections
    rationale: test
  - name: ocp4-kubelet-enable-server-cert-rotation
    rationale: test
  - name: ocp4-idp-is-configured
    rationale: test
  extends: ocp4-cis
  title: My modified nist profile with a custom value
---
apiVersion: compliance.openshift.io/v1alpha1
kind: TailoredProfile
metadata:
  name: cis-infra-tp-disablemixmany-rules
  namespace: openshift-compliance
spec:
  description: Test
  enableRules:
  - name: ocp4-kubelet-anonymous-auth
    rationale: test
  - name: ocp4-kubelet-enable-streaming-connections
    rationale: test
  - name: ocp4-kubelet-enable-server-cert-rotation
    rationale: test
  - name: ocp4-api-server-insecure-port
    rationale: test
  - name: ocp4-api-server-tls-cert
    rationale: test
  - name: ocp4-idp-is-configured
    rationale: test
  extends: ocp4-cis
  title: My modified nist profile with a custom value
---

/hold

Didn't get expected result with commit https://github.com/ComplianceAsCode/compliance-operator/commit/66a2bc6be7024a89bc726cb8f5c5609d23dda65e
$ oc get tp
NAME                                                STATE
cis-infra-tp-disablemix-enableonly-rules            ERROR
cis-infra-tp-disablemix-rules                       ERROR
cis-infra-tp-disablemixmany-rules                   ERROR
cis-infra-tp-disableonly-rules                      READY
cis-infra-tp-enable-only-rules                      READY
cis-infra-tp-enablemix-disablemix-rules             ERROR
cis-infra-tp-enablemix-disableonly-rules            ERROR
cis-infra-tp-enablemix-rules                        ERROR
cis-infra-tp-noextend-disablemix-enableonly-rules   ERROR
cis-infra-tp-noextend-disablemix-rules              ERROR
cis-infra-tp-noextend-disablemixmany-rules          ERROR
cis-infra-tp-noextend-disableonly-rules             ERROR
cis-infra-tp-noextend-enablemix-disablemix-rules    ERROR
cis-infra-tp-noextend-enablemix-disableonly-rules   ERROR
cis-infra-tp-noextend-enablemix-rules               ERROR
cis-infra-tp-noextend-enableonly-rules              READY

cis-infra-tp-disablemix-enableonly-rules            ERROR
cis-infra-tp-disablemix-rules                       ERROR
cis-infra-tp-disablemixmany-rules                   ERROR

I think you need to rerun the updated PB so the rule with get updated with correct annotation, it will be like a operator upgrade

/retest

@rhmdnd thanks for all the review, I think I have addressed all the comments

/hold for test

@Vincent056 I got the same result as before.  Details seen from https://docs.google.com/document/d/1R71db5EAL0O59JyYzYNfpzAMOZV4BSIqADS6-9NuzzM/edit#heading=h.h0v9uctdh6gy. Could you please help to double check what is wrong? Thanks.
Logs is available at: https://drive.google.com/drive/folders/1ZrDMZr92DpdLGfLMyjjzasm4WqUKzEDR
There are mainly two issues.
1. After co upgrade, the pb will become PENDING once it become VALID
$ oc get csv -w
NAME                         DISPLAY               VERSION   REPLACES                     PHASE
compliance-operator.v1.1.0   Compliance Operator   1.1.0                                  Replacing
compliance-operator.v1.2.0   Compliance Operator   1.2.0     compliance-operator.v1.1.0   Pending
compliance-operator.v1.2.0   Compliance Operator   1.2.0     compliance-operator.v1.1.0   InstallReady
compliance-operator.v1.2.0   Compliance Operator   1.2.0     compliance-operator.v1.1.0   Installing
compliance-operator.v1.2.0   Compliance Operator   1.2.0     compliance-operator.v1.1.0   Installing
compliance-operator.v1.2.0   Compliance Operator   1.2.0     compliance-operator.v1.1.0   Succeeded
compliance-operator.v1.1.0   Compliance Operator   1.1.0                                  Deleting
compliance-operator.v1.1.0   Compliance Operator   1.1.0                                  Deleting
^C$ oc get pb -w
NAME     CONTENTIMAGE                                 CONTENTFILE         STATUS
ocp4     ghcr.io/complianceascode/k8scontent:latest   ssg-ocp4-ds.xml     PENDING
rhcos4   ghcr.io/complianceascode/k8scontent:latest   ssg-rhcos4-ds.xml   PENDING
ocp4     ghcr.io/complianceascode/k8scontent:latest   ssg-ocp4-ds.xml     VALID
ocp4     ghcr.io/complianceascode/k8scontent:latest   ssg-ocp4-ds.xml     PENDING
rhcos4   ghcr.io/complianceascode/k8scontent:latest   ssg-rhcos4-ds.xml   VALID
rhcos4   ghcr.io/complianceascode/k8scontent:latest   ssg-rhcos4-ds.xml   PENDING
$ oc get pod -w
NAME                                              READY   STATUS      RESTARTS   AGE
compliance-operator-5485c4d97d-swqh2              1/1     Running     0          69s
ocp4-openshift-compliance-pp-594c9df46c-hjlxz     1/1     Running     0          61s
openscap-ocp4-ds-1-build                          0/1     Completed   0          137m
rhcos4-openshift-compliance-pp-57fc8d779c-9zl59   1/1     Running     0          133m
rhcos4-openshift-compliance-pp-7d5db749fc-tzknl   0/1     Init:1/2    0          61s
rhcos4-openshift-compliance-pp-7d5db749fc-tzknl   0/1     PodInitializing   0          78s
rhcos4-openshift-compliance-pp-7d5db749fc-tzknl   1/1     Running           0          81s
rhcos4-openshift-compliance-pp-57fc8d779c-9zl59   1/1     Terminating       0          134m
rhcos4-openshift-compliance-pp-57fc8d779c-9zl59   0/1     Terminating       0          134m
rhcos4-openshift-compliance-pp-57fc8d779c-9zl59   0/1     Terminating       0          134m
rhcos4-openshift-compliance-pp-57fc8d779c-9zl59   0/1     Terminating       0          134m
rhcos4-openshift-compliance-pp-57fc8d779c-9zl59   0/1     Terminating       0          134m
^C$ oc get pod
NAME                                              READY   STATUS      RESTARTS   AGE
compliance-operator-5485c4d97d-swqh2              1/1     Running     0          2m2s
ocp4-openshift-compliance-pp-594c9df46c-hjlxz     1/1     Running     0          114s
openscap-ocp4-ds-1-build                          0/1     Completed   0          138m
rhcos4-openshift-compliance-pp-7d5db749fc-tzknl   1/1     Running     0          114s

2. There are changes for tp status.  As a result,  the custom no extends TailoredProfile does not have any rules enabled. It is in ERROR status.
- apiVersion: compliance.openshift.io/v1alpha1
  kind: TailoredProfile
  metadata:
...
    name: cis-infra-tp-noextend-disablemix-enableonly-rules
    namespace: openshift-compliance
...
  spec:
    description: Test
    disableRules:
    - name: ocp4-api-server-insecure-port
      rationale: test
    - name: ocp4-api-server-tls-cert
      rationale: test
    title: My modified nist profile with a custom value
  status:
    errorMessage: Custom no extends TailoredProfile does not have any rules enabled
    id: xccdf_compliance.openshift.io_profile_cis-infra-tp-noextend-disablemix-enableonly-rules
    outputRef:
      name: cis-infra-tp-noextend-disablemix-enableonly-rules-tp
      namespace: openshift-compliance
    state: ERROR

/lable qe-approved
/unhold

/retest

I have just noted some minor nits, otherwise looks good to me.

thanks for the review, just addressed all the issue here

/test e2e-aws-parallel

The test cases look good to me. Continuing to review the actual migration logic.

/retest
Failed getting clusters for the parallel and serial e2e tests.

/hold for test

compliance.openshift.io/prune-outdated-references

@xiaojiey you need to add compliance.openshift.io/prune-outdated-references to remove migrated reference.

compliance.openshift.io/prune-outdated-references

Adding more test results:

after upgrade:
$ oc get rules -n openshift-compliance -o json | jq -r '.items[] | select(.metadata.annotations."compliance.openshift.io/rule-last-check-type" == "Platform" or .metadata.annotations."compliance.openshift.io/rule-last-check-type" == "Node") | .metadata.name'
ocp4-kubelet-anonymous-auth
ocp4-kubelet-authorization-mode
ocp4-kubelet-configure-client-ca
ocp4-kubelet-configure-event-creation
ocp4-kubelet-configure-tls-cipher-suites
ocp4-kubelet-enable-cert-rotation
ocp4-kubelet-enable-client-cert-rotation
ocp4-kubelet-enable-iptables-util-chains
ocp4-kubelet-enable-server-cert-rotation
ocp4-kubelet-enable-streaming-connections
ocp4-kubelet-eviction-thresholds-set-hard-imagefs-available
ocp4-kubelet-eviction-thresholds-set-hard-imagefs-inodesfree
ocp4-kubelet-eviction-thresholds-set-hard-memory-available
ocp4-kubelet-eviction-thresholds-set-hard-nodefs-available
ocp4-kubelet-eviction-thresholds-set-hard-nodefs-inodesfree
ocp4-kubelet-eviction-thresholds-set-soft-imagefs-available
ocp4-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree
ocp4-kubelet-eviction-thresholds-set-soft-memory-available
ocp4-kubelet-eviction-thresholds-set-soft-nodefs-available
ocp4-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree
ocp4-kubelet-read-only-port-secured

Check tp status after adding compliance.openshift.io/prune-outdated-references: "true" for each tailoredprofile:

$ oc get tp
NAME                                                STATE
cis-infra-tp-disablemix-enableonly-rules            READY
cis-infra-tp-disablemix-rules                       READY
cis-infra-tp-disablemixmany-rules                   READY
cis-infra-tp-disableonly-rules                      READY
cis-infra-tp-enable-only-rules                      READY
cis-infra-tp-enablemix-disablemix-rules             READY
cis-infra-tp-enablemix-disableonly-rules            READY
cis-infra-tp-enablemix-rules                        READY
cis-infra-tp-noextend-disablemix-enableonly-rules   ERROR
cis-infra-tp-noextend-disablemix-rules              ERROR
cis-infra-tp-noextend-disablemixmany-rules          READY
cis-infra-tp-noextend-disableonly-rules             ERROR
cis-infra-tp-noextend-enablemix-disablemix-rules    READY
cis-infra-tp-noextend-enablemix-disableonly-rules   READY
cis-infra-tp-noextend-enablemix-rules               READY
cis-infra-tp-noextend-enableonly-rules              ERROR
$ diff tp_before_prune.yaml  tp_after_prune.yaml  | grep -Ev "generation|resourceVersion"
7a8
>       compliance.openshift.io/prune-outdated-references: "true"
11c12
---
21c22
---
26,27d26
<     - name: ocp4-kubelet-anonymous-auth
<       rationale: test
32,34d30
<     enableRules:
<     - name: ocp4-kubelet-enable-streaming-connections
<       rationale: test
43,44d38
<     warnings: |
<       The following rules are migrated and need to be migrated or removed from the TailoredProfile: ocp4-kubelet-anonymous-auth,ocp4-kubelet-enable-streaming-connections
49a44
>       compliance.openshift.io/prune-outdated-references: "true"
53c48
---
63c58
---
68,69d62
<     - name: ocp4-kubelet-anonymous-auth
<       rationale: test
82,83d74
<     warnings: |
<       The following rules are migrated and need to be migrated or removed from the TailoredProfile: ocp4-kubelet-anonymous-auth
126a118
>       compliance.openshift.io/prune-outdated-references: "true"

@Vincent056
@yuumasato and I walked through the code and found a few things to adjust, but we can step through them together next week, too. I just wanted to get the updates we discussed proposed before the break.
Each is its own commit, which we can evaluate and consider incorporating if we all agree they're addressing legitimate concerns.
Overall, we cleaned up some of e2e test to include some additional assertions, added strict checks on the annotation, and reduced the overall patch size by removing untested code for deprecating variables and rules.

/retest

/label qe-approved
Verification pass with 4.15.0-0.nightly-2023-11-28-101923 + code in the PR:
Verification steps:
1. install compliance-operator.v1.1.0
2. apply tp
3. upgrade to compliance-operator.v1.2.0
4. Check the kubelet rule annotations and checkType:
$ oc get rule ocp4-kubelet-anonymous-auth -o=jsonpath={.metadata.annotations}
{"compliance.openshift.io/image-digest":"pb-ocp4f6cjw","compliance.openshift.io/profiles":"ocp4-pci-dss-node,ocp4-stig-node,ocp4-moderate-node,ocp4-cis-node,ocp4-high-node,ocp4-nerc-cip-node","compliance.openshift.io/rule":"kubelet-anonymous-auth","compliance.openshift.io/rule-last-check-type":"Platform","control.compliance.openshift.io/CIS-OCP":"4.2.2","control.compliance.openshift.io/NERC-CIP":"CIP-003-8 R6;CIP-004-6 R3;CIP-007-3 R6.1","control.compliance.openshift.io/NIST-800-53":"CM-6;CM-6(1)","policies.open-cluster-management.io/controls":"CIP-003-8 R6,CIP-004-6 R3,CIP-007-3 R6.1,CM-6,CM-6(1),4.2.2","policies.open-cluster-management.io/standards":"NERC-CIP,NIST-800-53,CIS-OCP"}
$ oc get rule ocp4-kubelet-anonymous-auth -o=jsonpath={.checkType}
Node
5. Check warnings:
$  oc get tp -o=jsonpath={.items[*].status.warnings}
The following rules changed check type and need to be removed from the TailoredProfile. If these rules are important for you, add them to a TailoredProfile of matching check type: ocp4-kubelet-anonymous-auth,ocp4-kubelet-enable-streaming-connections
 The following rules changed check type and need to be removed from the TailoredProfile. If these rules are important for you, add them to a TailoredProfile of matching check type: ocp4-kubelet-anonymous-auth
 The following rules changed check type and need to be removed from the TailoredProfile. If these rules are important for you, add them to a TailoredProfile of matching check type: ocp4-kubelet-anonymous-auth,ocp4-kubelet-enable-streaming-connections,ocp4-kubelet-enable-server-cert-rotation
 The following rules changed check type and need to be removed from the TailoredProfile. If these rules are important for you, add them to a TailoredProfile of matching check type: ocp4-kubelet-anonymous-auth
 The following rules changed check type and need to be removed from the TailoredProfile. If these rules are important for you, add them to a TailoredProfile of matching check type: ocp4-kubelet-anonymous-auth
 The following rules changed check type and need to be removed from the TailoredProfile. If these rules are important for you, add them to a TailoredProfile of matching check type: ocp4-kubelet-enable-streaming-connections,ocp4-kubelet-enable-server-cert-rotation,ocp4-kubelet-anonymous-auth
 The following rules changed check type and need to be removed from the TailoredProfile. If these rules are important for you, add them to a TailoredProfile of matching check type: ocp4-kubelet-enable-streaming-connections,ocp4-kubelet-anonymous-auth
 The following rules changed check type and need to be removed from the TailoredProfile. If these rules are important for you, add them to a TailoredProfile of matching check type: ocp4-kubelet-anonymous-auth
 The following rules changed check type and need to be removed from the TailoredProfile. If these rules are important for you, add them to a TailoredProfile of matching check type: ocp4-kubelet-anonymous-auth,ocp4-kubelet-enable-streaming-connections
 The following rules changed check type and need to be removed from the TailoredProfile. If these rules are important for you, add them to a TailoredProfile of matching check type: ocp4-kubelet-anonymous-auth,ocp4-kubelet-enable-streaming-connections,ocp4-kubelet-enable-server-cert-rotation
 The following rules changed check type and need to be removed from the TailoredProfile. If these rules are important for you, add them to a TailoredProfile of matching check type: ocp4-kubelet-enable-streaming-connections,ocp4-kubelet-enable-server-cert-rotation,ocp4-kubelet-anonymous-auth
 The following rules changed check type and need to be removed from the TailoredProfile. If these rules are important for you, add them to a TailoredProfile of matching check type: ocp4-kubelet-enable-streaming-connections,ocp4-kubelet-anonymous-auth
 The following rules changed check type and need to be removed from the TailoredProfile. If these rules are important for you, add them to a TailoredProfile of matching check type: ocp4-kubelet-anonymous-auth
6. Check the annotation compliance.openshift.io/prune-outdated-references: "true" for each tp. Check the tp change before and after:
$ diff tp_before_prune_1130.yaml tp_after_prune_1130.yaml | grep -Ev "generation|resourceVersion" | grep -E ">|<"
<       compliance.openshift.io/prune-outdated-references: "true"
>       compliance.openshift.io/prune-outdated-references: "true"
<         {"apiVersion":"compliance.openshift.io/v1alpha1","kind":"TailoredProfile","metadata":{"annotations":{},"name":"cis-infra-tp-disablemix-enableonly-rules","namespace":"openshift-compliance"},"spec":{"description":"Test","disableRules":[{"name":"ocp4-kubelet-anonymous-auth","rationale":"test"},{"name":"ocp4-api-server-insecure-port","rationale":"test"},{"name":"ocp4-api-server-tls-cert","rationale":"test"}],"enableRules":[{"name":"ocp4-kubelet-enable-streaming-connections","rationale":"test"}],"extends":"ocp4-cis","title":"My modified nist profile with a custom value"}}
<     - name: ocp4-kubelet-anonymous-auth
<       rationale: test
<     enableRules:
<     - name: ocp4-kubelet-enable-streaming-connections
<       rationale: test
<     warnings: |
<       The following rules changed check type and need to be removed from the TailoredProfile. If these rules are important for you, add them to a TailoredProfile of matching check type: ocp4-kubelet-anonymous-auth,ocp4-kubelet-enable-streaming-connections
<       compliance.openshift.io/prune-outdated-references: "true"
>       compliance.openshift.io/prune-outdated-references: "true"
<         {"apiVersion":"compliance.openshift.io/v1alpha1","kind":"TailoredProfile","metadata":{"annotations":{},"name":"cis-infra-tp-disablemix-rules","namespace":"openshift-compliance"},"spec":{"description":"Test","disableRules":[{"name":"ocp4-kubelet-anonymous-auth","rationale":"test"},{"name":"ocp4-api-server-insecure-port","rationale":"test"},{"name":"ocp4-api-server-tls-cert","rationale":"test"}],"extends":"ocp4-cis","title":"My modified nist profile with a custom value"}}
<     - name: ocp4-kubelet-anonymous-auth
<       rationale: test
<     warnings: |
<       The following rules changed check type and need to be removed from the TailoredProfile. If these rules are important for you, add them to a TailoredProfile of matching check type: ocp4-kubelet-anonymous-auth
<       compliance.openshift.io/prune-outdated-references: "true"
>       compliance.openshift.io/prune-outdated-references: "true"
<         {"apiVersion":"compliance.openshift.io/v1alpha1","kind":"TailoredProfile","metadata":{"annotations":{},"name":"cis-infra-tp-disablemixmany-rules","namespace":"openshift-compliance"},"spec":{"description":"Test","enableRules":[{"name":"ocp4-kubelet-anonymous-auth","rationale":"test"},{"name":"ocp4-kubelet-enable-streaming-connections","rationale":"test"},{"name":"ocp4-kubelet-enable-server-cert-rotation","rationale":"test"},{"name":"ocp4-api-server-insecure-port","rationale":"test"},{"name":"ocp4-api-server-tls-cert","rationale":"test"},{"name":"ocp4-idp-is-configured","rationale":"test"}],"extends":"ocp4-cis","title":"My modified nist profile with a custom value"}}
<     - name: ocp4-kubelet-anonymous-auth
<       rationale: test
<     - name: ocp4-kubelet-enable-streaming-connections
<       rationale: test
<     - name: ocp4-kubelet-enable-server-cert-rotation
<       rationale: test
<     warnings: |
<       The following rules changed check type and need to be removed from the TailoredProfile. If these rules are important for you, add them to a TailoredProfile of matching check type: ocp4-kubelet-anonymous-auth,ocp4-kubelet-enable-streaming-connections,ocp4-kubelet-enable-server-cert-rotation
<       compliance.openshift.io/prune-outdated-references: "true"
>       compliance.openshift.io/prune-outdated-references: "true"
<         {"apiVersion":"compliance.openshift.io/v1alpha1","kind":"TailoredProfile","metadata":{"annotations":{},"name":"cis-infra-tp-disableonly-rules","namespace":"openshift-compliance"},"spec":{"description":"Test","disableRules":[{"name":"ocp4-kubelet-anonymous-auth","rationale":"test"}],"extends":"ocp4-cis","title":"My modified nist profile with a custom value"}}
<     disableRules:
<     - name: ocp4-kubelet-anonymous-auth
<       rationale: test
<       compliance.openshift.io/prune-outdated-references: "true"
>       compliance.openshift.io/prune-outdated-references: "true"
<         {"apiVersion":"compliance.openshift.io/v1alpha1","kind":"TailoredProfile","metadata":{"annotations":{},"name":"cis-infra-tp-enable-only-rules","namespace":"openshift-compliance"},"spec":{"description":"Test","enableRules":[{"name":"ocp4-kubelet-anonymous-auth","rationale":"test"}],"extends":"ocp4-cis","title":"My modified nist profile with a custom value"}}
<     enableRules:
<     - name: ocp4-kubelet-anonymous-auth
<       rationale: test
<       compliance.openshift.io/prune-outdated-references: "true"
>       compliance.openshift.io/prune-outdated-references: "true"
<         {"apiVersion":"compliance.openshift.io/v1alpha1","kind":"TailoredProfile","metadata":{"annotations":{},"name":"cis-infra-tp-enablemix-disablemix-rules","namespace":"openshift-compliance"},"spec":{"description":"Test","disableRules":[{"name":"ocp4-kubelet-enable-streaming-connections","rationale":"test"},{"name":"ocp4-kubelet-enable-server-cert-rotation","rationale":"test"},{"name":"ocp4-idp-is-configured","rationale":"test"}],"enableRules":[{"name":"ocp4-kubelet-anonymous-auth","rationale":"test"},{"name":"ocp4-api-server-insecure-port","rationale":"test"},{"name":"ocp4-api-server-tls-cert","rationale":"test"}],"extends":"ocp4-cis","title":"My modified nist profile with a custom value"}}
<     - name: ocp4-kubelet-enable-streaming-connections
<       rationale: test
<     - name: ocp4-kubelet-enable-server-cert-rotation
<       rationale: test
<     - name: ocp4-kubelet-anonymous-auth
<       rationale: test
<     warnings: |
<       The following rules changed check type and need to be removed from the TailoredProfile. If these rules are important for you, add them to a TailoredProfile of matching check type: ocp4-kubelet-enable-streaming-connections,ocp4-kubelet-enable-server-cert-rotation,ocp4-kubelet-anonymous-auth
<       compliance.openshift.io/prune-outdated-references: "true"
>       compliance.openshift.io/prune-outdated-references: "true"
<         {"apiVersion":"compliance.openshift.io/v1alpha1","kind":"TailoredProfile","metadata":{"annotations":{},"name":"cis-infra-tp-enablemix-disableonly-rules","namespace":"openshift-compliance"},"spec":{"description":"Test","disableRules":[{"name":"ocp4-kubelet-enable-streaming-connections","rationale":"test"}],"enableRules":[{"name":"ocp4-kubelet-anonymous-auth","rationale":"test"},{"name":"ocp4-api-server-insecure-port","rationale":"test"},{"name":"ocp4-api-server-tls-cert","rationale":"test"}],"extends":"ocp4-cis","title":"My modified nist profile with a custom value"}}
<     disableRules:
<     - name: ocp4-kubelet-enable-streaming-connections
<       rationale: test
<     - name: ocp4-kubelet-anonymous-auth
<       rationale: test
<     warnings: |
<       The following rules changed check type and need to be removed from the TailoredProfile. If these rules are important for you, add them to a TailoredProfile of matching check type: ocp4-kubelet-enable-streaming-connections,ocp4-kubelet-anonymous-auth
<       compliance.openshift.io/prune-outdated-references: "true"
>       compliance.openshift.io/prune-outdated-references: "true"
<         {"apiVersion":"compliance.openshift.io/v1alpha1","kind":"TailoredProfile","metadata":{"annotations":{},"name":"cis-infra-tp-enablemix-rules","namespace":"openshift-compliance"},"spec":{"description":"Test","enableRules":[{"name":"ocp4-kubelet-anonymous-auth","rationale":"test"},{"name":"ocp4-api-server-insecure-port","rationale":"test"},{"name":"ocp4-api-server-tls-cert","rationale":"test"}],"extends":"ocp4-cis","title":"My modified nist profile with a custom value"}}
<     - name: ocp4-kubelet-anonymous-auth
<       rationale: test
<     warnings: |
<       The following rules changed check type and need to be removed from the TailoredProfile. If these rules are important for you, add them to a TailoredProfile of matching check type: ocp4-kubelet-anonymous-auth
<       compliance.openshift.io/prune-outdated-references: "true"
>       compliance.openshift.io/prune-outdated-references: "true"
<         {"apiVersion":"compliance.openshift.io/v1alpha1","kind":"TailoredProfile","metadata":{"annotations":{},"name":"cis-infra-tp-noextend-disablemix-enableonly-rules","namespace":"openshift-compliance"},"spec":{"description":"Test","disableRules":[{"name":"ocp4-kubelet-anonymous-auth","rationale":"test"},{"name":"ocp4-api-server-insecure-port","rationale":"test"},{"name":"ocp4-api-server-tls-cert","rationale":"test"}],"enableRules":[{"name":"ocp4-kubelet-enable-streaming-connections","rationale":"test"}],"title":"My modified nist profile with a custom value"}}
<     - name: ocp4-kubelet-anonymous-auth
<       rationale: test
<     enableRules:
<     - name: ocp4-kubelet-enable-streaming-connections
<       rationale: test
>     errorMessage: Custom TailoredProfile with no extends does not have any rules enabled
<     state: READY
>     state: ERROR
<         {"apiVersion":"compliance.openshift.io/v1alpha1","kind":"TailoredProfile","metadata":{"annotations":{},"name":"cis-infra-tp-noextend-disablemix-rules","namespace":"openshift-compliance"},"spec":{"description":"Test","disableRules":[{"name":"ocp4-kubelet-anonymous-auth","rationale":"test"},{"name":"ocp4-api-server-insecure-port","rationale":"test"},{"name":"ocp4-api-server-tls-cert","rationale":"test"}],"title":"My modified nist profile with a custom value"}}
<       compliance.openshift.io/prune-outdated-references: "true"
>       compliance.openshift.io/prune-outdated-references: "true"
<         {"apiVersion":"compliance.openshift.io/v1alpha1","kind":"TailoredProfile","metadata":{"annotations":{},"name":"cis-infra-tp-noextend-disablemixmany-rules","namespace":"openshift-compliance"},"spec":{"description":"Test","enableRules":[{"name":"ocp4-kubelet-anonymous-auth","rationale":"test"},{"name":"ocp4-kubelet-enable-streaming-connections","rationale":"test"},{"name":"ocp4-kubelet-enable-server-cert-rotation","rationale":"test"},{"name":"ocp4-api-server-insecure-port","rationale":"test"},{"name":"ocp4-api-server-tls-cert","rationale":"test"},{"name":"ocp4-idp-is-configured","rationale":"test"}],"title":"My modified nist profile with a custom value"}}
<     - name: ocp4-kubelet-anonymous-auth
<       rationale: test
<     - name: ocp4-kubelet-enable-streaming-connections
<       rationale: test
<     - name: ocp4-kubelet-enable-server-cert-rotation
<       rationale: test
<     warnings: |
<       The following rules changed check type and need to be removed from the TailoredProfile. If these rules are important for you, add them to a TailoredProfile of matching check type: ocp4-kubelet-anonymous-auth,ocp4-kubelet-enable-streaming-connections,ocp4-kubelet-enable-server-cert-rotation
<       compliance.openshift.io/prune-outdated-references: "true"
>       compliance.openshift.io/prune-outdated-references: "true"
<         {"apiVersion":"compliance.openshift.io/v1alpha1","kind":"TailoredProfile","metadata":{"annotations":{},"name":"cis-infra-tp-noextend-disableonly-rules","namespace":"openshift-compliance"},"spec":{"description":"Test","disableRules":[{"name":"ocp4-kubelet-anonymous-auth","rationale":"test"}],"title":"My modified nist profile with a custom value"}}
<       compliance.openshift.io/prune-outdated-references: "true"
>       compliance.openshift.io/prune-outdated-references: "true"
<         {"apiVersion":"compliance.openshift.io/v1alpha1","kind":"TailoredProfile","metadata":{"annotations":{},"name":"cis-infra-tp-noextend-enablemix-disablemix-rules","namespace":"openshift-compliance"},"spec":{"description":"Test","disableRules":[{"name":"ocp4-kubelet-enable-streaming-connections","rationale":"test"},{"name":"ocp4-kubelet-enable-server-cert-rotation","rationale":"test"},{"name":"ocp4-idp-is-configured","rationale":"test"}],"enableRules":[{"name":"ocp4-kubelet-anonymous-auth","rationale":"test"},{"name":"ocp4-api-server-insecure-port","rationale":"test"},{"name":"ocp4-api-server-tls-cert","rationale":"test"}],"title":"My modified nist profile with a custom value"}}
<     - name: ocp4-kubelet-enable-streaming-connections
<       rationale: test
<     - name: ocp4-kubelet-enable-server-cert-rotation
<       rationale: test
<     - name: ocp4-kubelet-anonymous-auth
<       rationale: test
<     warnings: |
<       The following rules changed check type and need to be removed from the TailoredProfile. If these rules are important for you, add them to a TailoredProfile of matching check type: ocp4-kubelet-enable-streaming-connections,ocp4-kubelet-enable-server-cert-rotation,ocp4-kubelet-anonymous-auth
<         {"apiVersion":"compliance.openshift.io/v1alpha1","kind":"TailoredProfile","metadata":{"annotations":{},"name":"cis-infra-tp-noextend-enablemix-disableonly-rules","namespace":"openshift-compliance"},"spec":{"description":"Test","disableRules":[{"name":"ocp4-kubelet-enable-streaming-connections","rationale":"test"}],"enableRules":[{"name":"ocp4-kubelet-anonymous-auth","rationale":"test"},{"name":"ocp4-api-server-insecure-port","rationale":"test"},{"name":"ocp4-api-server-tls-cert","rationale":"test"}],"title":"My modified nist profile with a custom value"}}
<     disableRules:
<     - name: ocp4-kubelet-enable-streaming-connections
<       rationale: test
<     - name: ocp4-kubelet-anonymous-auth
<       rationale: test
<     warnings: |
<       The following rules changed check type and need to be removed from the TailoredProfile. If these rules are important for you, add them to a TailoredProfile of matching check type: ocp4-kubelet-enable-streaming-connections,ocp4-kubelet-anonymous-auth
<         {"apiVersion":"compliance.openshift.io/v1alpha1","kind":"TailoredProfile","metadata":{"annotations":{},"name":"cis-infra-tp-noextend-enablemix-rules","namespace":"openshift-compliance"},"spec":{"description":"Test","enableRules":[{"name":"ocp4-kubelet-anonymous-auth","rationale":"test"},{"name":"ocp4-api-server-insecure-port","rationale":"test"},{"name":"ocp4-api-server-tls-cert","rationale":"test"}],"title":"My modified nist profile with a custom value"}}
<     - name: ocp4-kubelet-anonymous-auth
<       rationale: test
<     warnings: |
<       The following rules changed check type and need to be removed from the TailoredProfile. If these rules are important for you, add them to a TailoredProfile of matching check type: ocp4-kubelet-anonymous-auth
<         {"apiVersion":"compliance.openshift.io/v1alpha1","kind":"TailoredProfile","metadata":{"annotations":{},"name":"cis-infra-tp-noextend-enableonly-rules","namespace":"openshift-compliance"},"spec":{"description":"Test","enableRules":[{"name":"ocp4-kubelet-anonymous-auth","rationale":"test"}],"title":"My modified nist profile with a custom value"}}
<     enableRules:
<     - name: ocp4-kubelet-anonymous-auth
<       rationale: test
>     errorMessage: Custom TailoredProfile with no extends does not have any rules enabled
<     state: READY


Move broker and app samples folder to a separate repository

Currently Akri's repo contains example brokers and apps that are used for Akri demos but are not for production scenarios. They should be moved out of Akri's core repo into a project-akri/examples repo so that akri core can be maintained at a different rate than the examples. For example, security vulnerabilities on the examples are currently being flagged under Akri core. Vulnerabilities on the examples should not reflect on the core repository.

I would love to work on this, though this is a pretty big change so probably with some more help

Thanks @Rishit-dagli! It is a lot to port so let me know how I can help. I created https://github.com/project-akri/examples as a destination for the examples. The main objective is to move over the code, makefiles, dockerfiles, and GitHub workflows that are associated with the sample apps and brokers in the /samples directory

There are many associated files/content in /build (such as akri-python-containers.mk, akri-dotnet-containers.mk, ...) and /build/containers and this intermediate build.
Files in .github/workflows such as this one

@romoh and @bfjelds does moving forward with this migration sound good?

I can have a look too 😊

@Rishit-dagli are you still interested in taking this on or should we do a pass off to @jbpaux?

@Rishit-dagli or @jbpaux, I think this slipped by during the holiday period. We discussed this work in our community meeting today and would like to do the migration soon. Would either of you still be willing to take it on? I am happy to pair on this.
We were thinking:

for now: separate repo for akri-samples and akri-brokers
copy code to each of those repos
ensure workflows are building containers
decide versioning strategy -- we will need to investigate this more
remove code from akri

I can still have a look at it, maybe more next week. Would be great to have the PR for node update merged before as the workflows are for some of them related.


[function] array_subscript

although starrocks already have the array_generate in the latest version, which generate an array using steps, it's better to have a function to simplify the process to generate an array's subscript. By using array_generate, you can achieve this by array_generate(1, array_length(arr),1)
syntax
array_subscript(arr)

in which,
the arr can be any type of array, including array<array<>>. The function will return the subscript of the array.
examples
select array_subscript([2,5,3,7])
[1,2,3,4]

select array_subscript([[1,2],[0,3],[4,7]]);
[1,2,3]

Hi @kateshaowanjou , can you assign function-array_subscript  to me？ I will make it, Thank you.

@lqkkkkk  no worries！


Added Barcode-Java

"Feel pain, think about pain, accept pain, know pain!"
Literally no one:

John Cena wants to know your location :roll_eyes: :rofl:

I know it’s a mistake, but there are certain things in life where you know it’s a mistake, but you don’t really know it’s a mistake, because the only way to really know it’s a mistake is to make the mistake, and look back and say, ‘Yep, that was a mistake.’ So, really, the bigger mistake would be to not make the mistake because then you’d go your whole life not really knowing if something is a mistake or not. :1st_place_medal:


Role delete_reverseproxy_conf is not doing it's job ?

Hi,
Am trying to delete a stanza value, but the role is returning ok, even though the value it's on the webseal.conf. An nothing get deleted.
Here it's the playbook am running :
- hosts: isam_itaas_internet, isam_itaas_intranet
  connection: local
  tasks:
    - import_role:
        name: isam-ansible-roles/start_config
    - name: Removes the challenge type to be used with basic authentication
      import_role:
        name: isam-ansible-roles/delete_reverseproxy_conf
      vars:
        - delete_reverseproxy_conf_reverseproxy_id : "{{ webseal_oidc_instance_id }}"
        - delete_reverseproxy_conf_entries :
          - stanza_id: "server"
            entry_id: "auth-challenge-type"

Here the chunk of the relevant part of the webseal.conf :
[server]
...
# For example:
#
# auth-challenge-type = [+*MSIE*]ba, [-*MSIE*;+*]forms
#
# This configuration will present a basic authentication challenge to user
# agents containing 'MSIE' (Internet Explorer browsers) and a forms based
# challenge to all other user agents. See the WebSEAL administration guide
# for further information.
#
# Do not use authentication challenge types as a security or enforcement
# measure. If no challenge types can be determined for a given user agent
# string, WebSEAL will fall back to the list of all configured authentication
# mechanisms.
#
# This configuration item may be customized for a particular junction
# by adding the adjusted configuration item to a [server:{jct_id}] stanza,
# where '{jct-id}' refers to the junction point for a standard junction
# (include the leading '/'), or the virtual host label for a virtual host
# junction.
# auth-challenge-type =
auth-challenge-type = [+*MpgOAuthTokens*]ba, [-*MpgOAuthTokens*;+*]forms

And here it's the result :

So i was wandering if there is a known bug when using this role ?
Thanks for the support

delete_reverseproxy_conf role calls that action : ibmsecurity.isam.web.reverse_proxy.configuration.entry.delete even if value_id has not been provided.
After some investigations, it should instead calls ibmsecurity.isam.web.reverse_proxy.configuration.entry.delete_all, or the python lib should forward to delete_all if value_id is empty.

ibmsecurity.isam.web.reverse_proxy.configuration.entry.delete is fixed to handle when value_id is empty.


[SPARK-33152][SQL] This PR proposes a new logic to maintain & track constraints which solves the OOM or performance issues in query compilation due to Constraint Propagation Rule

What changes were proposed in this pull request?
This PR proposes new logic to store the constraint and track aliases in projection which eliminates the need of pessimistically generating all the permutations of a given constraint. It is also more effective in correctly identifying the filters which can be pruned, apart from minimizing the memory used as compared to the current code. This also has changes to push compound filters if the join condition is on multiple attributes and the constraint comprises of more than 1 attributes of the join conditions.
Right now the code retains the old logic of constraint management along with the new logic. It is controlled by a sql conf property "spark.sql.optimizer.optimizedConstraintPropagation.enabled" which is by default true. Once the PR is approved it would make sense to remove the old code & merge the code of ConstraintSet into ExpressionSet and removing certain if else blocks in the Optimizer & the function Optimizer.getAllConstraints and LogicalPlan.getAllValidConstraints.
It is the code of getAllValidConstraints which generates all the permutations of the base constraint.
The new logic is as follows:
In the class ConstraintSet which extends ExpressionSet, we track the aliases , along with the base constraint.
Any constraint which is added to the ConstraintSet is stored in the most canonicalized form (i.e consisting of only those attributes which are part of the output set and NOT the Alias's attribute).
for eg consider a hypothetical plan
Projection1 ( a, a as a1, a as a2, b , b as b1, b as b2, c, a +b as z)
|
Filter ( a + b > 10)
|
base relation (a, b, c, d)
At the node Filter the constraint set will just have constraint a + b > 10
At the Node Projection1 , the constraint set will have
constraint a + b > 10
and maintain following buffers
buff1 -> a , a1.attribute, a2. attribute
buff2 -> b, b1.attribute, b2.attribute
buff3 -> a + b, z.attribute
constraint a + b > 10 is already canonicalized in terms of output attributes.
Now suppose there are two filters on top of projection1
Filter( z > 10) and Filter ( a1 + b2 > 10)
To prune the above two filters, we canonicalize z as a + b ( from the data maintained in the constraintset) & check if the underlying set contains a +b > 10 & so can be pruned.
For Filter a1 + b2 > 10, we identify the buffer to which a1 & b2 belong to and replace it with 0th elements of the buffer, which will yield a +b > 10, and so filter can be pruned.
Now suppose there is another Project2 ( a1, a2, b1, b2, z, c)
i.e say attributes a & b are no longer part of outputset.
The idea is that "as much as possible try to make a constraint survive).
So in Project2 , the atttributes a & b are being eliminated.
we have a constraint a + b > 10 which is dependent on it.
so in the constraintset of the ProjectP2, we update it such that
a + b > 10 becomes ----> a1.attr + b1.attr > 10
buff1 a , a1.attribute, a2. attribute ---> a1.attribute, a2. attribute
buff2 b , b1.attribute, b2. attribute ---> b1.attribute, b2. attribute
buff3 a +b , z.attribute -->. a1.attr + b1.attr , z.attr
This way by tracking aliases & just storing the canonicalized base constraints we can eliminate the need of pessimistically generating all combination of constraints.
For inferring new Filter from constraints ,
we use following logic
New Filter = Filter.constraints -- ( Filter.child.constraints ++ Filter.constraints.convertToCanonicalizedIfRequired(Filter.conditions) )
So the idea is that new filter conditions without redundancy can be obtained by difference of current node's constraints & the child node's constraints & the condition itself properly canonicalized in terms of base attributes which will be part of the output set of filter node.
For inferring new filters for Join push down, we identify all the equality conditions & then the attributes are segregated on the lines of LHS & RHS of joins. So to identify filters for push down on RHS side, we get all equality atttributes of LHS side & ask the constraintset to return all the constraints which are subset of the passed LHS attributes. The LHS attributes are appropriately canonicalized & the constrainst identified.
Once the constraints are know, we can replace the attributes with the corresponding RHS attributes. This helps in identifying the compound filters for push down & not just single attribute filters.
Below is a description of the changes proposed.
ExpressionSet: Apart from adding some new functions, fixed the two bugs in the ExpressionSet where in filter & filterNot, e.canonicalized was being used. e is already canonicalized. Also in very complex expressions canonicalization of a canonicalized object ( especially if contains a join expression ) does not behave correctly in current spark code.
The ExpressionSet has added methods just to retain the existing constraints code.
ConstraintSet: This is the class which does the tracking of the aliases , stores the constraints in the canonicalized form, updates the constraints using available aliases if any of the attribute comprising constraint is getting eliminated. The contains method of this class is used for filter pruning. It also identifies those constraints which can generated new filters for push down in join nodes.
Rest all the changes are just to integrate the new logic as well as retain the old constraints logic.
Pls notice that related to tpcds plan stability , I had to add new golden files for q75. The change as such is trivial.
previously pushed filter was generated as
PushedFilters: [IsNotNull(cr_order_number), IsNotNull(cr_item_sk)]
and with the change it is
PushedFilters: [IsNotNull(cr_item_sk), IsNotNull(cr_order_number)]
This PR also eliminates the need of EqualNullSafe constraints for the alias.
It also is able to handle the literal boolean constraints.
Why are the changes needed?

This issue if not fixed can cause OutOfMemory issue or unacceptable query compilation times.
Added a test "plan equivalence with case statements and performance comparison with benefit of more than 10x conservatively"
in
org.apache.spark.sql.catalyst.plans.OptimizedConstraintPropagationSuite

with this PR the compilation time is 247 ms vs 13958 ms without the change
2) It is more effective in filter pruning as  is evident in some of the tests in org.apache.spark.sql.catalyst.plans.OptimizedConstraintPropagationSuite  where current code is not able to identify the redundant filter in some cases.

It is able to generate a better optimized plan for join queries as it can push compound predicates.

Does this PR introduce any user-facing change?
No
How was this patch tested?
I have been running the target /build/mvn clean install.
As seen below the sql, catalyst modules are passing cleanly.
The hive module reports failure due to 1 suite aborted, but no test failures.
I have been told by my colleagues that there is an issue in Hive module ( unrelated to this change).
I am looking further as to what is causing the abort.
In case if you have any information, please add.
02:25:27.539 WARN org.apache.hadoop.hive.conf.HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
02:25:27.539 WARN org.apache.hadoop.hive.conf.HiveConf: HiveConf of name hive.stats.retries.wait does not exist
Run completed in 2 hours, 27 minutes, 59 seconds.
Total number of tests run: 3123
Suites: completed 108, aborted 1
Tests: succeeded 3123, failed 0, canceled 0, ignored 18, pending 0
*** 1 SUITE ABORTED ***
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 3.1.0-SNAPSHOT:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 17.115 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 21.426 s]
[INFO] Spark Project Sketch ............................... SUCCESS [ 41.413 s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 20.063 s]
[INFO] Spark Project Networking ........................... SUCCESS [01:12 min]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 24.297 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 32.305 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 17.877 s]
[INFO] Spark Project Core ................................. SUCCESS [59:57 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [01:12 min]
[INFO] Spark Project GraphX ............................... SUCCESS [05:22 min]
[INFO] Spark Project Streaming ............................ SUCCESS [08:48 min]
[INFO] Spark Project Catalyst ............................. SUCCESS [16:55 min]
[INFO] Spark Project SQL .................................. SUCCESS [  02:25 h]
[INFO] Spark Project ML Library ........................... SUCCESS [42:14 min]
[INFO] Spark Project Tools ................................ SUCCESS [ 19.337 s]
[INFO] Spark Project Hive ................................. FAILURE [  02:33 h]
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SKIPPED
[INFO] Spark Integration for Kafka 0.10 ................... SKIPPED
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED
[INFO] Spark Avro ......................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  07:18 h
[INFO] Finished at: 2020-10-29T02:27:10-07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.scalatest:scalatest-maven-plugin:2.0.0:test (test) on project spark-hive_2.12: There are test failures -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command

Can one of the admins verify this patch?

Can one of the admins verify this patch?

I think I will have to rebase correctly as it is showing 526 files modified...

The PR is now ready for review.

The hive suit which got aborted is
org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite
Through IDE , below exception is seen.
Looking into it
org.scalatest.exceptions.TestFailedException: spark-submit returned with exit code 1.
Command line: './bin/spark-submit' '--name' 'prepare testing tables' '--master' 'local[2]' '--conf' 'spark.ui.enabled=false' '--conf' 'spark.master.rest.enabled=false' '--conf' 'spark.sql.hive.metastore.version=1.2.1' '--conf' 'spark.sql.hive.metastore.jars=maven' '--conf' 'spark.sql.warehouse.dir=/private/var/folders/vl/gk1hr_957qs1jmbwhmvkq2d80000gp/T/warehouse-33cd9b12-1a96-40cd-afad-4b57880723a6' '--conf' 'spark.sql.test.version.index=0' '--driver-java-options' '-Dderby.system.home=/private/var/folders/vl/gk1hr_957qs1jmbwhmvkq2d80000gp/T/warehouse-33cd9b12-1a96-40cd-afad-4b57880723a6' '/private/var/folders/vl/gk1hr_957qs1jmbwhmvkq2d80000gp/T/test8221341981962562358.py'
2020-10-29 18:20:28.01 - stderr> 20/10/29 18:20:28 WARN Utils: Your hostname, ALTERVICTIMFEWER.workdayinternal.com resolves to a loopback address: <IP_ADDRESS>; using <IP_ADDRESS> instead (on interface en0)
2020-10-29 18:20:28.011 - stderr> 20/10/29 18:20:28 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
2020-10-29 18:20:29.793 - stderr> 20/10/29 18:20:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-10-29 18:20:31.692 - stdout> Traceback (most recent call last):
2020-10-29 18:20:31.693 - stdout>   File "/private/var/folders/vl/gk1hr_957qs1jmbwhmvkq2d80000gp/T/test8221341981962562358.py", line 2, in
2020-10-29 18:20:31.693 - stdout>     from pyspark.sql import SparkSession
2020-10-29 18:20:31.693 - stdout>   File "", line 991, in _find_and_load
2020-10-29 18:20:31.694 - stdout>   File "", line 975, in _find_and_load_unlocked
2020-10-29 18:20:31.694 - stdout>   File "", line 655, in _load_unlocked
2020-10-29 18:20:31.695 - stdout>   File "", line 618, in _load_backward_compatible
2020-10-29 18:20:31.695 - stdout>   File "", line 259, in load_module
2020-10-29 18:20:31.696 - stdout>   File "/private/tmp/test-spark/spark-2.4.7/python/lib/pyspark.zip/pyspark/init.py", line 51, in
2020-10-29 18:20:31.697 - stdout>   File "", line 991, in _find_and_load
2020-10-29 18:20:31.697 - stdout>   File "", line 975, in _find_and_load_unlocked
2020-10-29 18:20:31.698 - stdout>   File "", line 655, in _load_unlocked
2020-10-29 18:20:31.699 - stdout>   File "", line 618, in _load_backward_compatible
2020-10-29 18:20:31.699 - stdout>   File "", line 259, in load_module
2020-10-29 18:20:31.7 - stdout>   File "/private/tmp/test-spark/spark-2.4.7/python/lib/pyspark.zip/pyspark/context.py", line 31, in
2020-10-29 18:20:31.7 - stdout>   File "", line 991, in _find_and_load
2020-10-29 18:20:31.701 - stdout>   File "", line 975, in _find_and_load_unlocked
2020-10-29 18:20:31.701 - stdout>   File "", line 655, in _load_unlocked
2020-10-29 18:20:31.702 - stdout>   File "", line 618, in _load_backward_compatible
2020-10-29 18:20:31.702 - stdout>   File "", line 259, in load_module
2020-10-29 18:20:31.703 - stdout>   File "/private/tmp/test-spark/spark-2.4.7/python/lib/pyspark.zip/pyspark/accumulators.py", line 97, in
2020-10-29 18:20:31.704 - stdout>   File "", line 991, in _find_and_load
2020-10-29 18:20:31.704 - stdout>   File "", line 975, in _find_and_load_unlocked
2020-10-29 18:20:31.705 - stdout>   File "", line 655, in _load_unlocked
2020-10-29 18:20:31.705 - stdout>   File "", line 618, in _load_backward_compatible
2020-10-29 18:20:31.706 - stdout>   File "", line 259, in load_module
2020-10-29 18:20:31.707 - stdout>   File "/private/tmp/test-spark/spark-2.4.7/python/lib/pyspark.zip/pyspark/serializers.py", line 72, in
2020-10-29 18:20:31.708 - stdout>   File "", line 991, in _find_and_load
2020-10-29 18:20:31.709 - stdout>   File "", line 975, in _find_and_load_unlocked
2020-10-29 18:20:31.709 - stdout>   File "", line 655, in _load_unlocked
2020-10-29 18:20:31.71 - stdout>   File "", line 618, in _load_backward_compatible
2020-10-29 18:20:31.71 - stdout>   File "", line 259, in load_module
2020-10-29 18:20:31.711 - stdout>   File "/private/tmp/test-spark/spark-2.4.7/python/lib/pyspark.zip/pyspark/cloudpickle.py", line 145, in
2020-10-29 18:20:31.712 - stdout>   File "/private/tmp/test-spark/spark-2.4.7/python/lib/pyspark.zip/pyspark/cloudpickle.py", line 126, in _make_cell_set_template_code
2020-10-29 18:20:31.712 - stdout> TypeError: an integer is required (got type bytes)
2020-10-29 18:20:31.737 - stderr> log4j:WARN No appenders could be found for logger (org.apache.spark.util.ShutdownHookManager).
2020-10-29 18:20:31.738 - stderr> log4j:WARN Please initialize the log4j system properly.
2020-10-29 18:20:31.

I took the latest master & merged it with this PR and ran the /dev/run-tests target.
I now have a clean run:
=== Metrics of Whole-stage Codegen ===
Total code generation time: 103.481957 seconds
Total compile time:<PHONE_NUMBER>71 seconds
19:44:29.389 WARN org.apache.spark.sql.hive.thriftserver.ThriftServerQueryTestSuite:
===== POSSIBLE THREAD LEAK IN SUITE o.a.s.sql.hive.thriftserver.ThriftServerQueryTestSuite, thread names: rpc-boss-3-1, derby.rawStoreDaemon, com.google.common.base.internal.Finalizer, Timer-3, BoneCP-keep-alive-scheduler, shuffle-boss-6-1, BoneCP-pool-watch-thread =====
[info] ScalaTest
[info] Run completed in 11 hours, 44 minutes, 19 seconds.
[info] Total number of tests run: 466
[info] Suites: completed 19, aborted 0
[info] Tests: succeeded 466, failed 0, canceled 0, ignored 17, pending 0
[info] All tests passed.
[info] Passed: Total 466, Failed 0, Errors 0, Passed 466, Ignored 17
^[[24~^[[24~
:/GC overhead

This still doesn't look right. I'd squash your branch changes and rebase on master.

@srowen Yes, while running the merge with changes locally I accidentally pushed the changes. I will fix it.
I have made some more changes which alleviates the need to generate EqualNullSafe constraints for projection aliases. They were in any case not needed with new logic. I was generating them only to ensure existing spark tests asserting on constraints pass.  I have made some minimal changes in spark & fixed a test in existing  InferFiltersFromConstraintsSuite where it was asserting on a not so optimal filter.

@srowen Fixed the merge issues. I will rebase it into a single commit.

i will fix the scala 2.13 issue

@srowen Hi. This PR is now stabilized.  I have also done away with the need of creating EqualNullSafe constraints for the aliases as with the new logic they are not needed.  I will add a couple of tests more which show that these changes produce a more optimized plan using filter push down of constraints like literal "false" in  outer join conditions as compared to the current code.

attempting to reopen the pr


Changes to SASS are not compiled to a new CSS file

Meteor supports automatic SASS recompilation but Meteoris does not.

I don't know about automatic SASS compile part.

Supposedly we merely need to add fourseven:scss.
Perhaps someone else will step in to help with that.


Need help in figuring out why sometimes on/off command is not sent to all bind devices (TZ-1431)

Question
If it's possible I'd like to get help on the issue I'm having.

I have a presence sensor configured as a router device.
presence sensor is bound directly to 1 light with  own on/off client cluster (and to a coordinator as well)
presence sensor has also a server on/off cluster
presence sensor under cirtain conditions gets an on command from a coordinator that in turn triggers the sensor to issue on command to own binds (aforementioned light and a coordinator itself)

The problem
most of the time it works and 2 commands are issued: to the light and to the coordinator.
However sometimes I can see that only 1 on command to a coordinator is issued, sometimes no commands at all.
Additional info:

the issue doesn't seem to appear when on command is generated upon internal conditions (one of the own sensors are active)
the command is sent with address_mode set to ESP_ZB_APS_ADDR_MODE_DST_ADDR_ENDP_NOT_PRESENT with esp_zb_zcl_on_off_cmd_req API call
there's a callback on command send status assigned via esp_zb_zcl_command_send_status_handler_register and that one gets called without errors.
there's another piece of code reacting to ESP_ZB_CORE_CMD_DEFAULT_RESP_CB_ID. This one doesn't receive the response from the lights (naturally)

I suspect it has something to do with how "busy" the lower level zigbee machinery is.
How could I troubleshoot it? Why mostly command is sent to all bound devices with no problems and sometimes it doesn't get sent at all or just partially?
Apparently lower level code is not able to send the command. Is there some dedicated callback to know when that happens? At the moment I'm catching this situation with a dedicated internal timer that times out.
Does it make sense to manually send dedicated commands to all relevant bound devices? (in other words not with ESP_ZB_APS_ADDR_MODE_DST_ADDR_ENDP_NOT_PRESENT but with e.g. ESP_ZB_APS_ADDR_MODE_16_ENDP_PRESENT)?
I'm attaching also a wireshark capture of the described situation.
The presence sensor has a short address of 0xce27
The target light in question has a short address of 0xe979
I was applying the filter _ws.col.def_src == "0xce27" || _ws.col.def_dst == "0xce27"
Packet 50 denotes a command from a coordinator to 0xce27 that is supposed to trigger on command from a presence sensor 0xce27 to the light 0xe979.
In this case the command is sent to the coordinator in a packet 87 with a sequence number 191.  This sequence number I also see via other means (I report back such cases to zigbee2mqtt via separate attributes) and I can see that I get the send status and waiting for the response.
At packets 199, 200, 201 one can see a repeated attempt after a timeout which this time succeeds.
timeout_tsn191_ce27_to_e979.pcapng.zip
Thanks in advance!
Additional context.
No response

an update
after changing a logic a bit (I was not properly processing command send status before) I'm getting ESP_ERR_TIMEOUT on those cases when I'm attempting to send a command.
Is there something to research more about it or should I just accept the fact that sometimes sending command just fails?
Are there any possible signs/indicators I could use to judge whether it makes sense to send the command right now or wait  (be notified?) until a better moment..?


Tag: Use commit date instead of current date

This will mean the tags will be reproducible, which will be less confusing to the users.
Follows PR #113.

I'm +1 personally.

@praiskup I don't see any opposition, so I'd propose we merge this.

@pkubatrh can we merge this?

I am on the fence regarding which approach is better but since you are both in favour I will not stall this. Its probably just a corner case anyway.
Merging, thanks!


Auto corrected by following Format Ruby Code

Auto corrected by following Format Ruby Code
Click here to configure it on awesomecode.io

Coverage decreased (-0.1%) to 77.403% when pulling b514c28cff444f7c84a3874133089a758beab932 on awesomecode-format-ruby-code-48632 into 69b6680b1a1363bef1f11dfa3c1efa9eebb9d14c on master.