Support "-style" option

Support "-style" option

Why are you using -style instead of -s and --style like all other switches?

Why are you using -style instead of -s and --style like all other switches?

Because clang-format also uses '-style' to configure the style.
https://clang.llvm.org/docs/ClangFormatStyleOptions.html

The new option should match the Python option standard though, and what the other options already are.

Can you extend this to support the "file" option for style? I can start using this again if it added support.

Can you extend this to support the "file" option for style? I can start using this again if it added that support :D .

If you mean that the option "-style=file", already supports it.
However, forms like "-style=clang-format.file" are not the scope of this project.
In this case, I have to modify it directly in clang-format.
Please refer to https://stackoverflow.com/questions/46373858/how-do-i-specify-a-clang-format-file

Can you extend this to support the "file" option for style? I can start using this again if it added that support :D .

If you mean that the option "-style=file", already supports it.
However, forms like "-style=clang-format.file" are not the scope of this project.
In this case, I have to modify it directly in clang-format.
Please refer to https://stackoverflow.com/questions/46373858/how-do-i-specify-a-clang-format-file

Yes I did mean "-style=file", thank you very much!

Would you mind renaming the option --style (double dashes) like other options?
I'm not sure why, but on my system, clang-format actually documents the double dashes options as well, and I suspect they work on other systems too.
$ which clang-format
/usr/bin/clang-format
$ clang-format --version
clang-format version 10.0.1
$ echo 'void f() {}' | clang-format --style=Mozilla
void
f()
{}
$ clang-format --help |& grep -- --style=
  --style=<string>           - Coding style, currently supports:
$ |

Would you mind renaming the option --style (double dashes) like other options?
I'm not sure why, but on my system, clang-format actually documents the double dashes options as well, and I suspect they work on other systems too.
$ which clang-format
/usr/bin/clang-format
$ clang-format --version
clang-format version 10.0.1
$ echo 'void f() {}' | clang-format --style=Mozilla
void
f()
{}
$ clang-format --help |& grep -- --style=
  --style=<string>           - Coding style, currently supports:
$ |

I've processed it. :smiley:

Could you rebase on top of master?
I could do it myself, but I'm afraid to lose your authorship.

Could you rebase on top of master?
I could do it myself, but I'm afraid to lose your authorship.

Oh! I did it!

Thank you!


emacsPackages.elpaPackages: upstream URLs are dead

Describe the bug
nix-build -A emacsPackages.elpaPackages.ada-mode
these derivations will be built:
  /nix/store/gvi60w59pjb9mimdfvqrgzw5phwm82rv-ada-mode-7.1.4.tar.drv
  /nix/store/yp1bsx5s97blz52fi9jf2m7qjdvmnjwa-emacs-ada-mode-7.1.4.drv
building '/nix/store/gvi60w59pjb9mimdfvqrgzw5phwm82rv-ada-mode-7.1.4.tar.drv'...

trying https://elpa.gnu.org/packages/ada-mode-7.1.4.tar
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0   275    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 404
error: cannot download ada-mode-7.1.4.tar from any mirror
builder for '/nix/store/gvi60w59pjb9mimdfvqrgzw5phwm82rv-ada-mode-7.1.4.tar.drv' failed with exit code 1
cannot build derivation '/nix/store/yp1bsx5s97blz52fi9jf2m7qjdvmnjwa-emacs-ada-mode-7.1.4.drv': 1 dependencies couldn't be built
error: build of '/nix/store/yp1bsx5s97blz52fi9jf2m7qjdvmnjwa-emacs-ada-mode-7.1.4.drv' failed

Seems like ELPA is compressing files using lz and subsequently moving everything from $file to $file.lz, since I've seen similar problems elsewhere. The nix-community emacs-overlay seems to not suffer from these issues, but running update-elpa didn't solve the issues here in nixpkgs.
Notify maintainers

cc @adisbladis

This instance is fixed by #131851.


CFP:  Support multiple XDP programs for cilium

Cilium Feature Proposal
When runs cilium in XDP mode, there are some requirements that we want to run multiple XDP programs along with Cilium.
For example:

We may want to capture the pacekts by  Geneve VNI or IPIP innner IP,  but hubble record feature is not powerful enough, it only supports filter packets by 5 tuples.
User want to have some customized metrics for packets ,  but not want to change the cilium codes since there are maintain effort
DDoS protect xdp programs runs before cilium XDP

So  this proposal is for Cilium that can supported an easy way to integrate with other XDP programs.

(For some more details, see https://netdev.bots.linux.dev/netconf/2023/index.html wrt 'bpf_mprog for XDP' in the slides and notes.)

It's not the best, but we are doing our own tail calls, so slightly modifying cilium code base, but keeping our code outside of cilium.

It's not the best, but we are doing our own tail calls, so slightly modifying cilium code base, but keeping our code outside of cilium.

@oblazek
Could you share more?
For my case, tailcall has some limitations.
I need to add tailcall before return in all the xdp programs, so need to change many places of the cilium codes.

@oblazek Could you share more? For my case, tailcall has some limitations. I need to add tailcall before return in all the xdp programs, so need to change many places of the cilium codes.

Hmm there are not many bpf programs that cilium has imo. But we are doing our own logic on the ingress part, so doing a tail call for example here: https://github.com/cilium/cilium/blob/main/bpf/bpf_xdp.c#L205 if that fails to find it - the packet is not dropped and is handled by the next one ep_tail_call(ctx, CILIUM_CALL_IPV4_FROM_NETDEV);


Claude sonnet 3.5

Github just announced claude sonnet preview in their copilots. I think if this feature appear in app would be better so i can switch the best model.

oops it seems #7 have what i need to see


NETOBSERV-1508 add selector, afinity and priority to components

Description
This PR expose NodeSelector, Affinity and PriorityClassName on each netobserv component in their relative advanced sections.
https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector
Example on a 2 Nodes cluster:
oc get nodes
NAME                         STATUS   ROLES    AGE   VERSION
ip-10-0-1-175.ec2.internal   Ready    worker   52m   v1.27.9+e36e183
ip-10-0-1-238.ec2.internal   Ready    worker   51m   v1.27.9+e36e183

oc get pods -n netobserv-privileged
NAME                         READY   STATUS    RESTARTS   AGE
netobserv-ebpf-agent-qs4f7   1/1     Running   0          4s
netobserv-ebpf-agent-t2b92   1/1     Running   0          4s

Set a custom annotation on a node:
kind: Node
apiVersion: v1
metadata:
  name: ip-10-0-1-238.ec2.internal
  labels:
    netobserv: 'true'
...

Set nodeSelector on eBPF agent config:
apiVersion: flows.netobserv.io/v1beta2
kind: FlowCollector
spec:
  agent:
    ebpf:
      advanced:
        nodeSelector:
          netobserv: 'true'
...

oc get pods -n netobserv-privileged
NAME                         READY   STATUS    RESTARTS   AGE
netobserv-ebpf-agent-hwp4c   1/1     Running   0          7m46s

Dependencies
n/a
Checklist
If you are not familiar with our processes or don't know what to answer in the list below, let us know in a comment: the maintainers will take care of that.

[X] Is this PR backed with a JIRA ticket? If so, make sure it is written as a title prefix (in general, PRs affecting the NetObserv/Network Observability product should be backed with a JIRA ticket - especially if they bring user facing changes).
[ ] Does this PR require product documentation?

[ ] If so, make sure the JIRA epic is labelled with "documentation" and provides a description relevant for doc writers, such as use cases or scenarios. Any required step to activate or configure the feature should be documented there, such as new CRD knobs.

[ ] Does this PR require a product release notes entry?

[ ] If so, fill in "Release Note Text" in the JIRA.

[ ] Is there anything else the QE team should know before testing? E.g: configuration changes, environment setup, etc.

[ ] If so, make sure it is described in the JIRA ticket.

QE requirements (check 1 from the list):

[X] Standard QE validation, with pre-merge tests unless stated otherwise.
[ ] Regression tests only (e.g. refactoring with no user-facing change).
[ ] No QE (e.g. trivial change with high reviewer's confidence, or per agreement with the QE team).

Rebased without changes

@jpinsonneau @msherif1234 : on the apply vs create, I'm proposing this workaround: jpinsonneau#4 This removes external/embedded resource descriptions, which allows shrinking the CRD & annotation size and makes the apply pass again.

I guess we loose autocompletion in the UI with this approach right ?
But if it's making the CRD lightweight I'm fine with that. We just need to document with examples these sections so the users are not lost

I guess we loose autocompletion in the UI with this approach right ? But if it's making the CRD lightweight I'm fine with that. We just need to document with examples these sections so the users are not lost

Yes .. or perhaps what we could do, instead of remove the descriptions, is to put a link to these object's doc, such as https://docs.openshift.com/container-platform/4.15/rest_api/autoscale_apis/horizontalpodautoscaler-autoscaling-v2.html

/lgtm

/ok-to-test

Rebased without changes

/label qe-approved


POC remote media playback so that source of audio could be from the controller(?)

POC remote media playback so that source of audio could be from the controller(?)

I could move this to stretch goals, but, I'm going to close it out for now as I don't think it's critical to the POC & VNF build phase.


Try to build grpc ourselves still and upgrade to 1.31

In #190, I got the binaries from APK that I could. This is attempting to build it from source. Since the grpc cli is now only built when we build the tests, I'm not sure this is going to complete within the pipeline timeout of 60s. I'm thinking the APK approach is easier to maintain for us, but wanted to give this option if we think it is better.

Yeah, we hit the 60 minute cap for free builds in azure pipelines with this approach (grpc got about 80% built according to logs). Since this is a public project, I think we can ask microsoft to increase the limit, but the question in my mind is if we want to spend any more time on this. My vote is no... @ido-namely @vtopc what do you think?

@abe545, let's choose solution, which is easier to maintain(APK?).
And about 60 minutes cap - we could switch to the GitHub Actions(looks like there is 6 hours limit - https://docs.github.com/en/actions/getting-started-with-github-actions/about-github-actions#usage-limits)

@abe545, let's choose solution, which is easier to maintain(APK?).
And about 60 minutes cap - we could switch to the GitHub Actions(looks like there is 6 hours limit - https://docs.github.com/en/actions/getting-started-with-github-actions/about-github-actions#usage-limits)

This is a github action! But the APK solution builds in 6 minutes, so I don't think we need to do anything else to upgrade to 1.31


adding more integration tests

reorganized tests, added ability to skip cluster tests if minikube is not ready. To facilitate running tests locally, introduced TEST_NAMESPACE where test resources are created/manipulated so as not to clash with existing minikube resources
Signed-off-by: Jim Ehrismann<EMAIL_ADDRESS>todo in subsequent PR:

[ ] figure out how to activate main menu items via spectron/WebDriverIO
[ ] introduce specific integration test selectors (new classes?)?
[ ] add more cluster operation tests

pod creation test is now failing on CI. Works locally, investigating


Improve prompt to execution plan

The plan creation sometimes doesn't add the needed information  for the action to be executed, hence, making the execution agent lack context
If we pass a complex prompt like "Swap ETH to 0.05 WBTC, then, swap WBTC to 1000 USDC and send 50 USDC to 0x10f8Bf6a479F320ead074411A4b0e7944eA8C9c1", the tasks are consistently not correct; it should explicitly buy 1000 USDC in the second step, rather than selling an amount of WBTC
we should have an "evaluation" step where we ensure that the tasks are correct and make sense with what has been told in the prompt

This issue is being tracked here: https://github.com/polywrap/AutoTx/issues/101


Can't use tab to navigate away from menu

Steps to reproduce:

Navigate to one of the browser menus using ALT + letter (eg ALT + F).
Try to tab away from menu.

Expected:
Tab will change focus to another part of browser - in Firefox this will navigate into the address bar.
Actual:
Nothing happens. A workaround if for user to select select next / previous tab option from the File menu but is slightly cumbersome.

Closing legacy issue - seems to work fine now (macOS)


Feat: Pass optional param with all actions

This would provide an easy way to pass extra data for action extensibility.
Good example would be for an Alert reducer, enabling us to pass "success/fail" messages to be consumed by Alert Effects/Reducers.
This would work for other use cases if we keep optional param as open as possible.

Major refactor though, will have to do it next week.

@JonnyBGod - if you can provide more details, i'll be happy to add this.  i don't have much experience with ngrx though, so i may need some guidance.

I will try to update the ngrx implementation next week.
Did a few changes since this first version.

but for reference related to this issue it should be changed to something like this:
actions:
  actions.create = class implements Action {
    public readonly type = actionTypes['CREATE'];

    constructor(public payload: any, **public meta?: any**) { }
  };

effects:
  protected create: Observable<LoopbackAction> = this.actions$
    .ofType(this.actionTypes.CREATE)
    .mergeMap((action: LoopbackAction) =>
      this.apiService.create(action.payload)
        .map((response) => new this.actions.createSuccess(response, **action.meta**))
        .catch((error) => concat(
          of(new this.actions.createFail(error, **action.meta**)),
          of(new LoopbackErrorActions.error(error, **action.meta**))
        ))
    );

without the ** of course.

Another refactor should be the reducers helper methods.
https://github.com/mean-expert-official/fireloop-starter/blob/develop/webapp/src/app/shared/sdk/reducers/auth.ts#L67-L85
should change to
export const getLoopbackAuthState = (state: any) => state.loopbackAuth;
export const getLoopbackAuthToken = (state: any) => state.loopbackAuth;
export const getLoopbackAuthUser = (state: any) => state.loopbackAuth.user;
export const getLoopbackAuthUserId = (state: any) => state.loopbackAuth.userId;

and similar with other reducers.

@JonnyBGod good to hear you can spend some time on this soon. I hope to get this puppy out asap.
Let's hop on a call when you have time so we can iron out the things and talk through the steps we need to take to finalize.

also still related to this issue, for the meta parameter to validate with TS we have to add the following to BaseModels.ts
export interface LoopbackAction extends Action {
  meta?: any;
}

@beeman Yes, sure will get back to you for the call hopefully mid next week.
Sorry for almost two months of absence here but been busy. But I see you got a few extra hand on it 👍

Yeah the starter is doing great with help from @brannon-darby and @kattsushi. We aim to release a beta version once the ngrx stuff lands in the sdk-builder alpha/beta


Create serverless.com.md

Add new serverless site!

LGTM. But I can't merge this

Woooow. This is cool!

It's online!
I added the "service" tag. I also added a "curated" boolean that is used to sort websites listing, so yours is more likely to be at the top (it's very beautiful, nice work!) ;)


Cannot use iframes on Firefox 99

Description
Hi,
After having updated my Firefox's browser to 99th version, iframes started to not working on my Playcanvas project.
Here's my demo project : https://launch.playcanvas.com/1393301
What I have on Brave :
https://cdn.discordapp.com/attachments/793382333339271178/965613825296584714/Screen_Shot_2022-04-18_at_4.03.21_PM.png
What I have on Safari iOS :
https://cdn.discordapp.com/attachments/942814078575984650/965614171574112276/IMG_7034.PNG
What I have on Firefox :
https://cdn.discordapp.com/attachments/793382333339271178/965614713138466836/Screen_Shot_2022-04-18_at_4.07.48_PM.png
Steps to Reproduce

Go to https://launch.playcanvas.com/1393301
See if the iframe in front of the camera is clear

Have a good day,
Corentin

I'm afraid this isn't an issue with the engine but more of the iframe implementation and/or browser  capability/version


Add a /help/ article documenting what browsers we support

This should be linked to from the page added in #14662, and describe how we aim to support all modern browsers, including Chrome, Firefox, and Edge, etc.
Something like this in terms of content:
https://support.discordapp.com/hc/en-us/articles/213491697-What-are-the-OS-system-requirements-for-Discord-
We can also link there from /apps.
Tagging this as difficult because I think it requires some careful writing to end up with a result that's better than having nothing.

@zulipbot add "good first issue"

@zulipbot claim

@zulipbot claim

@zulipbot claim

@zulipbot claim

@zulipbot claim

@timabbott I'd like to work on this issue.

@Preshh0 there's already a PR that's pretty much ready to merge for this issue, so please go ahead and find a different one to tackle.

Okay, thanks @alya.


Tag enhancement

This pull will resolve and close #11
This pull can supersede and close #10 - it includes the added logging and OpsGenie properties that this pull request added
The System.Text.Json dependency was updated for this request.

@nblumhardt  I've closed #9 and #10 as this PR supersedes them.
This PR will resolve #11 while adding better logging, adding the Seq event properties to the Opsgenie alert, and implementing a suppression setting.


Update helptext in dialogRule.xml

We really should define some standards here. In plugins labels start always (or 90%) with upper case and helptext also starts upper case and ends with a dot (if not an IP address or something like that)

Sure here we go:

first letter uppercase
all other letters preferably lowercase unless some kind of name or acronym
the dot only if multiple sentences are used (so at least two dots in total)

Sounds good?

Does this break translation?
Why a dot only when there is a second sentence .. where does this logic come from?
Perhaps a new issue for this?

yes, it will flush the translation.
bullet points are usually not sentences unless you want to write more elaborate bullet points. I guess it doesn't really matter here if we use a trailing dot
yes please

No, if you want to make consistent progress for the price of invalidating translations your PR should:

Uppercase the first letter of the line touched
add a space after comma for readablity

@mimugmail If we bounce the translation the first letter in uppercase is really not a thing "tbd"
The question about tbd is for form labels, which asks if every first letter of every word is uppercase or not :)


Remove Swarm Classic docs

Remove Swarm Classic docs (https://github.com/docker/docker.github.io/pull/10809)
Update table of contents
Add redirects from the deleted pages to the Swarm Overview page.

@justincormack PTAL. Created a new PR as there was an issue blocking me from pushing additional commits to the existing PR (https://github.com/docker/docker.github.io/pull/10809).


Integration is no longer loaded

The integration is no longer loaded. There is a "Failed to set up" message.
However, the hOn application on the smartphone is working.

Home Assistant Version: 2023.11.3
hOn Integration Version: 0.10.1
pyhOn Version: latest

Errors in the log:
`Logger: homeassistant.config_entries
Source: config_entries.py:402
First occurred: 13:27:06 (2 occurrences)
Last logged: 13:27:15
Error setting up entry<EMAIL_ADDRESS>for hon
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/config_entries.py", line 402, in async_setup
result = await component.async_setup_entry(hass, self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/hon/init.py", line 32, in async_setup_entry
hon = await Hon(
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyhon/hon.py", line 66, in create
await self.setup()
File "/usr/local/lib/python3.11/site-packages/pyhon/hon.py", line 97, in setup
appliances = await self.api.load_appliances()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyhon/connection/api.py", line 77, in load_appliances
async with self._hon.get(f"{const.API_URL}/commands/v1/appliance") as resp:
File "/usr/local/lib/python3.11/contextlib.py", line 204, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyhon/connection/handler/base.py", line 64, in get
async with self._intercept(*args, **kwargs) as response:
File "/usr/local/lib/python3.11/contextlib.py", line 204, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyhon/connection/handler/hon.py", line 61, in _intercept
kwargs["headers"] = await self._check_headers(kwargs.get("headers", {}))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyhon/connection/handler/hon.py", line 51, in _check_headers
await self.auth.authenticate()
File "/usr/local/lib/python3.11/site-packages/pyhon/connection/auth.py", line 260, in authenticate
if not await self._get_token(url):
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyhon/connection/auth.py", line 232, in _get_token
await self._error_logger(response)
File "/usr/local/lib/python3.11/site-packages/pyhon/connection/auth.py", line 96, in _error_logger
raise exceptions.HonAuthenticationError("Can't login")
pyhon.exceptions.HonAuthenticationError: Can't login
`

Hi, did you change your password recently?

Hi, did you change your password recently?

Hi, no, I have never changed my password.
It all happened suddenly, I noticed that the entities were all unavailable, I tried to reload the integration but it didn't work. I later deleted the integration and redid it, but now I no longer have any entities.

I also had this issue. The application on the phone was working.
But I was able to fix it.

Logout from hon Phone App.
Login on hon phone App, (in my case it asked me to change the password).
Change Password
On Home Assistant, refresh the integration. (If it dose not work then remove and add again)

Not working here either, unfortunately changing passwords didn't help. At first I had 2 factor activated so I deactivated it. Changed passwords, logged out and in again on the hOn app, but still the HA integration says it can't login.

Hi! I received the washer dryer today (model hwd100-b14959u1) but I have the same problem since the first configuration! Do you have any news on the resolution?
Thanks in advance, Domenico.

I solved the problem! The authentication failed, probably because I had registered for the application through Google. By creating a new account directly from the app, the HA authentication problem disappeared.


formula_cellar_checks: fix type from #16946

[x] Have you followed the guidelines in our Contributing document?
[ ] Have you checked to ensure there aren't other open Pull Requests for the same change?
[ ] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.

[x] Have you successfully run brew typecheck with your changes locally?
[ ] Have you successfully run brew tests with your changes locally?

Fixing my mistake from #16946.
https://github.com/Homebrew/homebrew-core/actions/runs/8421011317/job/23072966026?pr=166409
  Error: Return value: Expected type T.nilable(String), got type Module with value T::Private::Types::Void::VOID
  Caller: /opt/homebrew/Library/Homebrew/extend/os/mac/formula_cellar_checks.rb:127
  Definition: /opt/homebrew/Library/Homebrew/extend/os/mac/formula_cellar_checks.rb:69

Thanks @cho-m and @Bo98!


Should not fetch the tag of CITA.

We need a new way to show the version docs.
Why shouldn't we to fetch the tag of CITA?
It's not a sync progress. It may show the part of next version, there's a time between with rc and release version, maybe one or two weeks. The new feature of this time merged to develop and merged the the citahub-docs should be on the next version

@zhouyun-zoe We'll discuss it later.

自动发版的时候去读 rc 的 tag？
#67

还是读 正式版本的 tag：

之后可能不会有冻结的 tag https://github.com/cryptape/cita/blob/develop/release_guide.md
文档和代码版本还是要保持一致

这是文档和代码分离的一个问题，我尽量在 cita 这边控制

这边逻辑还是不变好了。
这个可以先关了

https://github.com/cryptape/cita/blob/develop/release_guide.md#citahub-docs-版本更新


requestAnimationFrame issue with IE9- trying to fallback transforms for legacy browsers

I am testing your polyfills on a local stored html site and rAF doesn't do what I want mainly because the polyfill eliminates my filtering options.
I have some fallback set for older than IE9

if ( window && window.atob ) {
  //basically replace translate with left and top positioning
}

but your polyfill adds support for atob and thus forces my IE9- to try to do translate.
At this point I don't know how to split IE9- for different tween properties and I don't know if your polyfill really works in my case.
I also don't consider using Modernizr.
Please advise.

I figured I need to add ie ie8 class via , seems the only solution for now.
Closed.


Improvments to the dateTimeConvert udf

Remove redundant validations, keeping it only at the blockVal level, instead of for every record
Prevent the creation of new DateTimeFormatter for every record
Fix bug which prevented SDF format in input to work
Add support for : in sdf pattern
Remove code which replaced spaces in function parameters
Remove the assumption that time column data type is always long
Allow graceful handling of null values in input, instead of failing

Codecov Report

Merging #1996 into master will decrease coverage by 0.06%.
The diff coverage is 82.35%.

@@            Coverage Diff             @@
##           master    #1996      +/-   ##
==========================================
- Coverage   69.82%   69.75%   -0.07%
==========================================
  Files         820      820
  Lines       39975    39930      -45
  Branches     5372     5377       +5
==========================================
- Hits        27911    27854      -57
- Misses      10214    10228      +14
+ Partials     1850     1848       -2

Impacted Files
Coverage Δ

...din/pinot/pql/parsers/pql2/ast/GroupByAstNode.java
100% <100%> (ø)
:arrow_up:

...ransform/function/DateTimeConversionTransform.java
76% <76.92%> (-9%)
:arrow_down:

...perator/transform/DateTimeConversionEvaluator.java
82.92% <82.92%> (ø)

...inot/common/utils/time/DateTimeFieldSpecUtils.java
94.28% <83.33%> (-2.1%)
:arrow_down:

...core/operator/dociditerators/AndDocIdIterator.java
55.17% <0%> (-31.04%)
:arrow_down:

...ore/query/scheduler/resources/ResourceManager.java
81.81% <0%> (-13.64%)
:arrow_down:

...pinot/core/operator/docidsets/OrBlockDocIdSet.java
84.9% <0%> (-13.21%)
:arrow_down:

...impl/dictionary/FloatOffHeapMutableDictionary.java
80% <0%> (-12.73%)
:arrow_down:

.../impl/dictionary/FloatOnHeapMutableDictionary.java
88.88% <0%> (-6.67%)
:arrow_down:

.../pinot/core/query/scheduler/PriorityScheduler.java
77.46% <0%> (-5.64%)
:arrow_down:

... and 36 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 19cbf16...0d73d3b. Read the comment docs.

Addressed the comments @kishoreg . PLease check

I forgot to click on submit for previous comments. I will review the latest changes now

Moved the DateTimeConvertor classes to another package(com.linkedin.pinot.common.datetime.convertor)
Also introduced classes DateTimeFormatSpec and DateTimeGranularitySpec to hold the getters, constructors and validators, hence getting rid of DateTimeFieldSpecUtils


[Bug] new right menu does not work with arrows, only mouse clicks

Recent versions (this one is 5.11.3) have a new right click menu:

It does not work with arrows. This creates an issue b/c I have AHK macros for exiting and reloading vind everytime I change the .vindrc
Is it fixable? Maybe getting back to the previous menu (such as in version 5.8)?
Thanks

This problem is a lack of functionality on the fluent-tray side. Please wait for an update of fluent-tray.
If you want to exit win-vind externally, win-vind.exe -c "<exit>" is sufficient.

Amazing. Thanks!!

Now fluent-tray supports arrow-selection. https://github.com/pit-ray/fluent-tray/releases/tag/v0.5.0


🧹 Make use of options lists for sentinel

Leverage options lists for better user experience.
Not implemented for the status field as ideally that would be multi-select. I.e. a user could choose 1, all, or any combination of values (New,Active,Closed).

ooh, multi-select from list sounds like a smart thing: https://github.com/frikky/Shuffle/issues/243


User management support

Thanks for open-sourcing the LLM gateway.
Curious to learn if you are going to add support for user management.
E.g., feature where I can share API keys with specific users and create allow-list for which LLMs can be accessed by each specific user key.
Use case could be providing keys to students in a call and allowing them access models on Anyscale and Together. And some students have their keys whitelisted for OpenAI access (per key quota would be great feature).

We have these as virtual keys in the hosted version at Portkey.ai. You should check that out, was built exactly for this purpose.
https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/virtual-keys
We're thinking about open-sourcing some of the middleware we've written for the gateway as well. But we won't be able to offer all the security guarantees, and hence have not done it so far.
Thanks for the suggestion! Let us know if you have something in mind or if the hosted product works.

Got it. I did look into this but maybe I misunderstood the feature.
When I create a virtual key for say Anyscale, I provide my Anyscale API key. I can name the virtual key but this doesn't give me another API key that I can then share with user.
Ideally, I want to do the following;

Share a form with my students where they type in their name, email address and choose a LLM they want API access for.
I upload the spreadsheet to the gateway (or manually enter it, doesn't matter) and for each user I generate an API key that I can share with them.
They use the API key generated n #2 above and make API calls to portkey gateway which then redirects them to the appropriate LLM (LLM to provider will be mapped).

Is this the way virtual-keys work? Because the virtual key I created is a modified version of the virtual key name. And it doesn't seem like this is the key my student can use to make API calls. Do correct me if I've misunderstood.

Hi @harshsinghal yes, this is how virtual keys work on the hosted platfrom!

You can put the same Anyscale API key above and put different virtual key names. This way, you'd have created multiple virtual keys that all map to the same, single Anyscale API key.
Currently, we auto generate the virtual key and its name for you:

You can just copy the "key" from above and share it with your students → They can then use it in their apps directly. This doc explains how.
If you are making the REST API requests on https://api.portkey.ai/v1/chat/completion route for example, you can pass the virtual key with the x-portkey-virtual-key header.

@harshsinghal Does the solution shared by @vrushankportkey solve your problem?

I'm yet to test this out and will update once I have. Thanks a ton for the detailed and prompt response folks.

@harshsinghal checking in! Please let us know if we can help anywhere!

Closing this for now. @harshsinghal if you have any questions, feel free to directly email me at<EMAIL_ADDRESS>


Save and load models

Hello there
Can we save the trained model in this example? Then is it possible to test the model we trained for another environment? How are we going to do? Thus, we can see the success and performance of the trained model more clearly.  Could you help?

Sorry for late reply. I am reconstructing this project so that it can be compatible with arbitrary tasks. Meanwhile, I will provide new examples of model training. Since the old example uses SB3, you can just save model by save(path, exclude=None, include=None).


compound assignment operators

This should work:
$x += 1;

as a shortcut for $x = $x + 1;.
The same for -=, *=, /=. This might be doable just from additions to parser.yy.
A separate ticket can track pre- and post-increment (++$x, $x++), if we want them too.

This might be doable just from additions to parser.yy.

I don't believe so, the nuance here is that you are performing and op and an assignment with a single expression, so basically you are performing two ops at once and this is not something bpftrace does elsewhere that I am aware of.
However, I believe that this can be done through a change to the parser as well as changes to the semantic analyzer and code generation.
There might be a simpler solution I haven't yet thought of, as I don't have a tremendous amount of experience working with yacc (aside from what i've picked up through this project).
I'm going to try and keep this problem in mind as I try and work out a solution to #412 as the two feel highly related.

Proof-of-concept: https://github.com/iovisor/bpftrace/pull/448
We just need to decide if we want to go this way (only editing bison files).
BTW, you might need https://github.com/iovisor/bpftrace/pull/447 to test the compound operator (I found this bug while working on the compound operator).

Oh wow that’s much simpler than what I had in mind!
I wonder if we can do something similar to implement ++ more simply... the only caveat I can think of is the need to implicitly initialize to 0.

I think we can do something similar to ++/-- if we make all uninitialized variables implicitly initialize to 0, regardless of the operation.

I agree, what you're proposing here would also let us close https://github.com/iovisor/bpftrace/issues/440


Update _pod.tpl

This is to fix issue when we specify podLabels in helm values.yaml.  Issue encountered: Error converting YAML to JSON: yaml: line 37: mapping values are not allowed in this context.

Hi @fernandokarnagi.
Thank you for your contribution! Could you double-check whether your patch wasn't addressed already by #510?

Hello @fernandokarnagi thanks a lot for participating in the SonarQube community.
Your fix has been shipped and released into 10.6.1 SonarQube Chart Closing the PR


Merge pull request #1 from robotframework/master

updating to the base repo

This doesn't seem to be a valid pull request.


Rate limit reached

I paid the 9 USD subscription. I didn't use it for long (about 10 minutes) and it already says You've used all your remaining tokens.

+1, just the system tells me my 10M tokens are used up... Crazy!

paid the $9 sub also ... I hit it after a couple of hours ... I was creating a small frontend for a rest based api ... at first I was impressed and all went well but as I started adding to the app, the app stopped deploying to netlifiy last night .... today it says I've hit the rate limit but there were no warnings last night ...

Hey, @Exodo0, @1WorldCapture and @jdwd40! As we have been working on optimizing the tokens usage and accidentally triggered the limit on the accounts that weren't close to the limit.
We've reverted the change so this should now be resolved and you should have your correct limit back. I'll close the issue, but in case a problem persist, please open a new one! Thank you for being an early adopter and for your patience as we're ironing out the experience on bolt!


UE cannot finish PDU establishment request (COTS UE + SRSRAN)

The issue
"Select SMF failed: DNN[internet] is not supported or not subscribed in the slice"
I'm using srsran configured with a plmn id 00101 while UE uses 90170 (with roaming enabled).
I'm using a Samsung A33 5G as the UE.
The srsran and UE work with other cores (open5Gs and OAI-CN) so I don't think that's the issue.
Versions
srsRAN Project 23.10.1
free5gc-compose v3.3.0
UE status when issue occurs
[
  {
    "AccessType": "3GPP_ACCESS",
    "CmState": "CONNECTED",
    "Guti": "00101cafe0000000009",
    "Mcc": "001",
    "Mnc": "01",
    "PduSessions": null,
    "Supi": "imsi-901700000028080",
    "Tac": "000001"
  }
]

Logs
dnn-issue.tar.gz

Hi @gckopper
I can see this in the AMF logs:
2024-02-05T18:27:12.881400195Z [INFO][AMF][Gmm][amf_ue_ngap_id:RU:1,AU:2(3GPP)][supi:SUPI:imsi-901700000028080] Select SMF [snssai: {Sst:1 Sd:010203}, dnn: internet]
2024-02-05T18:27:12.892019380Z [ERRO][AMF][Gmm][amf_ue_ngap_id:RU:1,AU:2(3GPP)][supi:SUPI:imsi-901700000028080] Select SMF failed: DNN[internet] is not supported or not subscribed in the slice[Snssai: {Sst:1 Sd:010203}]

but nothing in the SMF logs. Do you have other logs from the SMF ?

First of, thank you for the response!
Yes, those are all the logs the SMF is producing.
I was also able to figure out this issue, I had to add the phone's plmn to the plmnList of the SMF (in the config/smfcfg.yaml file)!
@@ -28,6 +28,8 @@ configuration:
   plmnList: # the list of PLMN IDs that this SMF belongs to (optional, remove this key when unnecessary)
     - mcc: 001 # Mobile Country Code (3 digits string, digit: 0~9)
       mnc: 01 # Mobile Network Code (2 or 3 digits string, digit: 0~9)
+    - mcc: 901
+      mnc: 70
   locality: area1 # Name of the location where a set of AMF, SMF, PCF and UPFs are located
   pfcp: # the IP address of N4 interface on this SMF (PFCP)
     # addr config is deprecated in smf config v1.0.3, please use the following config

However, there is a new issue.
[ERRO][AMF][Ngap][amf_ue_ngap_id:RU:3,AU:4(3GPP)][ran_addr:<IP_ADDRESS>:42476] Build InitialContextSetupRequest failed : SEQUENCE OF Size is lower than lowerbound

I've attached the pcaps generated by srsRAN once again and now all the core functions have their log level set to trace.
Looking at the wireshark dissection, the 5GSM cause for the rejection was a Network Failure (38).
lowerbound.tar.gz

TL;DR
There is a workaround this issues I was having and I was able to connect the COTS UE to the internet using srsRAN with a USRP. The configuration used is in the attached file.
What's the issue?
The issue seems to be the usage of roaming. The sim card I was using had a PLMN of 90170 and the cellphone (Samsung Galaxy A33 5G) used seems to block connections to this PLMN. Due to my inability to change the PLMN of sim card I was using roaming with a network configured for the test PLMN (00101). However, after changing the phone used to a Oneplus Nord 5G and changing the network to use the same PLMN as the sim card, it worked!
Takeaways

Samsung seems to block some PLMNs like the 90170 even though it has an international MMC

Our Samsung Galaxy A33 5G and my Samsung Galaxy A53 5G demonstrated this behavior.

Oneplus doesn't seem to block the 901 MMC, at least in the Nord 5G model we tested.

Files
These files contain the logs for all core functions, pcaps and configuration files used. The logs for srsRAN were not included because of their size of over 16MB compressed.
free5gc-srsran.tar.gz


Importance of db.Close()

Is db.Close() optional? My main concern is if panic could lead to a corrupt database.

I think is better to call db.Close function to close db. Because it is better to release the resource which is leveldb using, such as filelock, writer and reader of database file.I know what you're worried about.Don't worry, calling the close method will not cause problems with the database. The data in memory will be written to disk by WAL logging, and if you call the OpenDB method again with the same path, the data in memory will be restored to memory by reading the log.


Formatting

Hi @johnsoncodehk, I've been going down this rabbit hole for a few days now and I think I finally figured out what's going on.
You don't like semicolons. That's why you closed these issues without realizing the underlying problem:
https://github.com/johnsoncodehk/volar/issues/417
https://github.com/johnsoncodehk/volar/issues/477
I do like semicolons. The following code example looks fine to me.
<script setup lang="ts">
import { ref } from "vue";

const count = ref(0);
</script>

<template>
  <button @click="count++">{{ count }}</button>
</template>

When I use Volar to format it, and I have "typescript.format.semicolons": "remove", then everything builds fine. {{ count }} stays {{ count }}. Though all of the <script> semicolons go away, which I'm not happy with.
When I use Volar to format it, and I have  "typescript.format.semicolons": "insert", then everything breaks because {{ count }} turns into {{ count; }}.
I tried disabling Volar and using Prettier which fixes the problem. Prettier lets me have my semicolons in <script> and not in <template>. So https://github.com/johnsoncodehk/volar/issues/1078 is part of the problem here. I also realize you never saw this problem because you yourself are not a semicolon person.
Here are a few reasons why it took me so long to figure out what's going on:

I had to figure out that it was the semicolons in templates that were breaking everything. In a big file, the error showed up somewhere seemingly unrelated.
I had to figure out which builtin extensions to disable to prevent and allow Volar to format.
I had to figure out which settings affected the semicolon behavior.
I had to discover that modifying settings and disabling extensions doesn't always get picked up by the formatter unless I reload the entire VS Code window. So it took me a while to figure out how to consistently reproduce and fix the issues.
Takeover mode requiring disabling some of the builtin extensions, but volar using others, was a bit confusing for me.
Folks don't commit their .vscode/settings.json file, so it's hard to compare formatter settings.

All this is isn't your fault. Sorry this is a big dump of text. In the end I have a workaround - to avoid Volar's formatting system and rely on Prettier. Cheers.

Did you just go and fix every issue I mentioned only 34 minutes after I posted this?

I randomly fix. 😅


[KKT] add two folders Sparse and Dense for KKT formulations

cc @amontoison

Thanks @frapac!

Good to be merged?

Merging this PR as this is only cosmetic changes.


There may be an issue with results for set 1 day 18

Hi Robert,
Results from my implementation are correct for my input and are also the same as result.txt for set 2 day 18, but for set 1, results.txt has  :

part 1 = 4192
part 2 = 2520

whereas my implementation returns :

part 1 = 4310
part 2 = 2466

Using another solution (downloaded from AoC reddit thread)  I get the same results as mine for this input.
Of course I'm not 100% sure my code and the one I used to cross-check are not affected by the same flaw... but there may be an issue with set 1 day 18 result.txt.
Thank you and season's greetings,
R.

You are right.
It was a mistake during upöaos and is fixed now
Thanks for notification


unpivot macro doesn't quote column name

I have a table whose column names are quoted, like "BX 759 - long column name".
When I try to unpivot this table, the column names fetched don't contain quotes and the macro doesn't put the quotes in them, thus the macro fails.

Since this is to unpivot as https://github.com/dbt-labs/dbt-utils/pull/135 was for pivot, I'm updating from a bug to a feature request.


What's H

Hi,
I see some code like:
fmt.Printf("CKKS parameters: logN = %d, logSlots = %d, H(%d; %d), logQP = %d, levels = %d, scale= 2^%f, sigma = %f \n", params.LogN(), params.LogSlots(), params.HammingWeight(), btpParams.EphemeralSecretWeight, params.LogQP(), params.QCount(), math.Log2(params.DefaultScale()), params.Sigma())
What's H?  And what are params.HammingWeight(), btpParams.EphemeralSecretWeight?
BR

H is the number of non-zero coefficient in the secret. Please read the documentation of the related structure for more information.


Pytest does not play the tests in the "plugings" directory

Expected Behavior
Pytest and Django's test runner should run the same number of test.
Actual Behavior
There are 6 tests that Django's test runner plays that are not played by pytest. I identified these 6 tests to be the ones for the "large_banner" plugin.
Steps to Reproduce
The cause may be that the "plugins" directory has to be declared somewhere for pytest to consider it?

Pytest uses magic strings to discover tests. tests.py does not conform

pytest will run all files of the form test_*.py or *_test.py in the current directory and its subdirectories. More generally, it follows standard test discovery rules.

from https://docs.pytest.org/en/latest/getting-started.html#run-multiple-tests

Yep ! Actually I fixed it in this PR https://github.com/openfun/richie/pull/92

Fixed in beda3175433592838a6ab7ee41b958f800898762


Add GPU information

It would be nice if we could see which GPUs are being used by each container, even if it doesn't show how much GPU memory or processing it is consuming. Something like GPUs ids or numbers would be really useful.

Docker doesn't provide such kind of stats. See API response on right side:
https://docs.docker.com/engine/api/v1.40/#operation/ContainerStats
You have to ssh into server and use nvtop or something else https://www.cyberciti.biz/open-source/command-line-hacks/linux-gpu-monitoring-and-diagnostic-commands/


Cannot locate templates.speech_recognition.LM.custom_model.CustomModel

Hi,
Thanks a lot for this brand new ASR research toolkit. Such an invaluable and impactful work.
I was following your ASRfromScratch Tutorial. I was able get to LM training step with no problem. However, when I try to run "python train.py RNNLM.yaml" script as instructed, I am receiving the following error:
ImportError: There is no such class as templates.speech_recognition.LM.custom_model.CustomModel
The train.py script possibly cannot locate the custom_model.py. I reckon this is a python issue but I couldn't find an easy solution to fix it. Do you have any idea what might be causing this? Note that I have face similar problems with different scripts when I tried to run other recipes.
For the information, I am running the scripts within a python virtual environment dedicated only to SpeechBrain, but not a conda environment.
Thanks in advance

I just solved the issue. Apparently this was just a path issue. I have made the following changes on RNNLM.yaml within the tempales/LM directory:
line 66
ORIGINAL :
model: !new:templates.speech_recognition.LM.custom_model.CustomModel
MODIFIED
model: !new:custom_model.CustomModel
This change made the script work without any problem. I am still not sure why the original full path did not work on my system.


feature request: typescript rename refactor operation that propogates renames as far as possible

Rename refactoring of typescript performs a rename over the smallest domain it possibly can ...
Given this code:
const example = ({ arg1, arg2 }: { arg1: string; arg2: string }) => {
  console.log({ arg1, arg2 });
};

const exampleCallSite = ({ arg1 }: { arg1: string }) => {
  example({ arg1, arg2: "value2" });
};

triggring rename refactor on either the first or third occurences of arg1 in example produces this:
const example = ({ arg1: renamed, arg2 }: { arg1: string; arg2: string }) => {
  console.log({ arg1: renamed, arg2 });
};

const exampleCallSite = ({ arg1 }: { arg1: string }) => {
  example({ arg1, arg2: "value2" });
};

renaming the second occurrence produces this:
const example = ({ renamed: arg1, arg2 }: { renamed: string; arg2: string }) => {
  console.log({ arg1, arg2 });
};

const exampleCallSite = ({ arg1 }: { arg1: string }) => {
  example({ renamed: arg1, arg2: "value2" });
};

In each of these cases, the refactor performs an operation which changes the name of arg1 in the smallest possible set of places ...
This is I suppose trying to be more safe in that it cuts down on the number of places where the compiler might not find the places that need renaming -- but not propogating the change as far as possible is almost always the "wrong" rename operation for smaller codebases where the code needs to be compatible only with itself ...
I would really like a refactor operation that produced this result:
const example = ({ renamed, arg2 }: { renamed: string; arg2: string }) => {
  console.log({ renamed, arg2 });
};

const exampleCallSite = ({ renamed }: { renamed: string }) => {
  example({ renamed, arg2: "value2" });
};

I would really like a rename refactoring operation which propagates the rename as far as it can rather than as little as it can ... Ideally it would only perform the 'safe rename' style change to adjust call sites for 'external' code -- where 'external' would be some sort of reasonable heuristic for 'a code boundary which the compiler doesn't understand sufficiently well to propogate changes into '  ...
Or perhaps an even simpler and even less safe heuristic -- perform the 'safe rename' only at call-sites of functions defined outside the project.

Our renames try to be as safe as possible by default. You can opt out of this by setting  "typescript.preferences.renameShorthandProperties": false.
After setting this, do you get the behavior you are after?

Thank you @mjbvz !  This is making me much happier!  :)


Usage of safeParseAsync()

Hello,
we're aiming to use async transformation steps in the zod validation within the middleware and we're getting the following error:
"message": "Asynchronous transform encountered during synchronous parse operation. Use .parseAsync instead."

https://github.com/colinhacks/zod#async-transforms
Would it be possible to enhance the processRequestBody etc. functions to allow for those async zod parse calls? or add new ones that use the safeParseAsync function from zod?

@v4ssili In case you still did not find a work around I forked it and fixed this issue and some others.
Check it out: https://github.com/pshaddel/zod-express

thank you @pshaddel that's awesome!


difference between - and -- and ---

Correct usages include:
non-trivial process
Metropolis--Hastings step
The code---which is slow---is located at...
The - symbol is used to hyphenate words, ie, two words that combine to make an adjective.  The -- symbol is used to connect words that represent a relationship (like "push--pull").  The --- symbol is used to separate parenthetical phrases.

I knew the difference between - and --- but I didn't realize that -- meant something besides subtraction! Cool.


Revert #1284

This is a wholesale revert of #1284. The hope is to get AppVeyor working again in order to cut a 1.13.1 release.

Wait, this is not what I meant :-) I really don't want to see this reverted. Our CI is happy, which means it's probably some AppVeyor specific problem.
I hope this can be solved quickly and we can cut a 1.13.1 which would have Windows binaries, we can live without them for 1.13.0 IMHO.

OK, well I'll leave this open until we get the situation resolved in case it turns into a can of worms.

Sounds good! Thanks a lot Colin!

Closing in favor of https://github.com/libuv/libuv/pull/1405.


Fix dependencies and update icon

This PR is a re-package of https://github.com/dokan-dev/dokan-sshfs/pull/4 which added missing dependencies (and the commit from that PR is used) as well as a change of the icon of the build application to a prettier one already in the project.
This closes #4 and #1 by resolving the differences against master as of 658fa6c20, /cc @ration

Thank you for your contribution @sshirokov !
I also updated to dokan 1.0.0-rc4


MAINT: Bessel tweaks

I started making falling and rising factorial in another branch, but put a quick one here for now.

the docstring backslashes are causing issues

I started making falling and rising factorial

scipy.special already has a rising factorial ufunc (poch) but I guess it doesn't have an 'exact' keyword so it's not as useful for this PR, and ufuncs can't easily get keyword args.

Weird I didn't notice that.  This PR needs falling though, not rising.  (Also working on Thiran filters which use rising factorial.)   I also started a falling/rising factorial branch with exact= and broadcasting of inputs, but it's not finished yet.

This PR needs falling though, not rising.

FWIW if it weren't for the lack of exact= I think you could write falling_factorial(2*n - k, n) as poch(n - k + 1, n).

There is a pretty standard procedure for adding keyword arguments to ufuncs in special--write a function for each value of the keyword, stick them in generate_ufuncs.py with underscored names, and then add an extra python wrapper which selects the right function. See e.g. the spherical Bessel functions (in 0.18, not the earlier ones) which have a keyword argument derivative.

Anything else I need to do on this?

It looks reasonable to me, but can you briefly comment on what this accomplishes? Does it make it faster by not computing some redundant values that get cancelled out? Does it make it more accurate somehow?

Does it make it faster by not computing some redundant values that get cancelled out?

Yes, it no longer computes (54321)/(321) to get 5*4, for instance, and no longer computes all of the bessel polynomial when it only needs the last term.

Does it make it more accurate somehow?

Yes, I also changed to calculate the magnitude normalization based on frequency response from poles instead of transfer functions so it's more accurate especially at higher orders.

Great, can you add a comment at least about the accuracy bit in the
relevant place(s) in the code if not already there?

Well it's in the commit message and I removed the previous comment about it being inaccurate at high orders.  Should I make the commit message more verbose?

The problem with only having it in a commit message (as opposed to in a
code comment) is that the next person looking at the code won't see it and
benefit from the expertise/rationale. But if you think it's clear enough
from just looking at the code then it's fine as is.

Ok I elaborated on the commit messages and rebased

Thanks @endolith


minor flaws in OpenSSL-1.1.0 build system [MSVC]

Hi, Everyone,
There is a small feedback about OpenSSL-1.1.0 builds using MSVC:

in file \Configurations\10-main.conf compiler options /Ox /O2 /Ob2 could be reduced to /O2, since it include all of them (see /O2 and /Ox),

there is a message:

It looks like you don't have either nmake.exe or dmake.exe on your PATH,
so you will not be able to execute the commands from a Makefile.  You can
install dmake.exe with the Perl Package Manager by running:

    ppm install dmake

during OpenSSL configuration, regardless, that nmake.exe available in PATH and further build using MSVC run successfully.
But even in opposite case, it's weird. Since nmake.exe exists in the same directory as cl.exe. Thus if nmake not available, it mean that cl not available too and dmake won't help much,

there is some noise in stderr:

Could Not Find c:\libOPENSSL-1.1.0-dev\src\apps\libcrypto-1_1-x64.dll
Could Not Find c:\libOPENSSL-1.1.0-dev\src\test\libcrypto-1_1-x64.dll
Could Not Find c:\libOPENSSL-1.1.0-dev\src\apps\libssl-1_1-x64.dll
Could Not Find c:\libOPENSSL-1.1.0-dev\src\test\libssl-1_1-x64.dll

in the end of nmake run. Inherent to shared builds only, and have any consequences in practice,

file apps\openssl.cnf not copied into --openssldir=... specified directory during installation, unlike of Mingw-w64 builds,

OpenSSL MSVC builds doesn't generate html documentation, unlike of Mingw-w64 builds. Wherein ActivePerl, a recommended PERL package for OpenSSL MSVC builds, contain some "adopted" version of pod2html package,

All above relate to OpenSSL builds using MSVC, thus was reported as a single issue. But let me know, if some of paragraphs worth to be submitted as a separate issue.
Regards,
Alexander

in file \Configurations\10-main.conf compiler options /Ox /O2 /Ob2 could be reduced to /O2, since it include all of them (see /O2 and /Ox),

Good point.

there is a message:

It looks like you don't have either nmake.exe or dmake.exe on your PATH,
so you will not be able to execute the commands from a Makefile.  You can
install dmake.exe with the Perl Package Manager by running:

    ppm install dmake

during OpenSSL configuration, regardless, that nmake.exe available in PATH and further build using MSVC run successfully.

That message isn't generated by us.  What perl distribution and version did you use?

there is some noise in stderr:

Could Not Find c:\libOPENSSL-1.1.0-dev\src\apps\libcrypto-1_1-x64.dll
Could Not Find c:\libOPENSSL-1.1.0-dev\src\test\libcrypto-1_1-x64.dll
Could Not Find c:\libOPENSSL-1.1.0-dev\src\apps\libssl-1_1-x64.dll
Could Not Find c:\libOPENSSL-1.1.0-dev\src\test\libssl-1_1-x64.dll

in the end of nmake run. Inherent to shared builds only, and have any consequences in practice,

Weird...  How did you configure?

file apps\openssl.cnf not copied into --openssldir=... specified directory during installation, unlike of Mingw-w64 builds,

Right you are!  Will fix.

OpenSSL MSVC builds doesn't generate html documentation, unlike of Mingw-w64 builds. Wherein ActivePerl, a recommended PERL package for OpenSSL MSVC builds, contain some "adopted" version of pod2html package,

Yeah, and the same goes on VMS.  I've been meaning to fix that...

All above relate to OpenSSL builds using MSVC, thus was reported as a single issue. But let me know, if some of paragraphs worth to be submitted as a separate issue.

Nah, but might become more than one PR ;-)

That message isn't generated by us. What perl distribution and version did you use?

Used ActivePerl 5.22, as recommended in OpenSSL manual for MSVC builds. Configure output looks like:
    no-zlib         [default]
    no-zlib-dynamic [default]
Configuring for VC-WIN64A

It looks like you don't have either nmake.exe or dmake.exe on your PATH,
so you will not be able to execute the commands from a Makefile.  You can
install dmake.exe with the Perl Package Manager by running:

    ppm install dmake

CC            =cl
CFLAG         =-W3 -wd4090 -Gs0 -GF -Gy -nologo -DOPENSSL_SYS_WIN32 -DWIN32_LEAN_AND_MEAN -DL_ENDIAN -D_CRT_SECURE_NO_DEPRECATE -DUNICODE -D_UNICODE /MD /Ox /O2 /Ob2

   there is some noise in stderr:

Could Not Find c:\libOPENSSL-1.1.0-dev\src\apps\libcrypto-1_1-x64.dll
Could Not Find c:\libOPENSSL-1.1.0-dev\src\test\libcrypto-1_1-x64.dll
Could Not Find c:\libOPENSSL-1.1.0-dev\src\apps\libssl-1_1-x64.dll
Could Not Find c:\libOPENSSL-1.1.0-dev\src\test\libssl-1_1-x64.dll
in the end of nmake run. Inherent to shared builds only, and have any consequences in practice,

Weird... How did you configure?

Command
perl Configure shared VC-WIN64A && nmake
allow to reproduce it. If you're interested to figure out, on what stage of build this happen, here part of log:
dllmain.c
	C:\ProgramData\Perl64\bin\perl.exe util\mkdef.pl "crypto" 32 > libcrypto-1_1-x64.def
	C:\ProgramData\Perl64\bin\perl.exe -i.tmp -pe "s|^LIBRARY\s+crypto32|LIBRARY libcrypto-1_1-x64|;" libcrypto-1_1-x64.def
	DEL libcrypto-1_1-x64.def.tmp
	C:\ProgramData\Perl64\bin\perl.exe util\mkrc.pl libcrypto-1_1-x64.dll > libcrypto-1_1-x64.rc
	rc /folibcrypto-1_1-x64.res libcrypto-1_1-x64.rc
Microsoft (R) Windows (R) Resource Compiler Version 10.0.10011.16384

Copyright (C) Microsoft Corporation.  All rights reserved.

	link /nologo /debug /dll  /implib:libcrypto.lib /out:libcrypto-1_1-x64.dll /def:libcrypto-1_1-x64.def @C:\Users\user\AppData\Local\Temp\nmFA81.tmp || (DEL /Q libcrypto.* libcrypto-1_1-x64.* && EXIT 1)
   Creating library libcrypto.lib and object libcrypto.exp
	DEL /F apps\libcrypto-1_1-x64.dll
Could Not Find c:\libOPENSSL-1.1.0-dev\src\apps\libcrypto-1_1-x64.dll
	DEL /F test\libcrypto-1_1-x64.dll
Could Not Find c:\libOPENSSL-1.1.0-dev\src\test\libcrypto-1_1-x64.dll
	COPY libcrypto-1_1-x64.dll apps
        1 file(s) copied.
	COPY libcrypto-1_1-x64.dll test
        1 file(s) copied.
	cl -DOPENSSL_USE_APPLINK -DDSO_WIN32 -DNDEBUG -DOPENSSL_THREADS -DOPENSSL_NO_STATIC_ENGINE -DOPENSSL_PIC -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPOLY1305_ASM "-DENGINESDIR=\"C:\\Program Files\\OpenSSL\\lib\\engines\"" "-DOPENSSLDIR=\"C:\\Program Files\\Common Files\\SSL\"" -W3 -wd4090 -Gs0 -GF -Gy -nologo -DOPENSSL_SYS_WIN32 -DWIN32_LEAN_AND_MEAN -DL_ENDIAN -D_CRT_SECURE_NO_DEPRECATE -DUNICODE -D_UNICODE /MD /Ox /O2 /Ob2 /Zi /Fdossl_static /I include /I . -c /Fossl\bio_ssl.obj ssl\bio_ssl.c

[snip]

tls_srp.c
	C:\ProgramData\Perl64\bin\perl.exe util\mkdef.pl "ssl" 32 > libssl-1_1-x64.def
	C:\ProgramData\Perl64\bin\perl.exe -i.tmp -pe "s|^LIBRARY\s+ssl32|LIBRARY libssl-1_1-x64|;" libssl-1_1-x64.def
	DEL libssl-1_1-x64.def.tmp
	C:\ProgramData\Perl64\bin\perl.exe util\mkrc.pl libssl-1_1-x64.dll > libssl-1_1-x64.rc
	rc /folibssl-1_1-x64.res libssl-1_1-x64.rc
Microsoft (R) Windows (R) Resource Compiler Version 10.0.10011.16384

Copyright (C) Microsoft Corporation.  All rights reserved.

	link /nologo /debug /dll  /implib:libssl.lib /out:libssl-1_1-x64.dll /def:libssl-1_1-x64.def @C:\Users\user\AppData\Local\Temp\nm4BDE.tmp || (DEL /Q libssl.* libssl-1_1-x64.* && EXIT 1)
   Creating library libssl.lib and object libssl.exp
	DEL /F apps\libssl-1_1-x64.dll
Could Not Find c:\libOPENSSL-1.1.0-dev\src\apps\libssl-1_1-x64.dll
	DEL /F test\libssl-1_1-x64.dll
Could Not Find c:\libOPENSSL-1.1.0-dev\src\test\libssl-1_1-x64.dll
	COPY libssl-1_1-x64.dll apps
        1 file(s) copied.
	COPY libssl-1_1-x64.dll test
        1 file(s) copied.
	cl -DOPENSSL_USE_APPLINK -DDSO_WIN32 -DNDEBUG -DOPENSSL_THREADS -DOPENSSL_NO_STATIC_ENGINE -DOPENSSL_PIC -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPOLY1305_ASM "-DENGINESDIR=\"C:\\Program Files\\OpenSSL\\lib\\engines\"" "-DOPENSSLDIR=\"C:\\Program Files\\Common Files\\SSL\"" -W3 -wd4090 -Gs0 -GF -Gy -nologo -DOPENSSL_SYS_WIN32 -DWIN32_LEAN_AND_MEAN -DL_ENDIAN -D_CRT_SECURE_NO_DEPRECATE -DUNICODE -D_UNICODE /MD /Ox /O2 /Ob2 /Zi /Fddso /I include -c /Foengines\e_capi.obj engines\e_capi.c

So I've one PR for the little stuff, #1075.
[x] "/Ox /O2 /Ob2" => "/O2"
[x] Install apps\openssl.cnf
[x] Silence the noisy DELs
[ ] Check why Configure complains about nmake.exe and dmake.exe
[ ] Install HTML docs

Tested PR: https://github.com/openssl/openssl/pull/1075 for shared MSVC build.

all tests passed successfully,
openssl.cnf successfully installed to -openssldir=... specified directory,
no Could Not Find... noise in stderr.

Okie, so only two more items, are we agreed?

Check why Configure complains about nmake.exe and dmake.exe
Install HTML docs

There one more to add .
nmake clean is deleting a versionned file:
File removed - F:\crypto\x86cpuid.asm
File removed - F:\crypto\aes\aes-586.asm
File removed - F:\crypto\aes\aesni-x86.asm
File removed - F:\crypto\aes\vpaes-x86.asm
File removed - F:\crypto\bn\bn-586.asm
File removed - F:\crypto\bn\co-586.asm
File removed - F:\crypto\bn\x86-gf2m.asm
File removed - F:\crypto\bn\x86-mont.asm
File removed - F:\crypto\bn\asm\bn-c64xplus.asm
File removed - F:\crypto\camellia\cmll-x86.asm
File removed - F:\crypto\chacha\chacha-x86.asm
File removed - F:\crypto\des\crypt586.asm
File removed - F:\crypto\des\des-586.asm
File removed - F:\crypto\ec\ecp_nistz256-x86.asm
File removed - F:\crypto\md5\md5-586.asm
File removed - F:\crypto\modes\ghash-x86.asm
File removed - F:\crypto\poly1305\poly1305-x86.asm
File removed - F:\crypto\rc4\rc4-586.asm
File removed - F:\crypto\ripemd\rmd-586.asm
File removed - F:\crypto\sha\sha1-586.asm
File removed - F:\crypto\sha\sha256-586.asm
File removed - F:\crypto\sha\sha512-586.asm
File removed - F:\crypto\whrlpool\wp-mmx.asm
File removed - F:\engines\e_padlock-x86.asm
=> F:\crypto\bn\asm\bn-c64xplus.asm is versionned.

so only two more items, are we agreed?

Yes. And is it possible to make PERL utilize all CPU cores during documentation build?

@FdaSilvaYY writes:

There one more to add .
nmake clean is deleting a versionned file:
File removed - F:\crypto\x86cpuid.asm
...
=> F:\crypto\bn\asm\bn-c64xplus.asm is versionned.

Thanks.  I'll fix that in the same manner as for Unix.

@sav-ix writes:

And is it possible to make PERL utilize all CPU cores during documentation build?

I'm not sure how I'd do that, quite frankly, at least with the current structure...  I do have an idea, but that's a fairly gruesome change, not sure that's fit for 1.1.0 at this point.  I'll ponder it over the day.

quite frankly, at least with the current structure...

Parallel execution greatly increases speed of building OpenSLL binaries. Would be nice, if same became implemented for building documentation.

quite frankly, at least with the current structure...

Parallel execution greatly increases speed of building OpenSLL binaries. Would be nice, if same became implemented for building documentation.

I'm well aware of the concept...  making such a change would be quite invasive, though, and considering we're tight on time...
... but like I said, I'll ponder it over the say.

/Ox /O2 /Ob2 could be reduced to /O2

No, it couldn't. Original reason for this look was that some older compilers defined /Ox as something without /O2 and /O2 amended that. And /Ob2 has nothing to do with /O2 in sense that it's not combination of b and 2. The option is /Ob and controls inline expansion. If anything /O2 can be omitted (because we don't support that old compiler), but /Ox /O2 /Ob2 is not reducible to /O2. One can argue that /Ob2 can be omitted too... Once again, one can argue in favour of /Ox alone, one can argue in favour of /Ox /Ob2, but one can't argue in favour of replacing everything with /O2.

What I'm saying is that one shouldn't look at current version alone, but even at past version [which is the reason for why it looks the way it does].

/Ox /O2 /Ob2 could be reduced to /O2

No, it couldn't. Original reason for this look was that some older compilers defined /Ox as something without /O2 and /O2 amended that. And /Ob2 has nothing to do with /O2 in sense that it's not combination of b and 2. The option is /Ob and controls inline expansion. If anything /O2 can be omitted (because we don't support that old compiler), but /Ox /O2 /Ob2 is not reducible to /O2. One can argue that /Ob2 can be omitted too... Once again, one can argue in favour of /Ox alone, one can argue in favour of /Ox /Ob2, but one can't argue in favour of replacing everything with /O2.

Hmmm...  The list of options indicates that /Ox is equivalent to /Ob2gity /Gs (although it's own page doesn't, which is weird)), and the /O2 page indicates that it's equivalent to /Og /Oi /Ot /Oy /Ob2 /Gs /GF /Gy, which does contain all the options from /Ob2gity /Gs, so color me confused.
How old compilers do we support?

Still, btw, according to the /Ox page, it's still equivalent to /Ob2 /Og /Oi /Ot /Oy, which is still all part of /O2.
So I dunno...

I'm currently looking at the VS2005 manual, where it basically says the same, and also explicitely that:

In general, /O2 should be preferred over /Ox and /O1 over /Oxs.

as well as:

/Ox is mutually exclusive from: ... /O2

Ref: https://msdn.microsoft.com/en-us/library/59a3b321(v=vs.80).aspx

I might have overreacted, but point that it's not really appropriate to look at single compiler version still stands. Which old versions exactly I don't know (whichever that works), but in either case options should work for multiple versions. If I was to choose between /O2 and /Ox I'd go for /Ox, because it used to appear more sensible.

Wellllll, if the oldest I could find a manual for (for VS2005, on msdn) matches the newest (also on msdn) in this regard, I'm going to assume that all the versions between are essentially saying the same.
If we support compilers older than VS2005, then of course, that might be different, and I'd love to check the manuals for them if reachable.
In any case, if we've been using /Ox /O2 /Ob2 (that was the exact sequence before I changed it) and /Ox and /O2 are mutually exclusive, then I assume the last one wins, so we've essentially been using /O2 /Ob2 for a while already (and since /Ob2 is part of both /Ox and /O2, it's a no-op)...  Note: for VS2005 and newer

Basically, it seems the currently reachable manuals and you do not say the same thing.

the currently reachable manuals and you do not say the same thing.

Totally possible. Actually at it turns out the combination of flags originate from pre-OpenSSL era. So that what I likely to recall is looking and trying to make sense of it and at least noting that /Ob2 is amendment for /Ox, as it was originally b1. As for reachability of documentation. One has to recognize that we are talking about time when it was available to only MSDN subscribers (which I was not) on CD-ROM (which are all thrown away by now). But all right, I'm convinced that one can argue that /O2 can serve as replacement for all of them, sorry about noise. One can also say that even though /Ox might be preferable we should stick with whatever we used for that long. Once again, sorry about noise.

Nice feature, not going to happen in this release we still have too many real bugs to fix.

By "we should stick with whatever we used for that long" I meant /O2, as I for one don't see problem with omitting some options  even in upcoming release...

Once again, sorry about noise.

No need for appologies, it's good that we hammer these things out, so we get on the same page.

Nice feature, not going to happen in this release we still have too many real bugs to fix.

I have no idea what you're talking about...

Parallel doc builds

Ok, @FdaSilvaYY's point fixed in #1076, along with completion of cleanup.

This item seems to be an innocuous bug in ActiveState Perl 5.22:

Check why Configure complains about nmake.exe and dmake.exe

This is caused by our use of IPC::Cmd::can_run, and somewhere deep down, ends up having Config interacting with ExtUtils::MakeMaker in some off manner, print that string and then just return to whatever level up.
Like I said, innocuous.  Complain to ActiveState.

This last point was addressed yesterday:

Install HTML docs

That completes resolving the issues presented here.  Time to close.

Compared recent OpenSSL builds using Mings-64 and MSVC and here feedback on PRs: https://github.com/openssl/openssl/commit/e990ec5234d9daad66359833c40e4536d7fce499 , https://github.com/openssl/openssl/commit/2bc57c88d852d83ee31f68f5b84b36d54b4c3dc4

Mingw-w64 installation contain files:

bin/c_rehash
ssl/misc/c_hash
ssl/misc/c_info
ssl/misc/c_issuer
ssl/misc/c_name
ssl/misc/CA.pl
ssl/misc/tsget

which are missing in MSVC installation,

Html docs in Mingw-w64 installation contain files:

share\doc\openssl\html\man1\c_rehash.html
share\doc\openssl\html\man3\CRYPTO_malloc.html
share\doc\openssl\html\man3\OPENSSL_cleanse.html

which are missing in MSVC html docs,

Html docs in MSVC installation contain files:

html\man1\contributedundertheOpenSSLlicense.c_rehash.html
html\man1\OriginaltextbyJamesWestby.html
html\man3\OCSP_RESPONSE_free.html
html\man3\OCSP_response_create.html
html\man3\OCSP_request_sign.html
html\man3\OCSP_request_onereq_get0.html
html\man3\OCSP_REQUEST_free.html
html\man3\OCSP_request_onereq_count.html
html\man3\OCSP_request_add1_cert.html
html\man3\OCSP_request_add0_id.html
html\man3\OCSP_id_issuer_cmp.html
html\man3\OCSP_id_get0_info.html
html\man3\OCSP_id_cmp.html
html\man3\OCSP_copy_nonce.html
html\man3\OCSP_check_validity.html
html\man3\OCSP_check_nonce.html
html\man3\OCSP_cert_id_new.html
html\man3\OCSP_CERTID_free.html
html\man3\OPENSSL_cleanseCRYPTO_malloc.html
html\man3\OCSP_resp_get0.html
html\man3\OCSP_response_get1_basic.html
html\man3\OCSP_single_get0_status.html
html\man3\OCSP_resp_find.html
html\man3\OCSP_resp_count.html
html\man3\OCSP_resp_find_status.html
html\man3\OCSP_basic_add1_nonce.html

which are missing in Mingw-w64 html docs,

All files in html docs directory of Mingw-w64 installation contain empty tags:

<title></title>

while all files in html docs directory of MSVC installation contain appropriate text inside that tags,

Some files in html docs directory of Mingw-w64 installation contain simple text, eg:

 #include <openssl/evp.h>

 EVP_CIPHER_CTX *EVP_CIPHER_CTX_new(void);

while the same text in html docs directory of MSVC installation would be:

 #include <openssl/evp.h>

 EVP_CIPHER_CTX *EVP_CIPHER_CTX_new(void);

MSVC builds doesn't utilize all CPU cores during documentation build, unlike during binaries build using MSVC, or documentation build using Mingw-w64,

Noise in stderr still exists:

 	link /nologo /debug /dll  /implib:libcrypto.lib /out:libcrypto-1_1-x64.dll /def:libcrypto-1_1-x64.def @C:\Users\user\AppData\Local\Temp\nm8719.tmp || (DEL /Q libcrypto.* libcrypto-1_1-x64.* && EXIT 1)
   Creating library libcrypto.lib and object libcrypto.exp
	DEL /Q /F apps\libcrypto-1_1-x64.dll
Could Not Find c:\libOPENSSL-1.1.0-dev\src\apps\libcrypto-1_1-x64.dll
	DEL /Q /F test\libcrypto-1_1-x64.dll
Could Not Find c:\libOPENSSL-1.1.0-dev\src\test\libcrypto-1_1-x64.dll
	COPY libcrypto-1_1-x64.dll apps
        1 file(s) copied.

[snip]

   Creating library libssl.lib and object libssl.exp
	DEL /Q /F apps\libssl-1_1-x64.dll
Could Not Find c:\libOPENSSL-1.1.0-dev\src\apps\libssl-1_1-x64.dll
	DEL /Q /F test\libssl-1_1-x64.dll
Could Not Find c:\libOPENSSL-1.1.0-dev\src\test\libssl-1_1-x64.dll
	COPY libssl-1_1-x64.dll apps
        1 file(s) copied.

Dunno whether I was wrong, when reported above, that it was fixed, or it re-appeared because of some reason. Since /Q switch is not enough to silence errors, could be different solutions for this flaw:

send DEL errors to nul,
replace DEL commands with condition if exist apps\libssl-1_1-x64.dll (DEL ...), etc.

Some of issues above could describe expected behavior. In such case no need to reply on them.

I would appreciate it if a new issue was created instead of continuing on one that's closed. Please?


Calculate UV index - is my approach reasonable?

Hi,
I am using pvlib to calculate diffuse and direct incoming solar radiation at the surface and this works well (I account for clouds, albedo etc.).
    ....
    transmittance = (1.0 - cloud_covers) * 0.75
    # irrads is a DataFrame containing ghi, dni, dhi

    irrads = pvlib.irradiance.liujordan(solpos['apparent_zenith'].to_numpy(), transmittance, am_abs_array)

    total_irrad = pvlib.irradiance.get_total_irradiance(surface_tilt,
                                                        surface_azimuth,
                                                        apparent_zenith,
                                                        azimuth,
                                                        irrads['dni'],
                                                        irrads['ghi'],
                                                        irrads['dhi'],
                                                        albedo=albedo)
    results[:, 0] = total_irrad['poa_direct']
    results[:, 1] = total_irrad['poa_diffuse']

Now, I wanted to calculate the UV index based on the previous irradiance calculations by 1) Calculating the spectral distribution of the incoming solar radiation by multiplying by the solar energy spectrum 2) Consider only the contribution from the irradiance in the wavelengths 280-400 nm. When I do this I see that the values are too high and I wonder if my approach is flawed or whether I am doing something wrong here. Is there anyone else who has calculated UVI using pvlib?
I appreciate your help. Cheers, Trond
ps. I use the equation found here for UVI

Hi Trond,
Interesting question. We'd prefer usage questions to be asked in the google group or on stackoverflow. We use github discussions for software development.  Could you repost in one of those places?
I think we'll need to see more details regarding how the broadband irradiance from the pvlib functions is translated to spectral irradiance.
Cheers,
Cliff

@cwhanse Thank Cliff. I posted here:https://stackoverflow.com/questions/65111670/use-total-irradiance-to-calculate-uv-index


Include Hints detection/display in test automation

This is lower priority but would be nice if the output to test cases also tracked when ZipChord notices (and displays) hints.

Moved into #130


Upgrade to WildFly-17.0.0

FYI - Patch release 17.0.1.Final was released.
Can we get this onto master? So far it seems confined to the Camel 3 branch.


load python saved model in java

Hello,
I have created, trained and saved a tensorflow model using python
classifier = learn.DNNClassifier(hidden_units=[10, 20, 5], n_classes=5                                  ,feature_columns=feature_columns                                  ,model_dir= model_dir                                  )
The model files are in :
D:\java\workspace\APIJavaSampleCode\tfModels\dnn\ModelSave
files :
checkpoint
events.out.tfevents.1492792287.BIRINHOS-PC
graph.pbtxt
model.ckpt-1.data-00000-of-00001
model.ckpt-1.index
model.ckpt-1.meta
model.ckpt-100.data-00000-of-00001
model.ckpt-100.index
model.ckpt-100.meta
in Java I have the line code :
SavedModelBundle.load("D:/java/workspace/APIJavaSampleCode/tfModels/dnn/ModelSave");
The result error is :
Exception in thread "main" org.tensorflow.TensorFlowException: SavedModel not found in export directory: D:/java/workspace/APIJavaSampleCode/tfModels/dnn/ModelSave 	at org.tensorflow.SavedModelBundle.load(Native Method) 	at org.tensorflow.SavedModelBundle.load(SavedModelBundle.java:38) 	at tensorflow.HelloTF.main(HelloTF.java:32)
Can anyone help me loading a tf model in Java ?
Thanks

in java the tf version is 1.1.0-rc1
in python the tf version is (print(tf.version) ) :
1.0.1

I have download the git example model file "saved_model.pb" and it work (load in my java program).
maybe it is because is a binary format ?
I have to see how to set DNNClassifier() to save binary format.. and test..

well this like mission impossible ... it seams like it is impossible to have a model loaded in java after trained in python using tensorflow
I will turn to deeplearning4J ...

@rjpg see https://github.com/loretoparisi/tensorflow-java, if it can help!

hello,
thanks for the help. I found a solution and you can get the example code there :
http://stackoverflow.com/questions/43598953/loading-sklearn-model-in-java-model-created-with-dnnclassifier-in-python

hello! I  meet the same problem!  I  use tf 1.4 and train a model with python 3.6,now I want to load the model in java ,but it can not work! throws the exception SavedModel not found in export directory: d:/pythonworkspace/alerm/model/predicted........
can you share you point about how to solution the problem? or  the model train in python can not load in java ?

same with @moroseking .  I use tf 1.4 and train a model , when I load in java throws the exception SavedModel not found.

Same problem with TF 1.5 for Windows when trying to load this model. Code:
try (SavedModelBundle modelBundle = SavedModelBundle.load("D:/model/ende_gnmt_model_4_layer")) {
...
}

Exception:
org.tensorflow.TensorFlowException: SavedModel not found in export directory: D:/model/ende_gnmt_model_4_layer
	at org.tensorflow.SavedModelBundle.load(Native Method)
	at org.tensorflow.SavedModelBundle.load(SavedModelBundle.java:39)

Directory content:
translate.ckpt.data-00000-of-00001
translate.ckpt.index
translate.ckpt.meta

@dmak What if you use: "D:/model/ende_gnmt_model_4_layer.ckpt?

I had the same issue as @dmak. When looking at the SavedModelBundle.load Javadocs I noticed that "The model that is being loaded should be created using the Saved Model API". I was at first using the Saver API. After switching to the Saved Model API, it worked.

@timmolter It would be fantastic if you could add links (e.g. to Python API) to your above post to tell the difference between Saver API and aver Model API. Thanks!
The argument for SavedModelBundle.load() should be a directory containing a .pb or .pbtxt file. Just for case I've tried SavedModelBundle.load("D:/model/ende_gnmt_model_4_layer/translate.ckpt") and it didn't work either. I had to convert .ckpt to .pb using the "model freezing" approach described in How to convert .meta, .data and .index model files into one graph.pb file. After that the graph could be loaded like that:
import org.apache.commons.io.FileUtils;
import org.tensorflow.Graph;

try (Graph graph = new Graph()) {
    graph.importGraphDef(FileUtils.readFileToByteArray(new File("D:/translate.pb")));
}

I think what he means by Saver Api is:
saver = tf.train.Saver()
...
saver.save(sess, CHECKPOINT_PATH)

which saves out .chkpt, .pbtxt and variables files.
Using the SaverModel Api:
builders = saved_model_builder.SavedModelBuilder(export_path)
builders.add_meta_graph_and_variables(sess, ["myTag"], signature_def_map= {
        "predict": tf.saved_model.signature_def_utils.predict_signature_def(
            inputs= {"images": x_placeholder},
            outputs= {"scores": output})
        })
builders.save()

Which outputs:
.
├── saved_model.pb
└── variables
    ├── variables.data-00000-of-00001
    └── variables.index

which then can be loaded with:
tf.saved_model.loader.load(sess, ["myTag"], OUTPUT_DIR)
graph = tf.get_default_graph()


METRON-2169: Upgrade to Kafka 2.0.0 and Storm 1.2.1

Contributor Comments
This PR upgrades the Kafka client dependency from <IP_ADDRESS> to 2.0.0 and the Storm dependency from 1.0.3 to 1.2.1. Overall this upgrade was pretty straightforward. The only complex changes involved upgrading the KafkaComponent class to use new Kafka admin and testing classes.
@mmiklavc, all the comments you made on #1453 should be addressed here with the exception of excluding Storm.  I had to upgrade both Storm and Kafka versions or I get this.
@nickwallen, all the comments you made on #1488 should be addressed as well.
To isolate Kafka/Storm from the rest of the components, I upgraded these versions independently in all Maven profiles.  This will allow us to test newer client versions in full dev without having to upgrade all clients.  I created a Jira to reconcile these once we complete the upgrade work.
Testing
Right now I have only verified that it builds and tests pass.  I plan on running these tests in full dev:
Verify full dev comes up without error and data is landing in ES
Verify all REST Kafka functions are working as expected
Verify all Stellar Kafka functions are working as expected
Verify Kerberos
Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our Development Guidelines for the complete guide to follow for contributions.
Please refer also to our Build Verification Guidelines for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
For all changes:

[x] Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
[x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

For code changes:

[x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?

[x] Have you included steps or a guide to how the change may be verified and tested manually?

[x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh

[x] Have you written or updated unit tests and or integration tests to verify your changes?

[x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

[ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

For documentation related changes:

[x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via site-book/target/site/index.html:
cd site-book
mvn site

[x] Have you ensured that any documentation diagrams have been updated, along with their source files, using draw.io? See Metron Development Guidelines for instructions.

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that travis-ci is set up for your personal repository such that your branches are built there before submitting a pull request.

The latest commit fixes this failed test in ProfilerIntegrationTest:
2019-08-23 19:02:50 ERROR executor:0 -
java.lang.RuntimeException: java.lang.IllegalStateException: This consumer has already been closed.

My guess is that this upgrade changed the behavior of the Kafka Spout to close the consumer when a topology is killed.  ProfilerIntegrationTest is unique in that it starts and stops multiple topologies whereas most integration tests (EnrichmentIntegrationTest) only use one.  The fix I chose is to remove this step (consumer close) from the KafkaComponent.stop method.  The KafkaComponent.shutdownConsumer method is still available if closing the internal client is necessary.


Spelling mistake in the word "customsed" :)


Add Result property to WorkflowSignalChildReply

When signaling the ChildWorkflowFuture, the return value is a workflow.Future.  I assume that I am going to wait on the future to return either a result or an error.  I need to send the result back to the Neon.Cadence client in the reply.  As it stands, there is not Result property in the WorkflowSignalChildReply message type.  I am going to add one.  Tell me what you think about this.

No, signals don’t return replies, the semantics are fire-and-forget.


export endpoint token between demo and install to reuse same token

re: https://github.com/preludeorg/libraries/issues/221
will update powershell in a separate PR

I don't understand the change set here; most of it is renaming the env var, where the only applicable line appears to be 65 (export VARIABLE) in the demo.sh.

That's all correct. The renaming is just for consistency/ I think it's clearer. The exporting is the functional change - it goes with the changes in install.sh.   If user chooses to install a probe after running the demo script, the demo script exports ENDPOINT_TOKEN. meanwhile install.sh checks if ENDPOINT_TOKEN is set as an env variable, and if so, will reuse that token instead of registering a new endpoint

Why ENDPOINT TOKEN instead of PRELUDE though? Referring to the same thing by two names across our code base can get confusing. (I initially chose PRELUDE, so all our env vars, currently 3, could be spotted easily).

I just think ENDPOINT_TOKEN is more explicit - I've had multiple people just at Prelude ping me that "probes aren't working" b/c they are using an admin token instead of an endpoint token. what about PRELUDE_ENDPOINT_TOKEN across the board? It matches PRELUDE_ACCOUNT_ID and PRELUDE_ACCOUNT_SECRET better
(it also nags at me that we use "secret" when referring to accounts and "token" for endpoints, not sure if you feel the same ha)

We can definitely look at renaming these vars but let's separate that from this PR.
Renaming the vars is a larger change: docs, code, probes, handling existing deployments, etc, so while I'm all for it, we should isolate why someone would get confused first (poor docs, poor error messaging, etc).

Moved demo.sh back to PRELUDE_TOKEN (though that wasn't a public variable/argument so it shouldn't have been a problem...)
Also moved ENDPOINT_TOKEN in install.sh to an arg to stay consistent with other args there

I don't want to invent new env var names in one spot, so let's use our existing names:

PRELUDE_TOKEN
PRELUDE_API
PRELUDE_CA

For account ID/token pairs, we currently do not leverage these in env vars, so we can invent something for our install/demo scripts, assuming they are all consistent. PRELUDE_ACCOUNT_ID / PRELUDE_ACCOUNT_SECRET are fine by me, although the latter is likely odd if the user is required to send in a 'token' as a header.

I'm not adding any new env vars in this PR. Are you requesting that the install.sh argument descriptor names match the env vars? Ie. you want the help menu to look like:
mahinakaholokula@Mahinas-MacBook-Pro libraries % ./shell/probe/install.sh -h

Usage: install.sh [-h] -n PROBE_NAME

  -h                          Shows Usage
  -n PROBE_NAME               Probe Name; Default: nocturnal
  -t PRELUDE_TOKEN            Endpoint Token; Use PRELUDE_ACCOUNT_ID and PRELUDE_ACCOUNT_SECRET to register a new endpoint
  -a PRELUDE_ACCOUNT_ID       Prelude Account Id; asdf
  -s PRELUDE_ACCOUNT_SECRET   Prelude Account Secret;
  -e ENDPOINT_ID              Endpoint Id; Default: Mahinas-MacBook-Pro.local

@privateducky Originally was pushing the endpoint_token through so that a re-run of the demo script would work (endpoint token would still be active). With the trouble were having and looking at https://github.com/preludeorg/range/issues/13 - I'm just going to switch to re-registering the endpoint in install.sh. The demo script can pass through the endpoint id and install.sh will re-register with same endpoint name/ get a new token back

@makk94 exactly; match the env var names with the names in the params.


Added mkdocs documentation and pipeline

:tada: This PR is included in version 1.0.8 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


CASMCMS-8568: cmsdev: Add good path BOS CLI multi-tenancy tests

Summary and Scope
Update to a version of CMS tests that include BOS CLI good path multi-tenancy tests.
Testing
Tested all new/updated code paths on mug.
Risks and Mitigations
Low risk -- higher risk to not get this new test coverage, IMO.
Pull Request Checklist

[X] Version number(s) incremented, if applicable
[X] Copyrights updated
[X] License file intact
[X] Target branch correct
[X] CHANGELOG.md updated
[X] Testing is appropriate and complete, if applicable

/backport release/1.6


Occasional errors on startup

Similar to #26 in that I don't yet see how to deterministically reproduce, I occasionally get:
Error detected while processing CursorHold Autocommands for "*.lean":
E5108: Error executing lua /Users/julian/Development/lean.nvim/lua/lean/infoview.lua:37: attempt to index local 'result' (a number value)

when opening a lean file. Here it's rare, and only happens occasionally.
I don't know how result is getting to be a number there. Perhaps just ignoring it when it is will silence the error, but it'd be nice to know why it's happening.

I also get this error, it usually indicates that the server failed to start (e.g. because I'm in a Lean 3 project).

Fixed in 513ed1486c4c4228c6571dbaf3a89132915237a4 I believe, right? (Thanks!)

This was super annoying so I pushed a commit to ignore the errors.    513ed1486c4c4228c6571dbaf3a89132915237a4
The callback is also called if an error occurs (which is probably what we both experienced), in this case the first argument is the error value.  We might also want to check that (instead).

I meant to post this comment earlier, but there were github issues.

Hah all good what you did looks like what I hacked in locally so let's see how this goes!


excel sheet date field lost first 2 digit

In the excel sheet, let's say there is a column with values of date type. In the values of date type, let us say one value is '4/15/2015'. When  SheetJS gets the json data(use the api xlsxUtil.sheet_to_json(rosterFile.Sheets[rosterSheet])), the value '4/15/2015' is not there, but '4/15/15' is in the json. So it seems '2015' lost the first '20'. Is this a feature or bug?
Thanks
Jerry

@jerryhe88 I suspect this is the same as https://github.com/SheetJS/js-xlsx/issues/200 and https://github.com/SheetJS/js-xlsx/issues/181.  To be sure, if you manually change the format for code 14 (see https://github.com/SheetJS/js-xlsx/issues/181#issuecomment-76733831) to 'm/d/yyyy', do you see 2 or 4 digits?

thanks for quick reply. Not sure what needs to do after reading  #200 and #181.
The below is the file and code:
filePath.xlsx has data like '4/15/2015';
var fileContent = xlsx.readFile(filePath.xlsx);
var rowDatas = xlsxUtil.sheet_to_json(fileContent.Sheets['Template']);
after the above,
rowDatas has  '4/15/15'. Can you please provide sample codes or configs so that rowDatas can have '4/15/2015'?
thanks again
jerry

In 0.9.13 we're adding a dateNF option for the parsers to override the interpretation of the default dates:
var fileContent = XLSX.readFile(filePath.xlsx, {dateNF:"mm/dd/yyyy"});


This website hangs indefinitely: https://nugnes1920.com/

Platform
Which platform does the bug apply to?

Website

Describe the bug
Hanging.
To Reproduce
node src/drivers/npm/cli.js "https://nugnes1920.com/"
Expected behavior
A clear and concise description of what you expected to happen.
Additional context
Add any other context about the problem here.

Hey, thanks! I disabled the PayPal link because it can't be used anonymously. You can send me a tip here:
https://www.buymeacoffee.com/aliasio


CI: Run Cypress E2E tests in CI build

Overview
We're got a bunch of E2E tests now. But we never run them! We should run Cypress tests in Travis.
NOTE: tests may be failing b/c we don't run them often. This ticket will be blocked by any test failures that come up.

Cypress documentation for running in CI, includes an example for Travis

We should definitely run the E2E tests in a parallel job in the test stage. IIRC, if we're concerned about flakey tests, we can mark the E2E job as allowed to fail until we're confident


Handlers and Listeners

One other demanded feature would be to have handlers outside of the Component, and also listeners before and after action.
E.g.
Handlers:
you could provide an ref to the HOC, and just call from outside this.galleryRef.setActiveSlide(3) or this.galleryRef.setFullScreen()
Listeners:
beforeSwipe, afterSwipe, beforeScreen
I think that are the only things that I am missing from other galleries. Else I really like your gallery! :)

Hi @a-tonchev ! Thank you for raising this issue up! 😄
Thank you for your appreciation! I will add handlers, listeners and publish a new version before the end of next week. 👨🏻💻 Rest assured that there will be no breaking changes. I will notify you for the new version.

Hi, @a-tonchev ! I carefully considered your request of having handlers! I looked at some other components' interfaces and implementations. I read the React.js documentation at https://reactjs.org/docs/refs-and-the-dom.html#when-to-use-refs. The docs says: "Avoid using refs for anything that can be done declaratively. For example, instead of exposing open() and close() methods on a Dialog component, pass an isOpen prop to it."

So, it would be better if developers pass the index, isMaximized  and isAutoPlaying. So, I decide not to add handlers at this stage, unless there are other use cases. I thank you for bring this feature request up, my investigation led me to learn something new today!
Meanwhile, I think you are right that my component needs more listeners. 😄 I will add listeners as promised.

Hi @yifaneye,
In the same screenshot that you post is recommended to use refs for the usecases I mention:
ref.fullScreen(), ref.nextSlide(), ref.goToSlide(4), because this is triggering of imperative animation
ref.play() is also okay, because it is media playback
They tell us to avoid using ref for anything that can be done declaratively like close() open() or the other props that you provide like useThumbnails, arrowIcons etc...
There is no other way slightly to jump to slide 4 than using ref, and as you can see any other gallery is using handlers 😉

Hi @a-tonchev ,
Thank you for taking time to clarify your proposal. I really appreciate your effort. 😃 You mention about the good use cases. I think these use cases in the screenshots are for directly manipulating an element, not for controlling a component.
There is way to "jump to slide 4" other than using ref, that is by passing the index prop to my component.
I had a read on the difference between declarative vs imperative in React.js https://stackoverflow.com/questions/33655534/difference-between-declarative-and-imperative-in-react-js. I think my component provides an interface for developers to control it in a declarative style that is inline with React.js. 😉
Would you like to tell me your use case(s) please?

Hi @yifaneye,
Actually I am creator of a new online shop System SaaS, similar like Shopify. It is called JUST-SELL.online
Until now I used some galleries, but for now I stopped on yours :)
You can check a demo, where you can see your gallery in action here: https://demo2.shopsuy.com/
Well the use-case where I would like to jump on slide is when the user pick product variants (e. g. Colors) to be able the gallery to jump directly on the corresponding slide. For example the slide 4 is with the color red, and when the user is chosing from the dropdown of the product the color red the gallery will automatically open the slide 4 :)
Using refs for this reason is not something new and forbidden - as I sayd everyone is using it :)

You can make use of the useImperativeHandle API for this reason, this will attach to the refs any desired function from inside. Don't forget to use forwardRef if needed ;)
https://reactjs.org/docs/hooks-reference.html#useimperativehandle
https://www.google.com/amp/s/www.geeksforgeeks.org/react-js-useimperativehandle-additional-hook/amp/

Hi @a-tonchev ,
Thank you for sharing the demo! I really like your product! 😀 I think you can try set the current index with the index prop in your use case for now.
Thank you for suggesting the use of useImperativeHandle! I am going to have a try! I believe I am going to figure it out! 😉

Hi @a-tonchev !
I am pleased to report that I am getting there using useImperativeHandle! I am going to make some examples and refactor the code!

@yifaneye wow thanks man, you are the best. There is no hurry, i still have a lot of work on other issues ;)