v1.3.0 missing object expanders on web interface

[x] Review the documentation: https://docs.sentry.io/platforms/javascript/electron/
[x] Search for existing issues: https://github.com/getsentry/sentry-electron/issues
[x] Use the latest release: https://github.com/getsentry/sentry-electron/releases
[ ] Provide a link to the affected event from your Sentry account (I can DM a link if necessary but would rather not post one here)

Versions + Platform

[x] SKD version - @sentry/electron: 1.3.0,
[x] Electron version - electron: 7.1.12
[x] Platform - macOS

Description
We have a working version of our app running with v1.2.0 and our Extra data objects have + expanders in them that can be expanded to show more information.

When I upgrade to v1.3.0 these + expanders are replaced with [Object] and can not be expanded.

If I go back to v1.2.0, the expanders return

v1.2.1 introduced some changes to prepare for Electron 8 where IPC uses Chrome structured clone algorithm.
if you download the full JSON from Sentry, are the objects there?

The JSON has the same [Object]
"_persist": {
  "rehydrated":true,
  "version":27
},
"accounts": {
  "5711175e-e9c4-5a35-85c9-4fcd9edf504d": "[Object]",
  "42469d28-8b23-5ac4-974b-af9631a4f5b9":"[Object]",
},
"apps": {},

@HyperSprite can you confirm that your objects are also like this for v1.2.1 where this change was made?

On v1.2.1 I can see the expanders like we had in v1.2.0

Thanks for checking this. Then it looks like this difference was caused by all the Sentry dependencies getting bumped. Maybe @HazAT can shed some light on what could have changed?

We have just released v3.0.0 of the SDK which includes a lot of changes and improvements since this issue was opened.
If you can still reproduce this with v3, please open a new issue and I'll look into it.


Add Support for Just Enough Calculation

Just Enough Calculation is a mod that assists in calculating the materials needed for a recipe. It is very useful when working with large and / or complex recipes.
The mod also supports placeholders, which are meant as a way to signify more abstract things, such as a multiblock structure. Say you have a multiblock X, which requires blocks A, B, and C. You can create a placeholder, called X, and use it as the "output" portion of a recipe. In essence, this also assists with builds as well.
Currently though, almost all items from SlimefunEssentials are being registered as Placeholders.
Example with recipe for 5 reinforced ingots:

With their names being:

The naming scheme suggests its related to the SlimefunItemStack() method. For clarity, they are not all considered Damascus Steel
It seems that items without a registered recipe function normally:

The electric smeltery does not have a registered recipe from SlimefunEssentials, and thus is not considered a Placeholder by JEC.
I am unsure as to what leads JEC to assume SlimefunEssentials items as Placeholder items. Assuming the cause is identified and fixed, this would make SlimefunEssentials and JEC compatiable.

I can add it to my to do list, not guarantee when I'll look into it

@54M44R could you send the logs too?

This comment is to serve as a record that commit #037908e has fully implemented support for JEC.


Hi!
I have been using the playwright library for a while, because I may not know everything about it and its capabilities yet.
In the documentation for the Node JS library, I found the functionality I needed: LaunchServer (link to the documentation: https://playwright.dev/docs/api/class-browsertype#browser-type-launch-server ), but in this implementation on go, I did not see anything like that.
Is there any similar functionality and, if so, how to use it?
If it doesn't exist, what other workarounds are there to achieve a similar goal?
I'll explain what I need.:
I am developing a microservice that should "host" browsers, giving out addresses for connection. While other microservices will go to these browsers and work with them remotely via a ws connection.

Hello! I have also been using Playwright recently, and after reading your question, perhaps this snippet of code can be used to expose the port of the corresponding browser instance?

Hello! I have also been using Playwright recently, and after reading your question, perhaps this snippet of code can be used to expose the port of the corresponding browser instance?

Hello 👋!
I have recently been utilizing playwright-go, and upon reading your question, it occurred to me that the ExposeNetwork argument might be employed to connect to the remote browser instance?
BrowserTypeConnectOptions

Run driver with params:
https://github.com/playwright-community/playwright-go/blob/6b672dbd24505206feca7c2b660da8d616c6393f/tests/remote_server_test.go#L23
https://github.com/microsoft/playwright/blob/9fcf60464d983b70b3cafea3e230e4e6e4f92c15/packages/playwright-core/src/cli/program.ts#L263

Thanks for the suggested options. I'll try it soon and come back with the results


[dv] Fixup coverage collection to match OT dvsim flow

This helps improve architectural coverage as the bad immediates meant certain coverpoint bins for these instructions weren't getting recorded as hits when they should have been

@hcallahan-lowrisc could you take a quick look?


Integrate with vdirsyncer

Sync the diary entry to a simple iCalendar VEVENT, because VJOURNAL entries are rarely used. Document how to sync the diary to a Google calendar, for instance.

Enables the user to view the same journal entries on the mobile phone using a DAVx⁵ and a cloud (provider, on-prem) of choice

Todo:

[ ] Write the VEVENT from put_event() to a file instead of uploading it to the remote


Set "additionalProperties": false by default

Description
It's a feature request.
If one inserts data with scheme that differs from the collection creation call, by default, the tigris server allows to do so.
It's must have been impossible to do so, because it's impossible to read data after that in the gRPC client.
But it did.
Describe the solution you'd like
Set "additionalProperties": false by default.
Describe alternatives you've considered
However, tigris-cli allows to read data afterwards, maybe the solution is to make it possible to read data in gRPC client afterwards? But how, taking into account that there must be strict values (use json.RawMessage?).
Additional context

Add any other context or screenshots about the feature request here.

thank you for reporting, I agree best is to disable it by default to have strict schema validation.


SOME/IP specification cleanup

The following change is a reboot of the SOME/IP specifications in an attempt to drastically simplify to show how we map up-l1 messages to SOME/IP messages and uSubscription messages to SOME/IP-SD messages. the older specifications had a lot of uProtocol version 1.3.6 concepts (cloudevents) that were rather complicating the situation.
(based on PR-117, that can't be merged at the moment)
Fixes #93

@int0x27 I think we are OK to merge, next week we can add specific mappings of the uTransport API given the work that Peter has done on up-transport-vsomeip-rust.

OK, just rebased on main, squashed some commits


Update BQ labels on tables as they change

Describe the feature
Currently, labels apply when a table is created. However, for incremental models, the labels are not updated if they change in the future. This is also a problem as the labels become out of sync with the DBT project.
Describe alternatives you've considered
An alternative is to write a post-hook which applies the labels to the relation.
I'd prefer to contribute this feature back into the DBT project so other people can benefit from it.
Additional context
This feature is specific to BigQuery
Who will this benefit?
It will benefit users that access tables in BigQuery, the labels offer great metadata.
We also plan to have this visible in our metadata/catalog service.

@drewbanin I've got a version of this working where the labels config replaces what is already on the table.
One difference to the other materialization options is the case when all the labels have been removed. What should the behaviour be in this case? Looking at the BigQuery documentation, you aren't able to remove all the labels with SQL, you have to have at least one for it to work. You can have at least one label but removing them all can't be done, only with Python in this case.
Doing it within SQL, I see two options.

Update the labels with a blank i.e. no_labels = true. This would remove all the previous labels but would require a dummy label to be put back in it's place which would stay until a --full-refresh is triggered.

Not updating the labels at all, leaving them all intact. This is my least favourite of the two since it wouldn't be possible to ever remove all the labels (until of course a --full-refresh is triggered). Despite this being my least favourite, it wouldn't be a regression on the current capabilities.

If done in Python, I think it would be a case of getting fetching all the labels in the table and then subsequently setting them all to None like this. If this was the prefered option, I might require some assistance to get this working.

Hey there,
Has this issue been solved?

For anyone landing on this issue, we've implement a hacky solution for.
It consists of two macros
{% macro sync_labels() %}
    {%- set labels_dic = config.get('labels', {}) %}
    -- Get the labels from the config
    {%- set labels = [] %}
    {%- for key, value in labels_dic.items() %}
    -- Loop through the labels
        {%- set label_str = "('" ~ key.lower() ~ "','" ~ value.lower() ~ "')" %}
         -- CONCAT the key and value as a lowercase string with the correct format
        {%- do labels.append(label_str) %}
        -- Append the label to the list
    {%- endfor %}
    {%- set labels_string = labels | join(', ') %}
    -- Join the labels list into a string

    ALTER TABLE {{ this }}
    SET OPTIONS (
        labels = [{{ labels_string }}]
    );
    -- Apply the labels to the table
{% endmacro %}

{% macro sync_labels_for_incremental() %}
    {%- set model_materialization = config.get('materialized') %}
    {%- set labels_dic = config.get('labels', {}) %}
    {%- if model_materialization in('incremental') and is_incremental() %} -- For incremental models that run incrementally
        {%- if labels_dic %} -- If labels are provided, we apply them
            {{ sync_labels() }}
        {%- endif %}
    -- For full refresh tables and views, labels are applied at runtime automatically
    -- Ephemeral don't have labels
    -- We may need to add snapshots and seeds here when the framework supports them
    {%- endif %}
{% endmacro %}

Macro is then called as post_hook within the dbt_project.yml
models:
  my_dbt_project:
    +post-hook: ["{{ sync_labels_for_incremental() }}"]


[BUG] Update footer + header logo

Description of the bug
Replace with the new logo.
Please add screenshots (if applicable)

Additional information
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

@Panquesito7  you want this issue to be assigned to you?


Update docs: blocks per hour

The min blocks per hour is 12, this should be reflected here: https://github.com/MinecraftPortCentral/GriefPrevention/wiki/Advanced-Options

Done, look ok?


Error: Uncaught exception on main thread: Invalid field access : nodeType

Describe your problem here.
Error: Uncaught exception on main thread: Invalid field access : nodeType
Are you modding a build from source or with Lua?
Source
What is your build target?
Windows
Did you edit anything in this build? If so, mention or summarize your changes.
No response

boi u put nodetype instead of notetype XD

Where???

where do i put notetype

Where???

it said
Error: Uncaught exception on main thread: Invalid field access : nodeType
meaning it didnt expect the it to be nodetype u miss spelled it

i know i just mean where do i respell

i know i just mean where do i respell

it never said in the error the line and which .hx file

Where???

it said
Error: Uncaught exception on main thread: Invalid field access : nodeType
meaning it didnt expect the it to be nodetype u miss spelled it

Thanks Sherlock, but where did this bug happen????

dont  know it says it on kade ,ninja, cool, and psych

Where???

it said
Error: Uncaught exception on main thread: Invalid field access : nodeType
meaning it didnt expect the it to be nodetype u miss spelled it

Thanks Sherlock, but where did this bug happen????

boi i said it never said where it was

This issue is weird as hell and seems to be something that happens only to you, i don't think i can help with that.

the nodetype error keeps happening to me and its getting annoying


i try run it but it says no portscanner module found nmap

Checklist for submitting an issue to KickThemOut:

[ ] I have carefully read the README file and haven't managed to resolve my issue.
[ ] I have searched the issues of this repo and believe that this is not a duplicate.
[ ] I am running the latest version of KickThemOut.

OS name & version:
Python version:
Scapy version:
Nmap version:
Link of Gist:

Description:

Same problem, anyone have any ideas?

Okay so I found out the problem, run the following:
sudo apt-get update && sudo apt-get install nmap
git clone https://github.com/k4m4/kickthemout.git
cd kickthemout/
sudo -H pip3 install -r requirements.txt
sudo python3 kickthemout.py
So if you did the same thing I did this should fix it.

So now your probably going to run in to the following problem:
1.) Enter Gateway IP (e.g. <IP_ADDRESS>):
2.) Enter your gateway's MAC Address (MM:MM:MM:SS:SS:SS):
Many people have ran in to this problem, I have still not found the work around. I used the following process:
1.) windows button + r
2.) run cmd.exe
3.) type route print or arp -a (arp -a is easer to read)
Note: Go to your normal operating system to do this, i assume your running a VM and Windows.
Using arp -a: I have two Dynamic Types neither work. Internet Address is the IP, Physical Address is the Mac. After I load the data the results are 3 devices online, IP address range the 10.0.2.* . This is wrong, I'm going to run down the list later and check them all i will update if I have progress.

Have you had any luck with this issue??

Have you had any luck with this issue??


Fix: update delimiter of FindCommand and add name check in Name class

Update the format used in FindCommand from:
find keyword [, comma-seperated keywords]
to:
find keyword ["+"-seperated keywords]
example:
find rice + garlic + chicken

Add name check to remove names with invalid symbols (e.g. ",", "!", "@", etc)

@aidanaden don't forget to link this to the delimiter change PR


Integrate PodDefaults rock

Closes #151
This PR integrates the PodDefaults into admission-webhook for the CKF 1.9 bundle.
The new OCI image has been published in Dockerhub here.

Ok, a simple change of the rock doesn't seem to work, the charm stays on blocked with the following logs from the admission-webhook container:
juju debug-log -i admission-webhook --replay
unit-admission-webhook-0: 15:49:58 INFO juju.cmd running containerAgent [3.4.5 e5130b8c9e327c81818c849951575b9f3156307d gc go1.21.12]
unit-admission-webhook-0: 15:49:58 INFO juju.cmd.containeragent.unit start "unit"
unit-admission-webhook-0: 15:49:58 INFO juju.worker.upgradesteps upgrade steps for 3.4.5 have already been run.
unit-admission-webhook-0: 15:49:58 INFO juju.worker.probehttpserver starting http server on <IP_ADDRESS>:65301
unit-admission-webhook-0: 15:49:58 INFO juju.api cannot resolve "controller-service.controller-uk8sx.svc.cluster.local": lookup controller-service.controller-uk8sx.svc.cluster.local: operation was canceled
unit-admission-webhook-0: 15:49:58 INFO juju.api connection established to "wss://<IP_ADDRESS>:17070/model/6e8f958a-55c4-4cb3-81d6-e46a37fa48fa/api"
unit-admission-webhook-0: 15:49:58 INFO juju.worker.apicaller [6e8f95] "unit-admission-webhook-0" successfully connected to "<IP_ADDRESS>:17070"
unit-admission-webhook-0: 15:49:58 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-admission-webhook-0: 15:49:58 INFO juju.worker.logger logger worker started
unit-admission-webhook-0: 15:49:58 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
unit-admission-webhook-0: 15:49:58 INFO juju.worker.leadership admission-webhook/0 promoted to leadership of admission-webhook
unit-admission-webhook-0: 15:49:58 INFO juju.worker.caasupgrader abort check blocked until version event received
unit-admission-webhook-0: 15:49:58 INFO juju.worker.caasupgrader unblocking abort check
unit-admission-webhook-0: 15:49:58 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-admission-webhook-0
unit-admission-webhook-0: 15:49:58 INFO juju.worker.uniter unit "admission-webhook/0" started
unit-admission-webhook-0: 15:49:58 INFO juju.worker.uniter resuming charm install
unit-admission-webhook-0: 15:49:58 INFO juju.worker.uniter.charm downloading local:admission-webhook-0 from API server
unit-admission-webhook-0: 15:49:58 INFO juju.downloader downloading from local:admission-webhook-0
unit-admission-webhook-0: 15:49:58 INFO juju.downloader download complete ("local:admission-webhook-0")
unit-admission-webhook-0: 15:49:58 INFO juju.downloader download verified ("local:admission-webhook-0")
unit-admission-webhook-0: 15:50:06 INFO juju.worker.uniter hooks are retried true
unit-admission-webhook-0: 15:50:06 INFO juju.worker.uniter found queued "install" hook
unit-admission-webhook-0: 15:50:07 INFO unit.admission-webhook/0.juju-log Running legacy hooks/install.
unit-admission-webhook-0: 15:50:08 INFO unit.admission-webhook/0.juju-log Generating certificates if missing.
unit-admission-webhook-0: 15:50:08 INFO unit.admission-webhook/0.juju-log Generating certificates..
unit-admission-webhook-0: 15:50:08 WARNING unit.admission-webhook/0.install Generating RSA private key, 2048 bit long modulus (2 primes)
unit-admission-webhook-0: 15:50:08 WARNING unit.admission-webhook/0.install ....+++++
unit-admission-webhook-0: 15:50:08 WARNING unit.admission-webhook/0.install .........................................+++++
unit-admission-webhook-0: 15:50:08 WARNING unit.admission-webhook/0.install e is 65537 (0x010001)
unit-admission-webhook-0: 15:50:08 WARNING unit.admission-webhook/0.install Generating RSA private key, 2048 bit long modulus (2 primes)
unit-admission-webhook-0: 15:50:08 WARNING unit.admission-webhook/0.install ..............................................................................................+++++
unit-admission-webhook-0: 15:50:08 WARNING unit.admission-webhook/0.install ...............................................................................................................+++++
unit-admission-webhook-0: 15:50:08 WARNING unit.admission-webhook/0.install e is 65537 (0x010001)
unit-admission-webhook-0: 15:50:08 WARNING unit.admission-webhook/0.install Signature ok
unit-admission-webhook-0: 15:50:08 WARNING unit.admission-webhook/0.install subject=C = GB, ST = Canonical, L = Canonical, O = Canonical, OU = Canonical, CN = <IP_ADDRESS>
unit-admission-webhook-0: 15:50:08 WARNING unit.admission-webhook/0.install Getting CA Private Key
unit-admission-webhook-0: 15:50:08 INFO unit.admission-webhook/0.juju-log Certificates already exist.
unit-admission-webhook-0: 15:50:09 INFO unit.admission-webhook/0.juju-log HTTP Request: GET https://<IP_ADDRESS>/apis/apiextensions.k8s.io/v1/customresourcedefinitions "HTTP/1.1 200 OK"
unit-admission-webhook-0: 15:50:09 INFO unit.admission-webhook/0.juju-log Rendering manifests
unit-admission-webhook-0: 15:50:09 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://<IP_ADDRESS>/apis/rbac.authorization.k8s.io/v1/clusterroles/admission-webhook?force=true&fieldManager=lightkube "HTTP/1.1 201 Created"
unit-admission-webhook-0: 15:50:09 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://<IP_ADDRESS>/apis/rbac.authorization.k8s.io/v1/clusterroles/admission-webhook-kubeflow-poddefaults-admin?force=true&fieldManager=lightkube "HTTP/1.1 201 Created"
unit-admission-webhook-0: 15:50:09 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://<IP_ADDRESS>/apis/rbac.authorization.k8s.io/v1/clusterroles/admission-webhook-kubeflow-poddefaults-edit?force=true&fieldManager=lightkube "HTTP/1.1 201 Created"
unit-admission-webhook-0: 15:50:09 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://<IP_ADDRESS>/apis/rbac.authorization.k8s.io/v1/clusterroles/admission-webhook-kubeflow-poddefaults-view?force=true&fieldManager=lightkube "HTTP/1.1 201 Created"
unit-admission-webhook-0: 15:50:09 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://<IP_ADDRESS>/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/admission-webhook?force=true&fieldManager=lightkube "HTTP/1.1 201 Created"
unit-admission-webhook-0: 15:50:09 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://<IP_ADDRESS>/apis/admissionregistration.k8s.io/v1/mutatingwebhookconfigurations/admission-webhook?force=true&fieldManager=lightkube "HTTP/1.1 201 Created"
unit-admission-webhook-0: 15:50:09 INFO unit.admission-webhook/0.juju-log Reconcile completed successfully
unit-admission-webhook-0: 15:50:10 INFO unit.admission-webhook/0.juju-log HTTP Request: GET https://<IP_ADDRESS>/apis/apiextensions.k8s.io/v1/customresourcedefinitions "HTTP/1.1 200 OK"
unit-admission-webhook-0: 15:50:10 INFO unit.admission-webhook/0.juju-log Rendering manifests
unit-admission-webhook-0: 15:50:10 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://<IP_ADDRESS>/apis/apiextensions.k8s.io/v1/customresourcedefinitions/poddefaults.kubeflow.org?force=true&fieldManager=lightkube "HTTP/1.1 201 Created"
unit-admission-webhook-0: 15:50:10 INFO unit.admission-webhook/0.juju-log Reconcile completed successfully
unit-admission-webhook-0: 15:50:10 INFO unit.admission-webhook/0.juju-log Pebble plan updated with new configuration, replanning
unit-admission-webhook-0: 15:50:11 ERROR unit.admission-webhook/0.juju-log Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/venv/charmed_kubeflow_chisme/pebble/_update_layer.py", line 31, in update_layer
    container.replan()
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/venv/ops/model.py", line 2252, in replan
    self._pebble.replan_services()
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/venv/ops/pebble.py", line 1993, in replan_services
    return self._services_action('replan', [], timeout, delay)
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/venv/ops/pebble.py", line 2090, in _services_action
    raise ChangeError(change.err, change)
ops.pebble.ChangeError: cannot perform the following tasks:
- Start service "base-admission-webhook" (cannot start service: exited quickly with code 0)
----- Logs from task 1 -----
2024-09-13T15:50:11Z INFO Most recent service output:
    {"level":"info","ts":"2024-09-13T15:50:11Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
    I0913 15:50:11.802163      21 main.go:771] About to start serving webhooks: &http.Server{Addr:":4443", Handler:http.Handler(nil), DisableGeneralOptionsHandler:false, TLSConfig:(*tls.Config)(0xc0001f0d00), ReadTimeout:0, ReadHeaderTimeout:0, WriteTimeout:0, IdleTimeout:0, MaxHeaderBytes:0, TLSNextProto:map[string]func(*http.Server, *tls.Conn, http.Handler)(nil), ConnState:(func(net.Conn, http.ConnState))(nil), ErrorLog:(*log.Logger)(nil), BaseContext:(func(net.Listener) context.Context)(nil), ConnContext:(func(context.Context, net.Conn) context.Context)(nil), inShutdown:atomic.Bool{_:atomic.noCopy{}, v:0x0}, disableKeepAlives:atomic.Bool{_:atomic.noCopy{}, v:0x0}, nextProtoOnce:sync.Once{done:0x0, m:sync.Mutex{state:0, sema:0x0}}, nextProtoErr:error(nil), mu:sync.Mutex{state:0, sema:0x0}, listeners:map[*net.Listener]struct {}(nil), activeConn:map[*http.conn]struct {}(nil), onShutdown:[]func()(nil), listenerGroup:sync.WaitGroup{noCopy:sync.noCopy{}, state:atomic.Uint64{_:atomic.noCopy{}, _:atomic.align64{}, v:0x0}, sema:0x0}}
    {"level":"info","ts":"2024-09-13T15:50:11Z","logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"}
2024-09-13T15:50:11Z ERROR cannot start service: exited quickly with code 0
-----

unit-admission-webhook-0: 15:50:11 ERROR unit.admission-webhook/0.juju-log Failed to handle <InstallEvent via AdmissionWebhookCharm/on/install[1]> with error: Failed to replan
unit-admission-webhook-0: 15:50:12 INFO unit.admission-webhook/0.juju-log HTTP Request: GET https://<IP_ADDRESS>/api/v1/namespaces/kubeflow/services/admission-webhook "HTTP/1.1 200 OK"
unit-admission-webhook-0: 15:50:12 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://<IP_ADDRESS>/api/v1/namespaces/kubeflow/services/admission-webhook "HTTP/1.1 200 OK"
unit-admission-webhook-0: 15:50:12 INFO unit.admission-webhook/0.juju-log Kubernetes service 'admission-webhook' patched successfully
unit-admission-webhook-0: 15:50:12 INFO juju.worker.uniter.operation ran "install" hook (via hook dispatching script: dispatch)
unit-admission-webhook-0: 15:50:13 INFO unit.admission-webhook/0.juju-log logging:1: Generating certificates if missing.
unit-admission-webhook-0: 15:50:13 INFO unit.admission-webhook/0.juju-log logging:1: Certificates already exist.
unit-admission-webhook-0: 15:50:13 INFO juju.worker.uniter.operation ran "logging-relation-created" hook (via hook dispatching script: dispatch)
unit-admission-webhook-0: 15:50:13 INFO juju.worker.uniter found queued "leader-elected" hook
unit-admission-webhook-0: 15:50:14 INFO unit.admission-webhook/0.juju-log Generating certificates if missing.
unit-admission-webhook-0: 15:50:14 INFO unit.admission-webhook/0.juju-log Certificates already exist.
unit-admission-webhook-0: 15:50:14 INFO juju.worker.uniter.operation ran "leader-elected" hook (via hook dispatching script: dispatch)
unit-admission-webhook-0: 15:50:15 INFO unit.admission-webhook/0.juju-log Generating certificates if missing.
unit-admission-webhook-0: 15:50:15 INFO unit.admission-webhook/0.juju-log Certificates already exist.
unit-admission-webhook-0: 15:50:15 INFO unit.admission-webhook/0.juju-log HTTP Request: GET https://<IP_ADDRESS>/apis/apiextensions.k8s.io/v1/customresourcedefinitions "HTTP/1.1 200 OK"
unit-admission-webhook-0: 15:50:15 INFO unit.admission-webhook/0.juju-log Rendering manifests
unit-admission-webhook-0: 15:50:15 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://<IP_ADDRESS>/apis/rbac.authorization.k8s.io/v1/clusterroles/admission-webhook?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-admission-webhook-0: 15:50:15 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://<IP_ADDRESS>/apis/rbac.authorization.k8s.io/v1/clusterroles/admission-webhook-kubeflow-poddefaults-admin?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-admission-webhook-0: 15:50:15 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://<IP_ADDRESS>/apis/rbac.authorization.k8s.io/v1/clusterroles/admission-webhook-kubeflow-poddefaults-edit?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-admission-webhook-0: 15:50:15 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://<IP_ADDRESS>/apis/rbac.authorization.k8s.io/v1/clusterroles/admission-webhook-kubeflow-poddefaults-view?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-admission-webhook-0: 15:50:16 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://<IP_ADDRESS>/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/admission-webhook?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-admission-webhook-0: 15:50:16 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://<IP_ADDRESS>/apis/admissionregistration.k8s.io/v1/mutatingwebhookconfigurations/admission-webhook?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-admission-webhook-0: 15:50:16 INFO unit.admission-webhook/0.juju-log Reconcile completed successfully
unit-admission-webhook-0: 15:50:16 INFO unit.admission-webhook/0.juju-log HTTP Request: GET https://<IP_ADDRESS>/apis/apiextensions.k8s.io/v1/customresourcedefinitions "HTTP/1.1 200 OK"
unit-admission-webhook-0: 15:50:16 INFO unit.admission-webhook/0.juju-log Rendering manifests
unit-admission-webhook-0: 15:50:17 INFO unit.admission-webhook/0.juju-log HTTP Request: PATCH https://<IP_ADDRESS>/apis/apiextensions.k8s.io/v1/customresourcedefinitions/poddefaults.kubeflow.org?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-admission-webhook-0: 15:50:17 INFO unit.admission-webhook/0.juju-log Reconcile completed successfully
unit-admission-webhook-0: 15:50:17 INFO unit.admission-webhook/0.juju-log Pebble plan updated with new configuration, replanning
unit-admission-webhook-0: 15:50:17 ERROR unit.admission-webhook/0.juju-log Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/venv/charmed_kubeflow_chisme/pebble/_update_layer.py", line 31, in update_layer
    container.replan()
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/venv/ops/model.py", line 2252, in replan
    self._pebble.replan_services()
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/venv/ops/pebble.py", line 1993, in replan_services
    return self._services_action('replan', [], timeout, delay)
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/venv/ops/pebble.py", line 2090, in _services_action
    raise ChangeError(change.err, change)
ops.pebble.ChangeError: cannot perform the following tasks:
- Start service "base-admission-webhook" (cannot start service: exited quickly with code 0)
----- Logs from task 0 -----
2024-09-13T15:50:17Z INFO Service "admission-webhook" already started.
----- Logs from task 1 -----
2024-09-13T15:50:17Z INFO Most recent service output:
    {"level":"info","ts":"2024-09-13T15:50:17Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
    I0913 15:50:17.199753      32 main.go:771] About to start serving webhooks: &http.Server{Addr:":4443", Handler:http.Handler(nil), DisableGeneralOptionsHandler:false, TLSConfig:(*tls.Config)(0xc000502680), ReadTimeout:0, ReadHeaderTimeout:0, WriteTimeout:0, IdleTimeout:0, MaxHeaderBytes:0, TLSNextProto:map[string]func(*http.Server, *tls.Conn, http.Handler)(nil), ConnState:(func(net.Conn, http.ConnState))(nil), ErrorLog:(*log.Logger)(nil), BaseContext:(func(net.Listener) context.Context)(nil), ConnContext:(func(context.Context, net.Conn) context.Context)(nil), inShutdown:atomic.Bool{_:atomic.noCopy{}, v:0x0}, disableKeepAlives:atomic.Bool{_:atomic.noCopy{}, v:0x0}, nextProtoOnce:sync.Once{done:0x0, m:sync.Mutex{state:0, sema:0x0}}, nextProtoErr:error(nil), mu:sync.Mutex{state:0, sema:0x0}, listeners:map[*net.Listener]struct {}(nil), activeConn:map[*http.conn]struct {}(nil), onShutdown:[]func()(nil), listenerGroup:sync.WaitGroup{noCopy:sync.noCopy{}, state:atomic.Uint64{_:atomic.noCopy{}, _:atomic.align64{}, v:0x0}, sema:0x0}}
    {"level":"info","ts":"2024-09-13T15:50:17Z","logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"}
2024-09-13T15:50:17Z ERROR cannot start service: exited quickly with code 0
-----


Bot Suggestion

Suggestion Message

People should be able to decide when exactly the AutlHealthScreening message is sent, since there are different times when school starts for everyone.

Suggestion Metadata
Bug Reported By: Reaper#6666<PHONE_NUMBER>33464832)
Bug Reported At: Monday, March 7, 2022, 8:38:32 AM Eastern Standard Time

Already exists.


Integration Test - Release - 2019/02/21 21:58:35

Auto create for integration test. AnotherAuthorUpdatedArticleFlow

@PRMergerTest3 : Thanks for your contribution! The author, @v-caxian, has been notified to review your proposed change.


Added my profile

Thank You for you PR it appears the Prettier action has failed.
If you need any help let us know 🤓
Please add only to the bottom of the file. It makes it easier to see what has changed. 😄

I think the problem is your own checks not my problem

I think the checks are failing since they can't find the forked repository.

@Vyvy-vi your probably right.

From the log:

Error: A branch or tag with the name 'patch-1' could not be found


MS Graph Explorer returning 400 instead 200 for passwordlessMicrosoftAuthenticator

Describe the bug
A clear and concise description of what the bug is.
Using the MS Graph GET request: https://graph.microsoft.com/beta/policies/authenticationMethodsPolicy/authenticationMethodConfigurations/passwordlessMicrosoftAuthenticator  is giving a 400 error on Graph Explorer with no info or request IDs.
While running Fiddler at the same time, we see that we get a 200 OK in the trace.
To Reproduce
Steps to reproduce the behavior:

Go to 'Graph Explorer'
Click on 'Run Query'
Scroll down to 'The Response Preview'
See error: Error 400. With no Body or information.

Expected behavior
A clear and concise description of what you expected to happen.
The expected behavior is for this API request to Work or if not, provide the error description.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: [e.g. iOS] Windows 10
Browser [e.g. chrome, safari]: Edge, Chrome
Version [e.g. 22]: Latest

Smartphone (please complete the following information):

Device: [e.g. iPhone6]: N/A
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.
AB#7475

@mifarca The 400-Bad Request Error was returned because the request seems to be deprecated. This is indicated in the docs here
A possible solution could be running this method instead
https://graph.microsoft.com/beta/policies/authenticationMethodsPolicy/authenticationMethodConfigurations/microsoftAuthenticator
Find the link to the documentation here

@mifarca The 400-Bad Request Error was returned because the request seems to be deprecated. This is indicated in the docs here
A possible solution could be running this method instead
https://graph.microsoft.com/beta/policies/authenticationMethodsPolicy/authenticationMethodConfigurations/microsoftAuthenticator
Find the link to the documentation here

Closing this issue.

Closing this issue.


Configure coverage integration. Add package.json to track JS dependen…

…cies.

Changes Unknown when pulling 80c3f1eedbc3552b13e1d3f11332615a1a23a23e on coverage into ** on master**.


Deployment fixes

Purpose
Closes #286 (Fix default value for azureOpenAIChatGptModelName)
Closes #287 (Fix bash script if nesting)
Closes #289 (Fix bash script including ' in files argument value)
Does this introduce a breaking change?
[ ] Yes
[x] No

Pull Request Type
[x] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:

How to Test

Get the code

git clone [repo-address]
cd [repo-name]
git checkout [branch-name]

Open in devcontainer

Test the code

azd up

What to Check
Verify that the following are valid

Deployment succeeds

@SvenAelterman Thanks for the fixing PR


Primary and Secondary DNS

Can you please show me a working code to display the DNS address?

Can you please show me a working code to display the DNS address?

It's been a while since I've done anything PS3 related but something like this should be appropriate. Let me know if it works or not.
#include <vsh/xsetting.hpp>

xsetting_F48C0548_t* net = xsetting_F48C0548();
xsetting_F48C0548_t::net_info_t netInfo;
int error = net->GetNetworkConfig(&netInfo);
vsh::printf("GetNetworkConfig() returns %d and the primary DNS is %s and the secondary DNS is %s\n", error, netInfo.primaryDns, netInfo.secondaryDns);

Unfortunately, no. Just by including #include <vsh/xsetting.hpp>, the compiler shows a lot of errors. But let me explain what I'm trying to achieve.
I need Jordy's plugin - https://github.com/LuanTeles/ip_text - to be able to retrieve the DNS because I need it to display custom text depending on the DNS address. This will reflect the name of the custom server based on the DNS of it.

Unfortunately, no. Just by including #include <vsh/xsetting.hpp>, the compiler shows a lot of errors. But let me explain what I'm trying to achieve.
I need Jordy's plugin - https://github.com/LuanTeles/ip_text - to be able to retrieve the DNS because I need it to display custom text depending on the DNS address. This will reflect the name of the custom server based on the DNS of it.

I just added that exact same code to your project in xmb_plugin.cpp and it compiled with no errors
void PrintDns()
{
	xsetting_F48C0548_t* net = xsetting_F48C0548();
	xsetting_F48C0548_t::net_info_t netInfo;
	int error = net->GetNetworkConfig(&netInfo);
	stdc::printf("GetNetworkConfig() returns %d and the primary DNS is %s and the secondary DNS is %s\n", error, netInfo.primaryDns, netInfo.secondaryDns);
}

Thank you so much, without including the lib it's working just fine.

Ah, other thing, the plugin is showing the info while playing ps1 games, how can i hide it while on it?


⚠ Samsung has degraded performance

In cb7df99, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2545 ms

Resolved: Samsung performance has improved in b58e7eb.


Duplicate properties aren't removed in v4

Input
var CleanCSS = require('clean-css');
var input = 'a{font-size: 12px;font-size: 12px;}';
var output = new CleanCSS().minify(input);
console.log(output);

Output v3
{ styles: 'a{font-size:12px}',
  stats: {},
  errors: [],
  warnings: [] }

Output v4
{ styles: 'a{font-size:12px;font-size:12px}',
  stats:
   { efficiency: 0.08571428571428574,
     minifiedSize: 32,
     originalSize: 35,
     timeSpent: 10 },
  errors: [],
  inlinedStylesheets: [],
  warnings: [] }

Any reason why the duplicate properties aren't removed anymore in v4? Or am I missing a setting which does this?

It seems the duplicate properties are only removed in level 2 by https://github.com/jakubpawlowicz/clean-css/blob/master/lib/optimizer/level-2/remove-duplicates.js
Is there any option available which can toggle this behaviour?

You can pass in the following configuration to enable level 2:
var output = new CleanCSS({level: 2}).minify(input);

or if you only want that specific level 2 rule:
var output = new CleanCSS({
  level: 2,
  levels: {
    2: {
      all: false,
      removeDuplicateRules: true,
    },
  },
}).minify(input);

Both result in
{ styles: 'a{font-size:12px}',
  stats:
   { efficiency: 0.5142857142857142,
     minifiedSize: 17,
     originalSize: 35,
     timeSpent: 1 },
  errors: [],
  inlinedStylesheets: [],
  warnings: [] }

Ok, thanks @bgotink!


docs(router): new chapter

I believe this is an acceptable publishable draft.
Let's squash it during the rebase into master. I want to keep the commits in the branch because of an intermediate state with what looks like a bug for brian.

this is mostly merged with the exception of @wardbell's very last change that was force-pushed

closed by 728659ba71a86abcf10a153e96106fba8e69dc81


fix: expression property is unexpectedly added to the other step

Fixes: #385
Fixes: #319
Here is what's happening
https://github.com/KaotoIO/kaoto-next/issues/385#issuecomment-1832627798
So instead of storing model and schema as states and relying on useEffect() and the re-render triggered with it, make it local variable and try to cache with useMemo(). This way the #385 is addressed. But unfortunately #416 is not, would need to address separately.

For context, this comment describes why the issue is happening


The function  attachMediaStream URL.createObjectURL(b): not work on latest version of Chrome

This message appear:
When I will start to working.

Failed to execute 'createObjectURL' on 'URL': No function was found that matched the signature provided.

+1 having the same issue with Chrome 49

apparently occurs when trying to createObjectURL from null. o_remote_stream is null, for some reasons. Have to look at it further tomorrow

This issue was fixed weeks ago: https://github.com/DoubangoTelecom/sipml5/issues/242

This issue was fixed weeks ago: https://github.com/DoubangoTelecom/sipml5/issues/242

This issue was fixed weeks ago: https://github.com/DoubangoTelecom/sipml5/issues/242

This issue was fixed weeks ago: https://github.com/DoubangoTelecom/sipml5/issues/242

This issue was fixed weeks ago: https://github.com/DoubangoTelecom/sipml5/issues/242


[AutoDiff] NFC: document test suite.

Add README files explaining the differentiable programming test suite.
Add lit.local.cfg files so individual tests do not need to add
REQUIRES: asserts in these directories:

test/AutoDiff/compiler_crashers
test/AutoDiff/compiler_crashers_fixed

Motivation: it is important for compiler_crashers and compiler_crashers_fixed tests to require assertions enabled, because many of these tests crash on compiler assertions.

@swift-ci Please test


Add WebIO for examples in Day1/5_generic_programming.ipynb

Maybe not too interesting for the upcoming days but this fixed the WebIO issue for me.

Thanks! Can you also update the Manifest.toml? Opening Julia within the repo (with julia --project=.) and executing
using Pkg
pkg"resolve"

should do the job.

Looks like it already is part of the Manifest.toml. Also, pkg"resolve" performs no changes. But for some reason I need to additionally include it in the Project.toml to be able to run WebIO.install_jupyter_nbextension(). Otherwise I get:
ERROR: LoadError: ArgumentError: Package WebIO not found in current path:
- Run `import Pkg; Pkg.add("WebIO")` to install the WebIO package.

I have no experience with the package manager, so maybe this is as expected.

This is expected. My bad. I forgot that it was already in the manifest.


Image validation

server side image validation fix

doing it in middleware is the proper fix but I have a short time to fix it. so I did this in a quirky way...sorry

create new PR for image upload validation

creating new pull request with fix only


Expose more Zopfli algorithm options

The Option struct, which can be used by client code to instruct how they want Zopfli to compress data, currently has most of its fields private:
https://github.com/zopfli-rs/zopfli/blob/29b9589d459905af24aecc61c1406b3d0d3c724a/src/lib.rs#L29-L46
However, library users may find it useful to change the default values for these private options. For example, lowering numiterations is immensely needed when dealing with big files, because otherwise the optimization could take so long.
I've been using a patch that just makes these fields pub for some time without problems, but to upstream this improvement it would be nice to add some range checks: it does not make sense to set numiterations or blocksplittingmax to too high or negative values, for example.
@shssoichiro may be interested in this improvement, as tweaking these parameters may improve performance and/or compression in OxiPNG. For example, ZopfliPNG chooses the number of iterations like this:
  options.numiterations = insize < 200000
      ? png_options->num_iterations : png_options->num_iterations_large;

Addressed in c789bc4f09527fb628a300dc64115e99e8b93818.


Bug fixed in the Ternary Operators

The initial syntax was wrong. See the changelog.

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the master branch of freeCodeCamp.
[x] None of my changes are plagiarized from another source without proper attribution.
[x] My article does not contain shortened URLs or affiliate links.

Congrats @vladutilie on your first pull request (PR)! 🎉
Thank you for your contribution to the page! 👍
We're happy to accept these changes, and look forward to future contributions. 📝


OC-393 ~ Fixes missing receive.timeout property.

All is well.

All is well.

 All committers of the pull request should sign our Contributor License Agreement in order to get your pull request merged.2 out of 3 committers have signed the CLA.:white_check_mark: villgust:white_check_mark: kevinsmeets:x: web-flow

All is well.


Implement limits and improve memory management

Buffer sizes should be limited and memory management should be improved

This will be included in #10


Poor reported performance of DataFusion against DuckDB and Hyper

Describe the bug
There is a blog post that reports relatively poor performance of DataFusion compared to DuckDB and Hyper:
https://www.architecture-performance.fr/ap_blog/tpc-h-benchmark-of-hyper-duckdb-and-datafusion-on-parquet-files/
To Reproduce
I would like someone to try and reproduce the DataFusion performance reported in the blog and propose ways to improve the performance of DataFusion (perhaps by enabling some of the options that are off by default)
Expected behavior
No response
Additional context
@Dandandan  suggests on slack that with parallelized scan of a parquet file this benchmark may go faster

There was a question on slack asking if the Python bindings enable parallel scans.
The Python SessionContext constructor is here. It uses SessionConfig::default if no config is provided, so yes, parallel scans should be enabled if a recent version is being used.
https://github.com/apache/arrow-datafusion-python/blob/main/src/context.rs#L216-L233

Looking at https://github.com/apache/arrow-datafusion/blob/main/dev/changelog/19.0.0.md , it should be enabled since 19.0.0, while the article mentions it's using 20.0.0

FYI @djfrancesco we are looking into your article.
Is there any code you could share to replicate the results?

I am sorry because the code is a little messy... I might have use DataFusion in a wrong way. I tried several config options but without success. I write these blog posts mostly as notes to myself. I would be glad to update the post with better timings. The main command is the following:

No worries -- clearly if you can't get good performance out of DataFusion we have some things to improve (even if it is only UX).
Thank you for the article

Anybody has a preliminary idea why the figures look this bad? Mostly config related, algorithmic issues or code structure/performance related?

Related (in a sense that trying Datafusion with defaults yields questionable results): https://github.com/apache/arrow-datafusion/discussions/5141

Looks like the blog is broken, I can not open it now.

I took a quick look at the code, and have two observations:

The configs being created are not actually being used anywhere i.e. they are not being passed into the context
There is a comment that DataFusion does not support multiple Parquet files for a table, which is not true, but maybe this means the DataFusion tests are not comparable to the other tests if they are being run against one parquet file per table?

I created a quick PR to improve the docs around creating a context and will spend more time on this over the next few days. The UX could be improved.
https://github.com/apache/arrow-datafusion-python/pull/321

Thanks for your help @andygrove! I tried versions 20 and 21 and got this Error when using a SessionConfig instance:
      config.set("datafusion.execution.parquet.pushdown_filters", "true")
      ^^^^^^^^^^
  AttributeError: 'datafusion.SessionConfig' object has no attribute 'set'

Maybe it is a new feature ?

Hi @djfrancesco, how many partitions do you set for each engine? Are they the same? And could you share the cpu usage?

FWIW I am running the scripts in https://github.com/aetperf/sql_olap_bench now on my own machine (they are very nicely written @djfrancesco ) and will report back any interesting findings

So I have two interesting pieces of information:

I can reproduce the reported performance difference on my 8 core cloud machine

I also see DataFusion using only a single core
+--------------+--------------+----------+-----------------+-------------------+---------------------+--------------------+--------------+----------+-------------+
| l_returnflag | l_linestatus | sum_qty  | sum_base_price  | sum_disc_price    | sum_charge          | avg_qty            | avg_price    | avg_disc | count_order |
+--------------+--------------+----------+-----------------+-------------------+---------------------+--------------------+--------------+----------+-------------+
| A            | F            | 37734107 |<PHONE_NUMBER>0.73  |<PHONE_NUMBER>4.8700  |<PHONE_NUMBER>2.827692  | 25.522005853257337 | 38273.129734 | 0.049985 | 1478493     |
| N            | F            | 991417   |<PHONE_NUMBER>.38   |<PHONE_NUMBER>.0541   |<PHONE_NUMBER>.194375   | 25.516471920522985 | 38284.467760 | 0.050093 | 38854       |
| N            | O            | 74476040 |<PHONE_NUMBER>97.74 |<PHONE_NUMBER>07.6056 |<PHONE_NUMBER>72.497010 | 25.50222676958499  | 38249.117988 | 0.049996 | 2920374     |
| R            | F            | 37719753 |<PHONE_NUMBER>0.90  |<PHONE_NUMBER>4.6040  |<PHONE_NUMBER>9.831932  | 25.50579361269077  | 38250.854626 | 0.050009 | 1478870     |
+--------------+--------------+----------+-----------------+-------------------+---------------------+--------------------+--------------+----------+-------------+
4 rows in set. Query took 2.841 seconds.
❯

However, when I run datafusion against "datafusion created" parquet files from https://github.com/apache/arrow-datafusion/tree/main/benchmarks it is 3x faster though much less fast than hyper (like 100ms vs 928ms)
alamb@aal-dev:~/tpch_data/parquet_data_SF1$ datafusion-cli -f ~/sql_olap_bench/q1.txt
+--------------+--------------+-------------+-----------------+-------------------+---------------------+-----------+--------------+----------+-------------+
| l_returnflag | l_linestatus | sum_qty     | sum_base_price  | sum_disc_price    | sum_charge          | avg_qty   | avg_price    | avg_disc | count_order |
+--------------+--------------+-------------+-----------------+-------------------+---------------------+-----------+--------------+----------+-------------+
| A            | F            | 37734107.00 |<PHONE_NUMBER>0.73  |<PHONE_NUMBER>4.8700  |<PHONE_NUMBER>2.827692  | 25.522005 | 38273.129734 | 0.049985 | 1478493     |
| N            | F            | 991417.00   |<PHONE_NUMBER>.38   |<PHONE_NUMBER>.0541   |<PHONE_NUMBER>.194375   | 25.516471 | 38284.467760 | 0.050093 | 38854       |
| N            | O            | 74476040.00 |<PHONE_NUMBER>97.74 |<PHONE_NUMBER>07.6056 |<PHONE_NUMBER>72.497010 | 25.502226 | 38249.117988 | 0.049996 | 2920374     |
| R            | F            | 37719753.00 |<PHONE_NUMBER>0.90  |<PHONE_NUMBER>4.6040  |<PHONE_NUMBER>9.831932  | 25.505793 | 38250.854626 | 0.050009 | 1478870     |
+--------------+--------------+-------------+-----------------+-------------------+---------------------+-----------+--------------+----------+-------------+
4 rows in set. Query took 0.928 seconds.

The explain plan clearly shows DataFusion trying to parallelize the scan:
+---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| logical_plan  | Sort: lineitem.parquet.l_returnflag ASC NULLS LAST, lineitem.parquet.l_linestatus ASC NULLS LAST                                                                                |
|               |   Projection: lineitem.parquet.l_returnflag, lineitem.parquet.l_linestatus, SUM(lineitem.parquet.l_quantity) AS sum_qty, SUM(lineitem.parquet.l_extendedprice) AS sum_base_price, SUM(lineitem.parquet.l_extendedprice * Int64(1) - lineitem.parquet.l_discount) AS sum_disc_price, SUM(lineitem.parquet.l_extendedprice * Int64(1) - lineitem.parquet.l_discount * Int64(1) + lineitem.parquet.l_tax) AS sum_charge, AVG(lineitem.parquet.l_quantity) AS avg_qty, AVG(lineitem.parquet.l_extendedprice) AS avg_price, AVG(lineitem.parquet.l_discount) AS avg_disc, COUNT(UInt8(1)) AS count_order                                                                                |
|               |     Aggregate: groupBy=[[lineitem.parquet.l_returnflag, lineitem.parquet.l_linestatus]], aggr=[[SUM(lineitem.parquet.l_quantity), SUM(lineitem.parquet.l_extendedprice), SUM(CAST(lineitem.parquet.l_extendedprice AS Decimal128(38, 4)) * CAST(Decimal128(Some(100),23,2) - CAST(lineitem.parquet.l_discount AS Decimal128(23, 2)) AS Decimal128(38, 4))CAST(Decimal128(Some(100),23,2) - CAST(lineitem.parquet.l_discount AS Decimal128(23, 2)) AS Decimal128(38, 4))Decimal128(Some(100),23,2) - CAST(lineitem.parquet.l_discount AS Decimal128(23, 2))CAST(lineitem.parquet.l_discount AS Decimal128(23, 2))lineitem.parquet.l_discountDecimal128(Some(100),23,2)CAST(lineitem.parquet.l_extendedprice AS Decimal128(38, 4))lineitem.parquet.l_extendedprice AS lineitem.parquet.l_extendedprice * Decimal128(Some(100),23,2) - lineitem.parquet.l_discount) AS SUM(lineitem.parquet.l_extendedprice * Int64(1) - lineitem.parquet.l_discount), SUM(CAST(CAST(lineitem.parquet.l_extendedprice AS Decimal128(38, 4)) * CAST(Decimal128(Some(100),23,2) - CAST(lineitem.parquet.l_discount AS Decimal128(23, 2)) AS Decimal128(38, 4))CAST(Decimal128(Some(100),23,2) - CAST(lineitem.parquet.l_discount AS Decimal128(23, 2)) AS Decimal128(38, 4))Decimal128(Some(100),23,2) - CAST(lineitem.parquet.l_discount AS Decimal128(23, 2))CAST(lineitem.parquet.l_discount AS Decimal128(23, 2))lineitem.parquet.l_discountDecimal128(Some(100),23,2)CAST(lineitem.parquet.l_extendedprice AS Decimal128(38, 4))lineitem.parquet.l_extendedprice AS lineitem.parquet.l_extendedprice * Decimal128(Some(100),23,2) - lineitem.parquet.l_discount AS Decimal128(38, 6)) * CAST(Decimal128(Some(100),23,2) + CAST(lineitem.parquet.l_tax AS Decimal128(23, 2)) AS Decimal128(38, 6))) AS SUM(lineitem.parquet.l_extendedprice * Int64(1) - lineitem.parquet.l_discount * Int64(1) + lineitem.parquet.l_tax), AVG(lineitem.parquet.l_quantity), AVG(lineitem.parquet.l_extendedprice), AVG(lineitem.parquet.l_discount), COUNT(UInt8(1))]] |
|               |       Projection: CAST(lineitem.parquet.l_extendedprice AS Decimal128(38, 4)) * CAST(Decimal128(Some(100),23,2) - CAST(lineitem.parquet.l_discount AS Decimal128(23, 2)) AS Decimal128(38, 4)) AS CAST(lineitem.parquet.l_extendedprice AS Decimal128(38, 4)) * CAST(Decimal128(Some(100),23,2) - CAST(lineitem.parquet.l_discount AS Decimal128(23, 2)) AS Decimal128(38, 4))CAST(Decimal128(Some(100),23,2) - CAST(lineitem.parquet.l_discount AS Decimal128(23, 2)) AS Decimal128(38, 4))Decimal128(Some(100),23,2) - CAST(lineitem.parquet.l_discount AS Decimal128(23, 2))CAST(lineitem.parquet.l_discount AS Decimal128(23, 2))lineitem.parquet.l_discountDecimal128(Some(100),23,2)CAST(lineitem.parquet.l_extendedprice AS Decimal128(38, 4))lineitem.parquet.l_extendedprice, lineitem.parquet.l_quantity, lineitem.parquet.l_extendedprice, lineitem.parquet.l_discount, lineitem.parquet.l_tax, lineitem.parquet.l_returnflag, lineitem.parquet.l_linestatus                                                                                |
|               |         Filter: lineitem.parquet.l_shipdate <= Date32("10471")                                                                                |
|               |           TableScan: lineitem.parquet projection=[l_quantity, l_extendedprice, l_discount, l_tax, l_returnflag, l_linestatus, l_shipdate], partial_filters=[lineitem.parquet.l_shipdate <= Date32("10471")]                                                                                |
| physical_plan | SortPreservingMergeExec: [l_returnflag@0 ASC NULLS LAST,l_linestatus@1 ASC NULLS LAST]                                                                                |
|               |   SortExec: expr=[l_returnflag@0 ASC NULLS LAST,l_linestatus@1 ASC NULLS LAST]                                                                                |
|               |     ProjectionExec: expr=[l_returnflag@0 as l_returnflag, l_linestatus@1 as l_linestatus, SUM(lineitem.parquet.l_quantity)@2 as sum_qty, SUM(lineitem.parquet.l_extendedprice)@3 as sum_base_price, SUM(lineitem.parquet.l_extendedprice * Int64(1) - lineitem.parquet.l_discount)@4 as sum_disc_price, SUM(lineitem.parquet.l_extendedprice * Int64(1) - lineitem.parquet.l_discount * Int64(1) + lineitem.parquet.l_tax)@5 as sum_charge, AVG(lineitem.parquet.l_quantity)@6 as avg_qty, AVG(lineitem.parquet.l_extendedprice)@7 as avg_price, AVG(lineitem.parquet.l_discount)@8 as avg_disc, COUNT(UInt8(1))@9 as count_order]                                                                                |
|               |       AggregateExec: mode=FinalPartitioned, gby=[l_returnflag@0 as l_returnflag, l_linestatus@1 as l_linestatus], aggr=[SUM(lineitem.parquet.l_quantity), SUM(lineitem.parquet.l_extendedprice), SUM(lineitem.parquet.l_extendedprice * Int64(1) - lineitem.parquet.l_discount), SUM(lineitem.parquet.l_extendedprice * Int64(1) - lineitem.parquet.l_discount * Int64(1) + lineitem.parquet.l_tax), AVG(lineitem.parquet.l_quantity), AVG(lineitem.parquet.l_extendedprice), AVG(lineitem.parquet.l_discount), COUNT(UInt8(1))]                                                                                |
|               |         CoalesceBatchesExec: target_batch_size=8192                                                                                |
|               |           RepartitionExec: partitioning=Hash([Column { name: "l_returnflag", index: 0 }, Column { name: "l_linestatus", index: 1 }], 8), input_partitions=8                                                                                |
|               |             AggregateExec: mode=Partial, gby=[l_returnflag@5 as l_returnflag, l_linestatus@6 as l_linestatus], aggr=[SUM(lineitem.parquet.l_quantity), SUM(lineitem.parquet.l_extendedprice), SUM(lineitem.parquet.l_extendedprice * Int64(1) - lineitem.parquet.l_discount), SUM(lineitem.parquet.l_extendedprice * Int64(1) - lineitem.parquet.l_discount * Int64(1) + lineitem.parquet.l_tax), AVG(lineitem.parquet.l_quantity), AVG(lineitem.parquet.l_extendedprice), AVG(lineitem.parquet.l_discount), COUNT(UInt8(1))]                                                                                |
|               |               ProjectionExec: expr=[CAST(l_extendedprice@1 AS Decimal128(38, 4)) * CAST(Some(100),23,2 - CAST(l_discount@2 AS Decimal128(23, 2)) AS Decimal128(38, 4)) as CAST(lineitem.parquet.l_extendedprice AS Decimal128(38, 4)) * CAST(Decimal128(Some(100),23,2) - CAST(lineitem.parquet.l_discount AS Decimal128(23, 2)) AS Decimal128(38, 4))CAST(Decimal128(Some(100),23,2) - CAST(lineitem.parquet.l_discount AS Decimal128(23, 2)) AS Decimal128(38, 4))Decimal128(Some(100),23,2) - CAST(lineitem.parquet.l_discount AS Decimal128(23, 2))CAST(lineitem.parquet.l_discount AS Decimal128(23, 2))lineitem.parquet.l_discountDecimal128(Some(100),23,2)CAST(lineitem.parquet.l_extendedprice AS Decimal128(38, 4))lineitem.parquet.l_extendedprice, l_quantity@0 as l_quantity, l_extendedprice@1 as l_extendedprice, l_discount@2 as l_discount, l_tax@3 as l_tax, l_returnflag@4 as l_returnflag, l_linestatus@5 as l_linestatus]                                                                                |
|               |                 CoalesceBatchesExec: target_batch_size=8192                                                                                |
|               |                   FilterExec: l_shipdate@6 <= 10471                                                                                |
|               |                     ParquetExec: limit=None, partitions={8 groups: [[home/alamb/tpch_data/parquet_data_SF1/lineitem.parquet:0..20797508], [home/alamb/tpch_data/parquet_data_SF1/lineitem.parquet:20797508..41595016], [home/alamb/tpch_data/parquet_data_SF1/lineitem.parquet:41595016..62392524], [home/alamb/tpch_data/parquet_data_SF1/lineitem.parquet:62392524..83190032], [home/alamb/tpch_data/parquet_data_SF1/lineitem.parquet:83190032..103987540], [home/alamb/tpch_data/parquet_data_SF1/lineitem.parquet:103987540..124785048], [home/alamb/tpch_data/parquet_data_SF1/lineitem.parquet:124785048..145582556], [home/alamb/tpch_data/parquet_data_SF1/lineitem.parquet:145582556..166380062]]}, predicate=l_shipdate@10 <= 10471, pruning_predicate=l_shipdate_min@0 <= 10471, projection=[l_quantity, l_extendedprice, l_discount, l_tax, l_returnflag, l_linestatus, l_shipdate]                                                                                |
|               |                                                                                |
+---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Here is the duckdb generated schema:
❯  create external table lineitem stored as parquet location 'lineitem.parquet';
0 rows in set. Query took 0.003 seconds.
❯ describe lineitem;
+-----------------+-------------------+-------------+
| column_name     | data_type         | is_nullable |
+-----------------+-------------------+-------------+
| l_orderkey      | Int32             | YES         |
| l_partkey       | Int32             | YES         |
| l_suppkey       | Int32             | YES         |
| l_linenumber    | Int32             | YES         |
| l_quantity      | Int32             | YES         |
| l_extendedprice | Decimal128(15, 2) | YES         |
| l_discount      | Decimal128(15, 2) | YES         |
| l_tax           | Decimal128(15, 2) | YES         |
| l_returnflag    | Utf8              | YES         |
| l_linestatus    | Utf8              | YES         |
| l_shipdate      | Date32            | YES         |
| l_commitdate    | Date32            | YES         |
| l_receiptdate   | Date32            | YES         |
| l_shipinstruct  | Utf8              | YES         |
| l_shipmode      | Utf8              | YES         |
| l_comment       | Utf8              | YES         |
+-----------------+-------------------+-------------+
16 rows in set. Query took 0.001 seconds.

Here is the datafusion generated schema:
❯ describe lineitem;
+-----------------+-------------------+-------------+
| column_name     | data_type         | is_nullable |
+-----------------+-------------------+-------------+
| l_orderkey      | Int64             | NO          |
| l_partkey       | Int64             | NO          |
| l_suppkey       | Int64             | NO          |
| l_linenumber    | Int32             | NO          |
| l_quantity      | Decimal128(15, 2) | NO          |
| l_extendedprice | Decimal128(15, 2) | NO          |
| l_discount      | Decimal128(15, 2) | NO          |
| l_tax           | Decimal128(15, 2) | NO          |
| l_returnflag    | Utf8              | NO          |
| l_linestatus    | Utf8              | NO          |
| l_shipdate      | Date32            | NO          |
| l_commitdate    | Date32            | NO          |
| l_receiptdate   | Date32            | NO          |
| l_shipinstruct  | Utf8              | NO          |
| l_shipmode      | Utf8              | NO          |
| l_comment       | Utf8              | NO          |
+-----------------+-------------------+-------------+

So it seems to me there are two follow ups to explore:

Why is datafusion only using one core on this single large file?
Why is the time different with the two files?

I will write a smaller self contained reproducer for the "slow query on a single file" and maybe someone clever can look at that.

I wonder if running parquet-layout against the parquet file might prove insightful.
DataFusion is currently limited to row group level parallelism, and there certainly are parquet writers that write very large row groups which would cause issues for this - https://github.com/apache/arrow/issues/34280

The layout is on: https://github.com/apache/arrow-datafusion/issues/5995#issuecomment-1507321881

So it seems to me there are two follow ups to explore:

Why is datafusion only using one core on this single large file?
Why is the time different with the two files?

Why Decimal128 is introduced for l_quantity in Datafusion but DuckDB uses int32?

Why Decimal128 is introduced for l_quantity in Datafusion but DuckDB uses int32?

I do not know -- the parquet files for these tests are created within duckdb somehow (it looks like maybe they have included the dbgen data generator in their system natively)

Filed https://github.com/apache/arrow-datafusion/issues/5995 with a reproducer for the "one core scanning" issue

the parquet files for these tests are created within duckdb somehow (it looks like maybe they have included the dbgen data generator in their system natively)

Yes I used the DuckDB tpch extension : https://github.com/duckdb/duckdb/tree/master/extension/tpch
Each table is exported as a Parquet file using a COPY ( ) TO, with row group size 122880 (default in DuckDB) and snappy compression

https://github.com/apache/arrow-datafusion/pull/5997 should help with this, the file range logic was partitioning row groups based on the location of their ColumnMetadata, which is normally written at the end of a ColumnChunk, not their actual data. This causes issues because duckdb appears to not write the un-inlined metadata and instead just writes a file offset of 0. Technically this is probably not spec-compliant, but more reliable to partition based on where the actual page data is.

So it seems to me there are two follow ups to explore:

Why is datafusion only using one core on this single large file?
Why is the time different with the two files?

Why Decimal128 is introduced for l_quantity in Datafusion but DuckDB uses int32?

I think l_quantity should be of type Decimal.  I checked the schema generated by SparkSQL tpch tool.

col_name
data_type
comment

l_orderkey
bigint
null

l_partkey
bigint
null

l_suppkey
bigint
null

l_linenumber
int
null

l_quantity
decimal(12,2)
null

l_extendedprice
decimal(12,2)
null

l_discount
decimal(12,2)
null

l_tax
decimal(12,2)
null

l_returnflag
string
null

l_linestatus
string
null

l_commitdate
date
null

l_receiptdate
date
null

l_shipinstruct
string
null

l_shipmode
string
null

l_comment
string
null

l_shipdate
date
null

So it seems to me there are two follow ups to explore:

Why is datafusion only using one core on this single large file?
Why is the time different with the two files?

Why Decimal128 is introduced for l_quantity in Datafusion but DuckDB uses int32?

I think l_quantity should be of type Decimal.  I checked the schema generated by SparkSQL tpch tool.

col_name
data_type
comment

l_orderkey
bigint
null

l_partkey
bigint
null

l_suppkey
bigint
null

l_linenumber
int
null

l_quantity
decimal(12,2)
null

l_extendedprice
decimal(12,2)
null

l_discount
decimal(12,2)
null

l_tax
decimal(12,2)
null

l_returnflag
string
null

l_linestatus
string
null

l_commitdate
date
null

l_receiptdate
date
null

l_shipinstruct
string
null

l_shipmode
string
null

l_comment
string
null

l_shipdate
date
null

I wonder if running parquet-layout against the parquet file might prove insightful.
DataFusion is currently limited to row group level parallelism, and there certainly are parquet writers that write very large row groups which would cause issues for this - apache/arrow#34280. Longer-term I would like to eventually get back to #2504 but that is not likely in the next couple of months.

The flexibility of the parquet file causes different Writers to use different file generation strategies. The data in a Parquet file can be spread over the row groups and the pages using any encoding and compression the writer or user wants.
If the physical layout of the parquet file affects the way different query engines scan, should we introduce a standard TPC-H Parquet file and re-run the performance comparison test?
I also saw this issue in this paper: https://dl.gi.de/bitstream/handle/20.500.12116/40316/B3-1.pdf?sequence=1&isAllowed=y

we look at three different Parquet writers to show how much Parquet files differ even though they store the same data. Parquet Writer Comparison:

Generator
Rows per Row Group
Pages per Row Group
File Sizes(SF1,SF10,SF100)

Spark
3,000,000
150
192 MB, 2.1 GB, 20 GB

Spark uncompressed
3,000,000
150
333 MB, 3.3 GB, 33 GB

DuckDB
100,352
1
281 MB, 2.8 GB, 28 GB

Arrow
67,108,864
15 - 1800
189 MB, 2.0 GB, 20 GB

For each generator, we measure the number of rows and the number of pages that are stored per row group. The Spark and DuckDB Parquet writers store a fixed number of elements per page and a fixed number of pages per row group. Since Parquet does not force synchronization between the column chunks, there are writers such as Arrow that do not store the same number of elements per page. Arrow uses a fixed data page size between roughly 0.5MB and 1 MB. For DuckDB and Spark, the page sizes vary from 0.5 MB to 6 MB.
Even though we only cover three different Parquet writers, we have already observed two extremes. DuckDB and Arrow do not take advantage of the hierarchical data layout: DuckDB will only use one page per row group, and Arrow stores the entire dataset in one row group for scale factor 1 and 10 since each row group stores 67 million rows.

I think l_quantity should be of type Decimal. I checked the schema generated by SparkSQL tpch tool.

It is also what is specified in the TPC-H document pdf:
Column Name	Datatype Requirements
L_QUANTITY	decimal

In the TPC-H documentation, l_quantity is defined as a decimal type.
However, duckdb incorrectly defines it as an INTEGER type, as can be seen in the duckdb source code: https://github.com/duckdb/duckdb/blob/eaf507009fd06573d74cc5742a8643481883a0ff/extension/tpch/dbgen/dbgen.cpp#L405-L415

Arrow stores the entire dataset in one row group for scale factor 1 and 10 since each row group stores 67 million rows

FWIW this was a bug and should be fixed by

DuckDB not making use of the hierarchical layout is perplexing, I wonder if this was an intentional design decision, it certainly isn't in the spirit of the specification

I don't understand this statement
This is my understanding of the difference:
┌──────────────────────────────────┐         ┌──────────────────────────────────┐
│        RowGroup Metadata         │         │        RowGroup Metadata         │
│                                  │         │                                  │
├──────────────────────────────────┤         ├──────────────────────────────────┤
│ ┌───────────────────────────────┐│         │        RowGroup Metadata         │
│ │                               ││         │                                  │
│ └───────────────────────────────┘│         ├──────────────────────────────────┤
│ ┌───────────────────────────────┐│         │        RowGroup Metadata         │
│ │                               ││         │                                  │
│ └───────────────────────────────┘│         └──────────────────────────────────┘
│ ┌───────────────────────────────┐│         ┌──────────────────────────────────┐
│ │                               ││         │ ┌───────────────────────────────┐│
│ └───────────────────────────────┘│         │ │                               ││
│         RowGroup Data            │         │ └───────────────────────────────┘│
└──────────────────────────────────┘         │ ┌───────────────────────────────┐│
┌──────────────────────────────────┐         │ │                               ││
│        RowGroup Metadata         │         │ └───────────────────────────────┘│
│                                  │         │ ┌───────────────────────────────┐│
├──────────────────────────────────┤         │ │                               ││
│ ┌───────────────────────────────┐│         │ └───────────────────────────────┘│
│ │                               ││         │         RowGroup Data            │
│ └───────────────────────────────┘│         └──────────────────────────────────┘
│ ┌───────────────────────────────┐│         ┌──────────────────────────────────┐
│ │                               ││         │ ┌───────────────────────────────┐│
│ └───────────────────────────────┘│         │ │                               ││
│ ┌───────────────────────────────┐│         │ └───────────────────────────────┘│
│ │                               ││         │ ┌───────────────────────────────┐│
│ └───────────────────────────────┘│         │ │                               ││
│         RowGroup Data            │         │ └───────────────────────────────┘│
└──────────────────────────────────┘         │ ┌───────────────────────────────┐│
┌──────────────────────────────────┐         │ │                               ││
│        RowGroup Metadata         │         │ └───────────────────────────────┘│
│                                  │         │         RowGroup Data            │
├──────────────────────────────────┤         └──────────────────────────────────┘
│ ┌───────────────────────────────┐│         ┌──────────────────────────────────┐
│ │                               ││         │ ┌───────────────────────────────┐│
│ └───────────────────────────────┘│         │ │                               ││
│ ┌───────────────────────────────┐│         │ └───────────────────────────────┘│
│ │                               ││         │ ┌───────────────────────────────┐│
│ └───────────────────────────────┘│         │ │                               ││
│ ┌───────────────────────────────┐│         │ └───────────────────────────────┘│
│ │                               ││         │ ┌───────────────────────────────┐│
│ └───────────────────────────────┘│         │ │                               ││
│         RowGroup Data            │         │ └───────────────────────────────┘│
└──────────────────────────────────┘         │         RowGroup Data            │
                                             └──────────────────────────────────┘

               Arrow                                        DuckDB

Wouldn't the DuckDB implementation actually potentially be better (as it allows the metadata to be fetched with one contiguous IO.
Perhaps I am mis understanding

I don't understand this statement

Was in response so a single page per row group, not the fact it also has a dubious interpretation of the statistics specification. Inlining statistics makes a lot of sense, single page per row group seems to sacrifice a lot

Proposal for making the settings "fast by default": https://github.com/apache/arrow-datafusion/issues/6287

I think we have addressed this particular concern a long time ago -- we can work on additional performance improvements as we find them. CLosing this one as I don't think it is actionable now


Possible regression of #8815 ("Nordic: Directly accessing GPIOTE might create unstable firmware"...)

With Zephyr 1.13 I ran into a similar issue as described in #8252, which was actually fixed in #8815. After a certain point I was unable to receive interrupts on a pin but could confirm with an logic analyzer that the state had changed. Similar to last time the issue was only seen when PWM was used as well. As a workaround, I switched to the hardware based nrfx pwm driver which solved the problem (or at least it is hidden now). I suspect that the rewriting of the nrfx-based gpio shim in combination with the old pwm_sw driver reintroduced overwriting of GPIOTE channels.
cc: @cvinayak

Fixed in #11807


Set ref to fully-formed of the tag when trigger event is release

Fix #23943
When trigger event is release, ref should be like refs/tags/<tag_name> instead of CommitID

Isn't changing the format of ref_name breaking?
If yes, this change shouldn't be backported.
Isn't this going to cause future problems where we cannot distinguish between what is intended for a tag, and what is intended for a branch?

Isn't changing the format of ref_name breaking

Tend to agree, I'd say it is breaking as some workflows may rely on the "broken" behaviour.

I have a different opinion. It was previously incorrect, so what it broke was a mistake. I would say it's not breaking and should be backported.
See the comment.

The fully-formed ref of the branch or tag that triggered the workflow run. For workflows triggered by push, this is the branch or tag ref that was pushed. For workflows triggered by pull_request, this is the pull request merge branch. For workflows triggered by release, this is the release tag created. For other triggers, this is the branch or tag ref that triggered the workflow run. This is only set if a branch or tag is available for the event type. The ref given is fully-formed, meaning that for branches the format is refs/heads/<branch_name>, for pull requests it is refs/pull/<pr_number>/merge, and for tags it is refs/tags/<tag_name>. For example, refs/heads/feature-branch-1.

https://github.com/go-gitea/gitea/blob/ed7245f997e980fe5daa9e0d298539bc5e62ac59/routers/api/actions/runner/utils.go#L100
It follows https://docs.github.com/en/actions/learn-github-actions/contexts#github-context

🎺 🤖

🎺 🤖


Run cairo tests for workspace

Stack:

#508
#506
#502
#500
#490
#489
#465 ⬅

⚠ Part of a stack created by spr. Do not merge manually using the UI - doing so may have unexpected results.

✓ Commit merged in pull request #490


readln error when pipe is used.

Description:
When pipe is used, readln can't get line accurately.
It might ignore previous stdin buffer. So got EOF.
Steps to reproduce:
in.txt
A
B

test.bal
import ballerina/io;

public function main() {
    io:println(io:readln(""));
    io:println(io:readln(""));  #got error
}

and run cat in.txt|ballerina run test.bal or ballerina run test.bal < in.txt ,
and got error
A
error: No line found {}
	at ballerina/io:readln(<native>)
	   main(h.bal:5)

Affected Versions:
Ballerina 0.991.0
OS, DB, other environment details and versions:
Ubuntu 19.04 x64
Related Issues (optional):

Suggested Labels (optional):

Suggested Assignees (optional):

@JP3BGY Thanks for the reporting this bug. We will fix this in the upcoming release, which happen in the end of July.


feat: warn if parameters or placeholders are unused

Closes partially #543
Summary of Changes
Show a warning if a parameter or placeholder is unused and suggest removing it.

:tada: This PR is included in version 0.2.0 :tada:
The release is available on:

Visual Studio Marketplace
GitHub release

Your semantic-release bot :package::rocket:


Remove .envrc from the repository and ignore it

The rationale here is that .envrc is very host-specific (I, for example,
don't use nix, and other people probably don't use asdf in the same way)
Having it in the code like this forces me to ignore it when using
direnv, which makes all kinds of noise in my shell and editors.

Sorry, wrong source branch.


update request deferral handling

improve the logging of request deferrals.  Mostly just log only the first transition to a deferal so the log is not flooded when the tokens all expire.

Hi @jeffmcaffer, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!

It looks like you're a Microsoft contributor (Jeff McAffer). If you're full-time, we DON'T require a Contribution License Agreement. If you are a vendor, please DO sign the electronic Contribution License Agreement. It will take 2 minutes and there's no faxing! https://cla.microsoft.com.

TTYL, MSBOT;


Update doc version

For ref in 493ff42 I had to run
git pull origin gh-pages -X ours

to fix the merge conflicts.


slidesPerView: 'auto',Question

中文用户注意：

尽量用英文描述你的 issue
不要把内容堆彻在标题上，逻辑清晰地写在内容区
贴代码要提前格式化好，有颜色高亮那种，贴文本，不要贴图片
提问题前，必须仔细阅读 REMADE.md + 在已关闭的问题中寻找与自身相关的问题，90% 的可能它已经被解决
如果无法做到提一个合格、优秀的问题，则问题会被 close + block

BUG REPORT TEMPLATE
Vue.js version and component version
component: v2.6.7
Reproduction Link

A minimal JSBin, JSFiddle, Codepen, or a GitHub repository that can reproduce the bug.
You could start with this template: https://jsfiddle.net/39epgLj0/

Steps to reproduce
``
What is Expected?
What is actually happening?
I used slidesPerView: 'auto', ,when it go to last slide ,the mousewheel isn't work,so that I can't see the footer.

I should change the window size, it is normal;when i refresh it will not work

i have the same question,,,when i resize the windows ,it works well. and i try to find the refresh event but failed...

i resolved this problem with observer,,,the link is https://www.swiper.com.cn/api/observer/218.html
{ "observeParents": true, "observer": true }
and maybe you can close this issues

"observeParents": true, "observer": true

Everytime I refreshed the page I would see a single swiper at full width on the screen, and it would only reset by changing the browser width. Adding the above solved the issue and I wouldn't have found it without this post.
This really saved me. Thank you!


Feature/max visit short urls

Add expirable short urls. The user will receive an expired page if the URL is no longer active.

Pls remove generated files from the repo.

Intended. And you agreed to it. 🤪


Static category view

Closes #1132

Codecov Report

Merging #1135 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1135   +/-   ##
=======================================
  Coverage   63.43%   63.43%
=======================================
  Files         107      107
  Lines        5979     5979
  Branches      741      741
=======================================
  Hits         3793     3793
  Misses       2049     2049
  Partials      137      137

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f4d0414...70a49b9. Read the comment docs.

Its not done yet :)
But thx for trust.
Missing part is a frontend magic in category/index.html

TODO:

[ ]  filters menu for mobile site
[ ]  load more or pagination

@koradon @dominik-zeglen We'll replace "Load more" with traditional pagination (links to each page). We should use Bootstrap's markup for rendering pagination and style it to match storefront styles.

@dominik-zeglen It looks like this PR contains work that you've done in another PR (JS code organization). Please remove these changes from here. Also please remove the merge commit. In our flow we use rebasing to update branches.


fortify scan is not triggered

We configured the execution of a fortify scan in the pipeline config.
steps:

 ...
  executeFortifyScan:
    dockerImage: 'my.repo:version'
    fortifyCredentialId: 'FORTIFY_CREDENTIALS'

### Stage-specific configuration
stages:

  fortifyScan:
    sscUrl: 'https://test.ssc/ssc'
    fortifyCredentialId: 'FORTIFY_CREDENTIALS'
    fortifyProjectName: 'MyApp'
    projectVersionId: 'MyVersion'

But all 3rd party checks (incl. Fortify Scan) are skipped.
Stage "Third-party Checks" skipped due to when conditional
[Pipeline] parallel
[Pipeline] { (Branch: Checkmarx Scan)
[Pipeline] { (Branch: WhiteSource Scan)
[Pipeline] { (Branch: SourceClear Scan)
[Pipeline] { (Branch: Fortify Scan)
[Pipeline] { (Branch: Additional Tools)
[Pipeline] { (Branch: SonarQube Scan)
[Pipeline] stage
[Pipeline] { (Checkmarx Scan)
[Pipeline] stage
[Pipeline] { (WhiteSource Scan)
[Pipeline] stage
[Pipeline] { (SourceClear Scan)
[Pipeline] stage
[Pipeline] { (Fortify Scan)
[Pipeline] stage
[Pipeline] { (Additional Tools)
[Pipeline] stage
[Pipeline] { (SonarQube Scan)
Stage "Checkmarx Scan" skipped due to when conditional
[Pipeline] }
Stage "WhiteSource Scan" skipped due to when conditional
[Pipeline] }
Stage "SourceClear Scan" skipped due to when conditional
[Pipeline] }
Stage "Fortify Scan" skipped due to when conditional
[Pipeline] }
Stage "Additional Tools" skipped due to when conditional
[Pipeline] }
Stage "SonarQube Scan" skipped due to when conditional

This is probably due to the "productive branch". The idea is to not run everything for pull request pipelines, but limit some stages (one of those is fortify) to the productive branch (by default that is master, but you can configure it to be something else, cf https://github.com/SAP/cloud-s4-sdk-pipeline/blob/master/configuration.md#general-configuration)

Think you're right. I will test.
Maybe it makes sense to logout why a stage is not running.

The builds on DockerHub are still in progress, might take around 30 minutes I assume, see https://hub.docker.com/repository/docker/ppiper/node-browsers for that

Maybe it makes sense to logout why a stage is not running.

I agree that the usability is not ideal in that case. Thanks for your feedback. I'm not sure how we could improve this. Adding individual log messages would be quite some maintenance effort, and also clutter the log output even more.
Please let me know when this issue can be closed.

Step is triggered now.
What are the requirements to the fortify image ? must this image have a maven installation or do I have to configure a mavenExecute step ?
Error is:
10:19:15  + mvn -Dmaven.repo.local=s4hana_pipeline/maven_local_repo --settings settings.xml --file pom.xml --batch-mode -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn fortify:translate fortify:scan -Dfortify.sca.verbose=false -Dfortify.sca.source.version=1.8 -Dfortify.sca.buildId=sdk-pipeline-java-cf-0.0.1-2020-04-23T075101UTC+33bbfb51609371a6f53612f3ccee79f5df859348 -Dfortify.sca.64bit=true -Dfortify.sca.Xmx=3000M -Dfortify.sca.exclude=/resources//,/target//,/unit-tests//,/integration-tests// -DskipNgComponents=true
10:19:15<EMAIL_ADDRESS>1<EMAIL_ADDRESS>mvn: not found

Another question ... as your are using the maven plugin to scan the sources with fortify, the scan is limited to Java source code.
How can a scan over NodeJS and/or Javascript  be performed ?

Hello,
I'll try to explain, what the fortify docker image should contain.
The docker image includes a fortify client installation in general with all step you need to install the fortify client.
The fortify client has to be fit to your SSC version.
Requirement:

Client installation package
valid license file
maybe the certificate of the SSC server

Steps:

install the client in /opt/fortify
install the server certificate in a java truststore. (you'll find a jre in the fortify client directory)
perform a ./fortifyupdate --acceptKey to update the role template

I hope it will help you.
Best Falk

This is exactly what we have. But the execution is asking for a maven installation.

Could you send the Dockerfile content?

I added a maven installation to the dockerfile and running a test now.
If this is not successful I will send you the original docker file.
One noe we're running in kubernetes. As you invoke fortify via maven I would assume that ther emust be an installation in the image for it.

@FalkFiedler
Hello,
I have one question ...  as it seems the maven call uses it's own maven repo you create in the pipeline. I guess you are not taking care of installing the fortify maven plugin into it.
[ERROR] No plugin found for prefix 'fortify' in the current project and in the plugin groups [org.apache.maven.plugins, org.codehaus.mojo] available from the repositories [local (/workingdir/**s4hana_pipeline/maven_local_repo**), our-and-public-releases (https://af.mycompany.net/artifactory/our-and-public-releases)] -> [Help 1]
Need some information how to install that...
I would prefer to not use the execution over maven as it limits the scanning to java based projects.
What about other programming languages?
Preffered way would be to build a step in gpp / sdk pipeline which is delegating the invocaction to command line sourcecanalyzer invocations.
Step must support

update rulepack from ssc
clean
translate
scan
upload to ssc

Hi,
okay understand, you want to use the fortify maven plugins.
Our goals are:
"clean ${executeFortifyScanConfiguration.fortifyPrefix}:clean",
"compile ${executeFortifyScanConfiguration.fortifyPrefix}:translate",
"${executeFortifyScanConfiguration.fortifyPrefix}:scan"
The fortify client only contains the sources for the plugin.
You'll find the sources in the ${base_directory}/plugins/maven
In this directory you'll find the archive file maven-plugin-bin.tar.gz

unpack the archive
mvn install:install-file -Dfile=pom.xml -DpomFile=pom.xml
mvn install:install-file -Dfile=sca-maven-plugin/sca-maven-plugin-${Plugin_version_number}.jar -DpomFile=sca-maven-plugin/pom.xml

The fortify prefix has to be configured. We use the configuration in our case:
fortifyPrefix: 'com.fortify.sca.plugins.maven:sca-maven-plugin'
I suggest to get the current roles at the end of the docker file
fortifyupdate --acceptKey -url ${SSC_URL}
fortifyupdate -url ${SSC_URL}

we have already installed the plugin.
# Install maven
RUN apt-get -y install maven
RUN echo $M2_HOME
ADD settings.xml /root/.m2/settings.xml

and
# Install fortify maven plugin
ADD maven-plugin-bin.tar.gz maven-plugin-bin
RUN apt-get install dos2unix
RUN dos2unix maven-plugin-bin/install.sh
RUN ["/bin/bash", "-c", "cd maven-plugin-bin && chmod +x install.sh && ./install.sh"]

But it is not finding the plugin at all. Assume that we install it in the wrong local repo...

Hello,
did you installed the fortify client?
The fortify maven plugin isn't available in distribution repositories.
Do you know a file with a filename similar to HPE_Security_Fortify_SCA_and_Apps_${Your SSC version}_linux_x64.run?

I find the plug in in the local user repository ~/.m2/repository/com/fortify/sca/plugins/maven/sca-maven-plugin

But the pipeline ist creating a new one per -D option
mvn -Dmaven.repo.local=s4hana_pipeline/maven_local_repo --settings settings.xml
/workingdir/**s4hana_pipeline/maven_local_repo

Yes, you are right. All maven commands in the pipeline are executed with the option -Dmaven.repo.local=s4hana_pipeline/maven_local_repo. The reason for that is that we also stash and unstash the m2 directly to share it between stages.
Therefore all dependencies in the ~/.m2/ directory will be ignored.
You could either try to install the plugins into this directory (e.g. copy the m2 directory to this location in the Docker image or setting the m2 directory -D option for the installation with the environment variable MAVEN_OPTS). However, unstashing the m2 directory from previous stages would merge this directory and could have side-effects.
The alternative approach would be to have the plugins a nexus repostiory configures in your settings.yml.
To get the prefix mapping working you should configure the plugin either in your pom.xml or add the corresponding plugin group into your settings.xml which you configure a project settings file for the step mavenExecute.
I would also like to mention that, as you might already know, project Piper is working on a new implementation of the fortify step. As far as I know this new step might not use the maven plugins anymore and therefore might also remove some of the restrictions you would have with the current approach. We would also adopt this new step in the Cloud SDK pipeline. Therefore, I would recommend to limit the invests into this step which will be deprecated soon.

thanks for the information. I heared about it, but wanted to have a look in the current implementation.
As you might know there is a Jenkins Plugin from Microfocus available. Fyi this has some issues as well. Not Sure if they are already fixed.

The new step was release and is now used in the cloud sdk pipeline. therefore I will close this ticket.


Allow operator IN list for resource element

Category
Cedar language or syntax features/changes
Describe the feature you'd like to request
Is there any reason why the following syntax is not permitted for resource element:
resource in [PhotoApp::Photo::"vacationPhoto.jpg",  PhotoApp::Photo::"birthdayPhoto.jpg"] ?
Whereas the same functionnality functionnality is ok for the action element:
//matches any of the listed actions
action in [Action::"listAlbums", Action::"listPhotos", Action::"view"]
Describe alternatives you've considered
If I want to have the same functionnality for the resource element, I have considered the following approaches:
1/ First approach
permit (
principal == PhotoApp::User::"alice",
action == PhotoApp::Action::"viewPhoto",
resource
) when {
resource in [PhotoApp::Photo::"vacationPhoto.jpg",PhotoApp::Photo::"birthdayPhoto.jpg"]
};
It looks weird to have no defined constraints on resource in scope, whereas there is a constraint in the condition clause related to the resource ID itself
2/ Second approach
Have 2 permissions:
permit (
principal == PhotoApp::User::"alice",
action == PhotoApp::Action::"viewPhoto",
resource == PhotoApp::Photo::"vacationPhoto.jpg"
);
permit (
principal == PhotoApp::User::"alice",
action == PhotoApp::Action::"viewPhoto",
resource == PhotoApp::Photo::"birthdayPhoto.jpg"
);
But that would lead to a huge number of permissions
Additional context
No response
Is this something that you'd be interested in working on?

[ ] 👋 I may be able to implement this feature request
[ ] ⚠ This feature might incur a breaking change

As part of the design of the Cedar language we decided that that role-based policies should ideally be associated with a single principal or resource (or principal group or resource group), so we softly enforce this by limiting what you can write in the policy scope while leaving the condition unconstrained to accommodate any other uses cases.
Two approaches you may not have considered:

policy templates.

You can write one policy template
permit (
  principal == PhotoApp::User::"alice",
  action == PhotoApp::Action::"viewPhoto",
  resource == ?resource
);

Which you link twice, once for each resource  you want the policy to apply to. This is essentially your second approach, but without duplicating the rest of the policy.

resource groups.

permit (
  principal == PhotoApp::User::"alice",
  action == PhotoApp::Action::"viewPhoto",
  resource in PhotoApp::Album::"shared-with-alice"
);

where the two resources you want to grant access to are a member of that group.