[SPARK-31272][SQL] Support DB2 Kerberos login in JDBC connector

What changes were proposed in this pull request?
When loading DataFrames from JDBC datasource with Kerberos authentication, remote executors (yarn-client/cluster etc. modes) fail to establish a connection due to lack of Kerberos ticket or ability to generate it.
This is a real issue when trying to ingest data from kerberized data sources (SQL Server, Oracle) in enterprise environment where exposing simple authentication access is not an option due to IT policy issues.
In this PR I've added DB2 support (other supported databases will come in later PRs).
What this PR contains:

Added DB2ConnectionProvider`
Added DB2ConnectionProviderSuite
Added DB2KrbIntegrationSuite docker integration test
Changed DB2 JDBC driver to use the latest (test scope only)
Changed test table data type to a type which is supported by all the databases
Removed ignore from DockerJDBCIntegrationSuite since the jenkins integration is not true at the moment
Removed double connection creation on test side
Increased connection timeout because DB2 docker takes quite a time to start

Why are the changes needed?
Missing JDBC kerberos support.
Does this PR introduce any user-facing change?
Yes, now user is able to connect to DB2 using kerberos.
How was this patch tested?

Additional + existing unit tests
Additional + existing integration tests
Test on cluster manually

Test build #121279 has started for PR 28215 at commit 59d34fa.

Can one of the admins verify this patch?

cc @HeartSaVioR

Test build #121279 has finished for PR 28215 at commit 59d34fa.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121279/
Test FAILed.

Looks unrelated.

Retest this please.

Test build #121287 has started for PR 28215 at commit 59d34fa.

Test build #121287 has finished for PR 28215 at commit 59d34fa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121287/
Test PASSed.

Re-executed the docker tests and passed.

Test build #121369 has started for PR 28215 at commit eb4878c.

Test build #121369 has finished for PR 28215 at commit eb4878c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121369/
Test FAILed.

retest this, please

Test build #121374 has started for PR 28215 at commit eb4878c.

Test build #121374 has finished for PR 28215 at commit eb4878c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121374/
Test PASSed.

cc @dongjoon-hyun @vanzin since you know this area well

Hi, @gaborgsomogyi . It would be great if you proceed Updating and enabling DB2IntegrationSuite in a separate PR first.

Test build #121526 has started for PR 28215 at commit e56c7bd.

Test build #121526 has finished for PR 28215 at commit e56c7bd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121526/
Test PASSed.

The check failure is valid, it's failing to compile in that environment.

Oh, nice catch. That compilation error seems to only come with JDK11.
That said, Spark PR builder may need to run at least two different environments; that's lucky we found the issue from Github Action, but it has been the source of false alarm hence not considered seriously in many case. E.g. we may not able to find this if we see failure in Linter build side on Github Action, as building Spark will be cancelled if Linter build fails.

Ah gosh! Fixed it.

Test build #121616 has started for PR 28215 at commit 1ecacec.

Test build #121616 has finished for PR 28215 at commit 1ecacec.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121616/
Test PASSed.

Merging to master.

Is there additional required for this? All the other test (including new DB2IntegreationSuite in #28325) passed, but this one fail.
...
DB2KrbIntegrationSuite:
org.apache.spark.sql.jdbc.DB2KrbIntegrationSuite *** ABORTED ***
  Exception encountered when invoking run on a nested suite - The code passed to eventually never returned normally. Attempted 128 times over 2.016876443166667 minutes. Last failure message: Login failure for<EMAIL_ADDRESS>from keytab /Users/dongjoon/PRS/SPARK-PR-28325/external/docker-integration-tests/target/tmp/spark-35512019-33ef-4cad-a54a-2bec69a3d4c2/db2.keytab. (DockerJDBCIntegrationSuite.scala:158)
...
Run completed in 4 minutes, 27 seconds.
Total number of tests run: 42
Suites: completed 8, aborted 1
Tests: succeeded 42, failed 0, canceled 0, ignored 0, pending 0
*** 1 SUITE ABORTED ***

Hi, Guys.
Currently, all the other suites including DB2IntegrationSuite passed . Only DB2KrbIntegrationSuite fails. This PR seems to need more verification. Although this was verified manually, the newly added integration test should be valid.
How can we handle this commit?

Is it fine to check it on Monday as a start?

Logs would be helpful from test side and from docker instance side to see what have gone wrong.

Sure, no rush for this because this is an integration test we will not revert this urgently. I just wondered this IT was tested or not when this PR was merged.

I'm re-executing them after each change because it's only couple of minutes. I've just tried it out and works on my machine. On Monday I'll put it into a loop but logs would speed up the process because such case we dont have to spend time on reproduction.

Could you share your work environment? Then, I can try that if possible. I tried master branch in my two machines, but there was no luck. I'm using Docker Desktop community (<IP_ADDRESS>).

Hi, @gatorsmile and @maropu .
Could you verify this new JDBC integration test suite additionally, please?
I'm asking your help since it might be a dev environment issue and I failed so far.

@dongjoon-hyun here is my environment (just checked the latest master and still works):

MacBook Pro
macOS Catalina 10.15.3 (19D76)
openjdk version "1.8.0_242"
Docker desktop community <IP_ADDRESS> (40693)

That said maybe my environment is just a lucky one and would be good to take a look at the logs from test side and from container side as well.

I run it by myself and I checked it passed on my MacOS (Sierra, Docker Desktop community v<IP_ADDRESS>).

Thank you, @maropu !

Thank you, @gaborgsomogyi . I'll try to lower my DockerDesktop version.


Create a delete request email for Public Cloud

To create an email template that alerts the admins of the Public Cloud team that a delete request has been submitted and needs their attention. This email is triggered by the delete request button currently being worked on by Oamar.

The email template is visible but it is not triggered after the delete request is submitted.
@Amritpal-Nijjar Was sending it a part of the task?

I added adminDelete emails to the route

Now admins receive the Deletion request email:


Popup theme resource binding not working

Describe the bug
I'm using the Popup control in my custom gallery control, and when I load/switch the theme to Dark in the application, the Popup background and foreground are not updated, but the colors are updated to the popup child via theme resource binding.
When I did this with template binding, this issue did not occur, and it only occurred in desktop applications, while UWP applications worked fine.
Steps to reproduce the bug

Download the attached simple reproducible WinUI Desktop application Popup_DarkTheme.zip
Click the Popup open button
Note Popup background and foreground not updated in Dark theme.

Expected behavior
Popup background and foreground should be updated properly in Dark theme.
Screenshots
Gallery View

Gallery Popup View

Version Info
Project-Reunion NuGet package version 0.5.0

NuGet package version:
[Microsoft.ProjectReunion 0.5.0]

Windows 10 version
Saw the problem?

Insider Build (xxxxx)

May 2020 Update (19041)

November 2019 Update (18363)

May 2019 Update (18362)

October 2018 Update (17763)

Hey @KanniyappanP, thanks for filing this, we'll track this issue in the microsoft-ui-xaml repo using the issue you created there: microsoft/microsoft-ui-xaml#4802


The EvacuationHandler doesn't have the correct locator_responders

The DEA startup codes: https://github.com/cloudfoundry/dea_ng/blob/master/lib/dea/bootstrap.rb#L351-L360 as below:
def start
setup_signal_handlers
download_buildpacks
snapshot.load
start_nats
setup_sweepers
start_nats_staging_request_handler
the EvacuationHandler is initialized in method setup_signal_handlers, but the @responders is initialized in method start_nats_staging_request_handler, so that the EvacuationHandler doesn't have the correct locator_responders, when DEA drain process starts, the EvacuationHandler could not stop the DEA heartbeat timer, that causes CloudController still send staging requests to this DEA which is in drain phase.

https://github.com/cloudfoundry/dea_ng/commit/c8d1f7a622a4b856c040f92fc721cfd02214f452


Create Sherlock example for LLM agents pipeline

Is this a new feature, an improvement, or a change to existing functionality?
New Feature
How would you describe the priority of this feature request
High
Please provide a clear description of problem this feature solves
As part of the Sherlock work, an example showing how to use Morpheus to execute multiple LLM agents which are able to use tools and execute multiple LLM queries in order to solve a complex task.
Describe your ideal solution
Purpose
The purpose of this example is to illustrate how a user could build a pipeline that will integrate an LLM agent into a Morpheus pipeline. This example builds on the previous two examples, #1298 and #1306, by adding the ability to execute multiple LLM queries as part of a chain and also to augment those queries with external information if needed.
Using LLM agents allows the LLMs to be able to execute "tools" that can help them solve their task. For example, if your question was "What is Leonardo DiCaprio's girlfriends age raised to the 0.43 power?", the LLM might not know who Leonardo DiCaprio's girlfriend is, but it will know it can find that out from a quick search. If you perform the internet search for the LLM and ask it the question again with the result from the internet search, it will have the information necessary to properly respond.
Scenario
This example will show two different implementations of an agent pipeline but the pipeline and components could be used in many scenarios with different requirements. At a high level, the following illustrates different customization points for this pipeline and the specific choices made for this example:

LLM Service

This pipeline could support any type of LLM service which is compatible with our LLMService interface.
Such services include OpenAI, NeMo, or even running locally using llama-cpp-python
For this example, we will focus on using OpenAI as the LLM service. This service has been chosen to be different from the previous example which uses NeMo. Additionally, the models in OpenAI have been trained to give responses that work well in the ReAct agent architecture.

Agent type

There are many different styles of agents that are available today. Each one of these agent types alters the pattern for asking the LLM what the next task is, running that operation, and then asking the LLM what to do next. That repeats until the LLM is satisfied with the output.
For this example, we will be using the ReAct agent type. This type of agent is the most popular and has proven to work well and be reliable.

Agent tools

Depending on the problem trying to be solved, there are many different tools that can be made available to the LLM agent such as an internet search, VDB retrievers, calculators, Wikipedia, etc.
For this example, we will be using the internet search tool and an llm-math tool. This allows the LLM agent to execute arbitrary Google searches and to perform math equations if necessary

LLM Library

Most of the popular LLM libraries, Langchain, llama-index, and Haystack, have the ability to run LLM agents to execute these higher level tasks.
For this example, we will be using the Langchain library to provide all of our LLM agents. This allows us to run agents directly in a Morpheus pipeline reducing the overhead of migrating an existing system to Morpheus, and also eliminates the need to replicate the work already done by these libraries.

Implementation
This example will be composed of two different click commands.
Simple Morpheus agent pipeline
The simple Morpheus pipeline is built using the following components:

An InMemorySourceStage to hold the LLM queries in a DataFrame
A DeserializationStage to convert the MessageMeta objects into ControlMessages needed by the LLMEngine

New functionality was added to the DeserializeStage to support ControlMessages and add a default task to each message.

A LLMEngineStage then wraps the core LLMEngine functionality

An ExtracterNode pulls the questions out of the DataFrame
A LangChainAgentNode runs the Langchain agent executor for all provided input. This node will utilize the agents arun interface to run the agents asynchronously.
Finally, the responses are put back into the ControlMessage using a SimpleTaskHandler

The pipeline concludes with an InMemorySink stage to store the results.

Kafka Morpheus pipeline
The Kafka Morpheus agent pipeline is functionally similar to the simple agent pipeline, however it uses a Kafka message bus to send messages to the pipeline instead of an in memory source. The reason we are using a Kafka message bus is to show the capabilities of the async LLMEngine functions running inside of a Morpheus pipeline. Normally, when running Langchain agents using the arun interface, they will all run simultaneously. But the Langchain system cannot handle new requests that come in while the current batch is running. In Morpheus, when new messages are sent to the pipeline while existing ones are processing, they will immediately begin processing increasing the parallelism. This helps show why running these LLM agents in a pipeline is advantageous over a normal python library.
The major differences between the simple Morpheus agent pipeline and the Kafka Morpheus agent pipeline are:

The source for the pipeline is a KafkaSourceStage to pull messages off of the bus as they are created
Message responses are either printed directly to the console or output to another Kakfa topic.

Completion Criteria
The following items need to be satisfied to consider this issue complete:

[x] A README.md containing information on the following:

[x] Background information about the problem at hand
[x] Information about the specific implementation and reasoning behind design decisions (use content from this issue)
[x] Step-by-step instructions for running the following:

[x] How to run the Simple Morpheus agent pipeline

[x] Including Information on how to get an OpenAI API key and what environment variable to set.
[x] Including information on how to get a Serpapi.com API key and what environment variable to set
[x] The command to run the Morpheus pipeline

[x] How to run the Kafka Morpheus agent pipeline

[x] Including instructions on how to set up the Kafka cluster for sending messages
[x] Including Information on how to get an OpenAI API key and what environment variable to set.
[x] Including information on how to get a Serpapi.com API key and what environment variable to set
[x] Including instructions on how to view the output Kafka topic where responses will be placed
[x] The command to run the Morpheus pipeline

[x] The README.md should be linked in the developer docs

[x] A functioning Simple Morpheus agent pipeline command which satisfies the following:

[x] Should run without error using all default arguments
[x] Correctly run the LLM agents using the OpenAI chat model
[x] Provide information about the success or failure of the pipeline. Including the number of queries run, throughput and total runtime.

[x] A functioning Kafka Morpheus agent pipeline command which satisfies the following:

[x] Should run without error using all default arguments
[x] Correctly run the LLM agents using the OpenAI chat model
[x] Output the responses to the console or to a separate output Kafka topic
[x] Provide information about the success or failure of the pipeline. Including the number of queries run, throughput and total runtime.

[x] Tests should be added which include the following

[x] Test successfully running the Simple Morpheus agent pipeline
[x] Test successfully running the Kafka Morpheus agent pipeline

Dependent Issues
The following issues should be resolved before this can be completed:
- [ ] #1178
- [ ] #1179
- [ ] #1277
- [ ] #1278
- [ ] #1299
- [ ] #1302
- [ ] #1303

Additional context
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct
[X] I have searched the open feature requests and have found no duplicates for this feature request

Closing as it's complete - I think the auto-close PR linkage stuff only works when it's going into the default branch.


Update trace4cats-kernel, ... to 0.12.0-RC1+191-347e4fc7

Updates

io.janstenpickle:trace4cats-kernel
io.janstenpickle:trace4cats-testkit

from 0.12.0-RC1+156-7ea07b63 to 0.12.0-RC1+191-347e4fc7.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "io.janstenpickle" } ]

labels: library-update, semver-build-metadata

Superseded by #7.


Distinguish between web and worker in EventsMetadata.source

Right now, the EventsMetadata object is hardcoded to identify the source as "openedx/{service}/web." In order to be compliant with https://open-edx-proposals.readthedocs.io/en/latest/architectural-decisions/oep-0041-arch-async-server-event-messaging.html#id3, we should be distinguishing between web and workers. It's not immediately obvious to me where we can get this information. The only setting I can find that distinguishes web from worker is NEW_RELIC_APP_NAME, which seems very fragile.

Even for sync messages, the "/web" suffix would be inaccurate for any signals sent in a worker. This does not just affect async (event bus) messages.
At least for edx.org configs, there isn't an easy way to have different settings between web and workers, so introducing a setting would not help.
One solution noted in https://github.com/openedx/openedx-events/issues/77#issuecomment-1328294890 is to just drop the "/web" part of the source. We also have a source host. Do we want to update OEP-41 as well?

FYI: @ormsbee @mariajgrimaldi


[SPARK-18946][ML] sliceAggregate which is a new aggregate operator for high-dimensional data

In many machine learning cases, driver has to aggregate high-dimensional vectors/arrays from executors.
TreeAggregate is good solution for aggregating vectors to driver, and you can increase depth of tree when data is large.
However, treeAggregate would still failed, when the parition number of RDD and the dimension of vector grows up.
We propose a new operator of RDD, named sliceAggregate, which split the vector into n slices and each slice is assigned a key(from 0 to n-1). The RDD[key, slice] will be transform to RDD[slice] by using reduceByKey operator.
Finally driver will collect and compose the n slices to obtain result.

I run an experiment which calculate the statistic values of features.
The number of samples is 1000. The feature dimension ranges from 10k to 20m, the comparition of time cost between treeAggregate and sliceAggregate is shown as follows. When feature dimension reach 20 million, treeAggregate was failed.
The table of time cost(ms) between sliceAggregate and treeAggregate.

feature dimension
sliceAggregate
treeAggregate

10K
617
607

100K
1470
967

1M
4019
4731

2.5M
7679
13348

5M
14722
22858

7.5M
20821
36472

10M
28526
50184

20M
47014

The code relate to this experiment is here .
JIRA Issue: https://issues.apache.org/jira/browse/SPARK-18946

Can one of the admins verify this patch?

Just to be clear - this is essentially just splitting an array up into smaller chunks so that overall communication is more efficient? It would be good to look at why Spark is not doing a good job with one big array. Is the bottleneck really the executor communication (shuffle part)? Or is it collecting the big array back at the end of tree aggregation (ie this patch sort of allows more concurrency in the collect operation)?
cc @dbtsai @sethah @yanboliang  who were looking at linear model scalability recently.

Hi @ZunwenYou Do you know what's the reason that treeAggregate failed when  feature dimension reach 20 million?
I think this potentially can help with the 2G disk shuffle spill limit. (to be verified).
Also we should evaluate the memory consumption due to the slice and copy.

Hi, @MLnick
You are right, sliceAggregate splits an array into smaller chunks before shuffle.
It has three advantage
Firstly, the shuffle data is less than treeAggregate during the whole transformation operation.
Secondly, as your description, it allows more concurrency, not only during the collect operation of driver, but also in the process of run seqOp and combOp.

Hi, @MLnick
You are right, sliceAggregate splits an array into smaller chunks before shuffle.
It has three advantage
Firstly, the shuffle data is less than treeAggregate during the whole transformation operation.
Secondly, as your description, it allows more concurrency, not only during the collect operation of driver, but also in the process of run seqOp and combOp.
Thirdly, as I observed, when an record is larger than 1G Bit(an array of 100 million dimension), shuffle among executors becomes less efficiency. At the same time, the rest of executos is waiting. I am not clear the reason for this.

Hi, @hhbyyh
In our experiment, the class MultivariateOnlineSummarizer contains 8 arrays, if the dimension reaches 20 million, the memory of MultivariateOnlineSummarizer is 1280M(8Bit* 20M * 8).
The experiment configuration as follows:
spark.driver.maxResultSize 6g
spark.kryoserializer.buffer.max 2047m
driver-memory 20g
num-executors 100
executor-cores 2
executor-memory 15g
RDD and aggregate parameter:
RDD partition number 300
treeAggregate depth 5
As the description of configuration, treeAggregate will run into four stages, each stage task number is 300, 75, 18, 4.
At the last stage of treeAggrate, tasks will be killed, because executors throw exception java.lang.OutOfMemoryError: Requested array size exceeds VM limit.
I set treeAggregate depth=7, executor-memory=30g, the last stage still failed.

Is the speedup coming mostly from the MultivariateOnlineSummarizer stage?
See https://issues.apache.org/jira/browse/SPARK-19634 which is for porting this operation to use DataFrame UDAF and for computing only the required metrics (instead of forcing computing all as is done currently). I wonder how that will compare?

Hi, @MLnick
Firstly, sliceAggregate is a common aggregate for array-like data. Besides MultivariateOnlineSummarizer  case, it can be used in many large machine learning cases. I chose MultivariateOnlineSummarizer to do our experiment, just because it is really a bottleneck of LogisticRegression in ml package.
This is a good improvement for MultivariateOnlineSummarizer,  but I do not think it's a good idea to compare these two improvement. In my opinion, it is reasonable to compare sliceAggregate to treeAggregate.

@ZunwenYou yes I understand that the sliceAggregate is different from SPARK-19634 and more comparable to treeAggregate. But I'm not sure, if we plan to port the vector summary to use DataFrame based UDAF, whether we can incorporate the benefit of sliceAggregate.
So my point would probably be to try to see how much benefit accrues from (a) using UDAF mechanism and (b) not computing unnecessary things. Then we can compare to the benefit here and decide.

I'm not totally certain there will be some huge benefit with porting vector summary to UDAF framework. But there are API-level benefits to doing so. Perhaps there is a way to incorporate the sliceAggregate idea into the summarizer or into catalyst operations that work with arrays...

cc @yanboliang - it seems actually similar in effect to the VL-BFGS work with RDD-based coefficients?

ping @yanboliang ,  please has a look at this improvement.

@MLnick It looks like VF-LBFGS has a different scenario. In VF algos, the vectors will be too large to store in driver memory, so we slice the vectors  into different machines (stored by `RDD[Vector], and the use partitionID as slice key).
and , in VF-LBFGS, there're only very few large vectors(usually 4-10 vectors) need to aggregate together. so, what this PR do looks different with VF-LBFGS.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?


[BEAM-5758] add fanout and reiteration to LoadTests

In GBK and Combine load tests there were missing both options: fanout and reiteration. Both are parametrised and can be passed by pipeline options: fanout and iterations

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Choose reviewer(s) and mention them in a comment (R: @username).
[x] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

Run Python Load Tests Smoke

R: @lgajowy could you take a look at it? cc: @pabloem

Run Python Load Tests Smoke

I waited for tests to pass, @lgajowy can you take a look if now everything is correct?

LGTM


[Task]: Fix switching wallets when on the proposals details page

Task description
When switching wallets on the proposals details page the governance dashboard correctly resets but it stays in an infinite loading state.
Requirements

Find and fix the issue so that the governance dashboard correctly loads

Creation checklist

[ ] I have assigned this task to the correct people
[X] I have added the most appropriate labels
[X] I have linked the correct milestone and/or project

May be related to this issue: https://discord.com/channels/397872799483428865/988073611287994418/1072062437597126718

Closed, since it doesn't occur on dev branch anymore.


flight_data_to_arrow_batch does not support RecordBatches with no columns

Describe the bug
Attempting to use flight_data_to_arrow_batch on flight data that does not have any columns returns an InvalidArgumentError("must either specify a row count or at least one column").
To reproduce
Call flight_data_to_arrow_batch on a FlightData and Schema that does not have any columns.
Expected behavior
Calling flight_data_to_arrow_batch should use the row information in the RecordBatch header and return a valid record batch instead of returning an error.
Additional context
Support for RecordBatches with no columns but with a row count was already added in #1552. flight_data_to_arrow_batch already has the row count information provided as an argument. It gets read there in message.header_as_record_batch(), but then the subsequent reader::read_record_batch call seems to discard that info.

Thanks for the report, I'll bash this one out before the next release unless someone else picks it up 😀


chore: [Hub Bot] Refresh metadata 2024-06-10

Updates Plugin Definitions

@edgarrmondragon this was starting to get suspicious that its only making a few updates a week. I noticed that the extractors github action matrix is still marked as pending in the most recent completed job. I dont know what the cause is but its possible we went over the max 256 value for a matrix so its hanging 🤔  A job matrix can generate a maximum of 256 jobs per workflow run. https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration.

@edgarrmondragon this was starting to get suspicious that its only making a few updates a week. I noticed that the extractors github action matrix is still marked as pending in the most recent completed job. I dont know what the cause is but its possible we went over the max 256 value for a matrix so its hanging 🤔 A job matrix can generate a maximum of 256 jobs per workflow run. https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration.

I see. I'm pretty that's the cause. Might be worth giving a quick try at migrating the fan-out logic from GHA to Python.


Serverless: Installing requirements from /home/playground/serverless/.serverless/requirements/requirements.txt ...
Invalid requirement: '../pythonReqExample'
It looks like a path. Does it exist ?

  Error --------------------------------------------------

  null

     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Stack Trace --------------------------------------------

Error: null
    at installRequirements (/home/playground/serverless/node_modules/serverless-python-requirements/lib/pip.js:259:11)
    at installRequirementsIfNeeded (/home/playground/serverless/node_modules/serverless-python-requirements/lib/pip.js:445:3)
    at ServerlessPythonRequirements.installAllRequirements (/home/playground/serverless/node_modules/serverless-python-requirements/lib/pip.js:524:29)
From previous event:
    at PluginManager.invoke (/home/.nvm/versions/node/v9.5.0/lib/node_modules/serverless/lib/classes/PluginManager.js:390:22)
    at PluginManager.run (/home/.nvm/versions/node/v9.5.0/lib/node_modules/serverless/lib/classes/PluginManager.js:421:17)
    at variables.populateService.then.then (/home/.nvm/versions/node/v9.5.0/lib/node_modules/serverless/lib/Serverless.js:157:33)
    at runCallback (timers.js:756:18)
    at tryOnImmediate (timers.js:717:5)
    at processImmediate [as _immediateCallback] (timers.js:697:5)
From previous event:
    at Serverless.run (/home/.nvm/versions/node/v9.5.0/lib/node_modules/serverless/lib/Serverless.js:144:8)
    at serverless.init.then (/home/.nvm/versions/node/v9.5.0/lib/node_modules/serverless/bin/serverless:43:50)
    at <anonymous>

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information -----------------------------
     OS:                     linux
     Node Version:           9.5.0
     Serverless Version:     1.29.2

Let me know if you need any more info.
Thanks!

I've got you, fix incoming...

Thanks @AndrewFarley. I was just starting on this too. Removing targetFolder as the cwd in the spawn arguments seemed to do the trick for me. Haven't run the full test suite yet, so I'll just let you finish up you fix 😄

But I think the fix is only for non-docker. I don't think there's a reasonable way to make a path like that work with docker.

@dschep Want to have a brief conversation?  I already did a fix which is totally different... inside generate requirements I auto-prepended two more cd backwards.  And I was going to add a "error" if you do this with docker enabled, thoughts?
  // Fix any items with ../ in them...
  for (var i=filteredRequirements.length; i--;) {
    if (filteredRequirements[i].startsWith('../')) {
      filteredRequirements[i] = '../../' + filteredRequirements[i];
    }
  }

Ah. yeah, I think I prefer my fix. There's no reason to run with the target as the working directory AFAICT. Tests seem to be passing with my change. This is the change:
diff --git a/lib/pip.js b/lib/pip.js
index 044f6c7..39d52ab 100644
--- a/lib/pip.js
+++ b/lib/pip.js
@@ -227,7 +227,7 @@ function installRequirements(targetFolder, serverless, options) {
     const preparedPath = dockerPathForWin(options, targetFolder);
     cmdOptions.push(getStripCommand(options, preparedPath));
   }
-  let spawnArgs = { cwd: targetFolder, shell: true };
+  let spawnArgs = { shell: true };
   if (process.env.SLS_DEBUG) {
     spawnArgs.stdio = 'inherit';
   }

Yeah, I like your change a lot more.    :)

Cool, yeah I was thinking we could still incorporate your idea of throwing an error if you find a requirement starting with ../ or / and docker is enabled.

Hey @dschep @AndrewFarley
Any estimation on when this is going to be merged and published?

I can merge it anytime, I was waiting for @AndrewFarley to push a few of his changes, but they aren't necessary for it to work. For now you can also just install the plugin from #246
npm i github:unitedincome/serverless-python-requirements#rel-dep

I got pulled into work just merge that fix.  :)
On Mon, 17 Sep 2018 at 15:28, Daniel Schep<EMAIL_ADDRESS>wrote:

I can merge it anytime, I was waiting for @AndrewFarley
https://github.com/AndrewFarley to push a few of his changes, but they
aren't necessary for it to work. For now you can also just install the
plugin from #246
https://github.com/UnitedIncome/serverless-python-requirements/pull/246
npm i github:unitedincome/serverless-python-requirements#rel-dep
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/UnitedIncome/serverless-python-requirements/issues/245#issuecomment-422015990,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAcsk6SKY0DHCCo1_PFGMU-jrtlAz_SCks5ub6ONgaJpZM4Wq0IQ
.

released in v4.2.4


Secure Stop on Windows Edge browser

Hi,
There's something I want to check for sure.
https://docs.microsoft.com/en-us/playready/features/secure-stop-pk#secure-stop-2
According to this page, when implementing Secure Stop Service, we can only receive Secure Stop Data from SL3000 environment.
So, is it correct that Secure Stop service is not available to web player in Windows Edge browser?

edge can also use SL3000

The current EME specification has entirely cut support for the persistent-usage-record session type which is required for Secure Stop.
https://www.w3.org/TR/encrypted-media/#dom-mediakeysessiontype
As a result, because Chromium has also removed support for it, the entire Secure Stop feature is no longer supported in Edge.


Change get_firmware_version() function to a property

This makes the change referenced in issue #7

PyLint believes the property access in __init__() seem to have no effect because the result is ignored. I think perhaps it's used to force it to raise the exception if it failed to detect the PN532. For now I am going to add # noinspection PyStatementEffect but if there is a better way this should be handled let me know and I can take care of it.

If this gets merged there are some code snippets in this learn guide section that would need to get updated to use the new property instead of get_firmware_version() I've updated all of the examples in the repo but I'm not sure if that is going to change the snippets.

I wonder if just assigning the return to something would make it happy?
_ = self.firmware_version

and if it gripes about that, then:
ic, ver, rev, support = self.firmware_version

I will give that a try this evening.

Ah, thank you. That got it. I tested the change just in case and all is well.


Dimension in the model and dimension in the checkpoint are incompatible.

Hi,
I trained the model on my own data, where change in alphabet shown here.
alphabet='0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ/.>,()-')
I used create_dataset.py file to create mdb file.
File "demo.py", line 31, in <module>
    MORAN.load_state_dict(MORAN_state_dict_rename)
  File "/home/payal/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 519, in load_state_dict
    .format(name, own_state[name].size(), param.size()))
RuntimeError: While copying the parameter named ASRN.attentionL2R.char_embeddings, whose dimensions in the model are torch.Size([38, 256]) and whose dimensions in the checkpoint are torch.Size([71, 256]).

Can you help, please? Couldn't understand what is the issue?

Because the parameter in my demo.pth is used for 38 classes. You can skip the unmatched parameters and load other parameters. Please ref to the PyTorch offical docs.


allow custom element so that controls can be added

This is the v2 version of the code, and fixes https://github.com/jacobmischka/svelte-flatpickr/issues/9
I've not done a merge this complex before, so what I would propose is that we merge this and release 1.2.0, and then we can rebase master from it, and then do the v3 code.
Do you think that is the right way to approach this?

I'd much rather not rebase, ha. Let me see if I can come up with something real quick, one second.

Oh, you already made a v2 branch. I don't think we even need to rebase then, do we? Can't we just release 1.2.0 from v2 directly and not touch master?

Oh, also, sorry, but this is a yarn project, please get that package-lock outta here :sunglasses:

@jacobmischka Yeah, that's one way of managing it. We can do that instead if you prefer
Re yarn - it's ok, nobody is perfect :stuck_out_tongue_closed_eyes: . I'll remove the package lock ;)

Cool, done and published to npm

@jacobmischka I think you've forgotten to push the v1.x.x tag for this to github.
I need to make a fix for a race condition on initialisation, but I need the tag to do this :)

Ok ignore above comment - not entirely sure what I was seeing here :) It's 1.2.0 I was looking for.

Ha okay cool. Sorry for the delay on my end, I deployed that from my laptop and I actually haven't been on it to check for the tag in a few days.


[FLINK-16510] Ensure task manager process (and k8s pod) termination on fatal error

These are the missing changes from https://github.com/lyft/flink/commit/adcecf34fc2719e93d700de237ed9430697c793b
Let's verify they address the issue discussed in https://github.com/lyft/streamperfbench/pull/1065#issuecomment-651899129
We will need to work on an upstream solution regardless.

Thanks! I'll verify that these changes unblock the stuck pods we have observed through https://jira.lyft.net/browse/STRMCMP-916.

I created version 1.10-lyft20200701sysexit.

PR to build the streamingplatform image: https://github.com/lyft/streamingplatform/pull/1461

I've updated the builder version in https://github.com/lyft/streamperfbench/pull/1065 and will run the benchmark again.


change qnamaker to make sure there is a //sufix

js side fix for https://github.com/Microsoft/botbuilder-tools/issues/502

Pull Request Test Coverage Report for Build 1468

2 of 2 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.01%) to 70.164%

Totals

Change from base Build 1463:
0.01%

Covered Lines:
2292

Relevant Lines:
3116

💛  - Coveralls


Parse forest point cloud data

Description
Add parsing for point cloud data.

@mgcth I can help here as well. Can you provide some more info in the issue?

I've though about

generating height maps, e.g geotiff files as they should be easier to use in analysis. I think each point has a classification, e.g. ground, water, forest, building, unknown etc.
try cesium to serve the point cloud as 3D tiles we can visualise on the map


Treat comment as non-empty block body

If block is empty intentionally it can contain a comment. Here is how sonar-java plugin handles this: https://github.com/SonarSource/sonar-java/blob/321e3d4994d3134f29d58ab23b96d1e46d95f8f9/java-checks/src/main/java/org/sonar/java/checks/EmptyBlockCheck.java#L62

Seems reasonable to me.
Please update the test case and use the rule directly instead of copying the hasCommentInside code :)

Done.

Thanks!


Paste on click doesn't work when using keyboard shortcuts

No idea why

Seems to have something to do with the underlying app using the same keyboard shortcuts. It works fine in Gnome terminal, but not in Chrome (presumably because Chrome also has Ctrl + N). Also, using a longer timeout fixes it, but the timeout has to be reeeeeealy long (~200ms).

@SUPERCILEX there's one more issue that when you open that dialog with shortcut and then select entry with up down arrow and then when you click enter button present in numlock then nothing happens but when you press that big enter button then it works fine so please take a look at this also.

I've noticed it quite a lot of times that the paste on click didn't work when manually opening the menu and selecting the item from the list. Though I can not reliably reproduce it.


Add support for the new electron package name

A rename of electron-prebuilt to electron is in the works: https://github.com/electron-userland/electron-prebuilt/issues/160
Bothelectron-prebuilt and electron will be published in tandem through the end of 2016.
This PR updates electron-packager to add support for users of the electron package, while maintaining existing functionality for electron-prebuilt.
@kevinsawicki @jlord @zcbenz @iolsen

You could probably minimize the new electron fixture.

I have a fix in place and the test suite is passing for me locally but CI is failing.
I removed get-package-info because I was hitting an issue described right in its readme:

This is an error from resolve that means it was not able to find all the requested props. It would be nice to have a more descriptive error but there's no way to do that until this PR is merged.

I changed that code to just require the package.json file in the given directory, then use lodash.get to go spelunking for deep properties.
I will dig into the CI failures tomorrow.

Okay this is ready for review.
The previous push passed on 5 of 6 CI boxes; hopefully we'll see all green this time. (CI takes about an hour)

Happy to squash commits if that is still the convention here.

Happy to squash commits if that is still the convention here.

Yup, see https://github.com/electron-userland/electron-packager/blob/master/CONTRIBUTING.md#filing-pull-requests (last bullet point)
I will try to review later tonight. (Though I just noticed that somehow test coverage decreased with this PR...hopefully it's because CI isn't done yet.)

Hmmm. The tests no longer hit the "unable to find app name/electron version" code? Also, the test failure on Linux/Node 6 is troubling, but I'm not sure why it's happening (I reran the test run on Travis with the same result).

Thanks for all the feedback, @malept. I pushed up some changes to the docs and DRY'd things a bit.
I feel like the npm install stuff can live without being optimized for now.
As for the CI, the tests definitely seem flaky. I keep seeing 5/6 passed builds, but the failing environment is not consistently the same. Let's see what happens on this go around.

I have not been thrilled with this project's test runner (or tests, for that matter). Any suggestions/improvements would be greatly appreciated.

Travis results from three recent pushes (without meaningful code changes):

Squashed it all down into a single commit. Pretty sure this is ready to go. Let's defer on improving the test suite for another PR.

I'm :+1: except for that one new infer test, which I'd like to see fixed before merging.

Let's defer on improving the test suite for another PR.

Agreed.

except for that one new infer test

What's wrong with it?

It's mostly duplicate code from the other test (pretty much everything except 3 lines).

Simplified the test in https://github.com/electron-userland/electron-packager/pull/435/commits/61ba66c627216f240f5d2b9367f39359b60b2d7e (good call), and re-squashed commits.

@zeke Thanks! Please release 😄

Thanks for helping out, @malept. Landed in 7.5.0

Hmmm. The tests no longer hit the "unable to find app name/electron version" code? Also, the test failure on Linux/Node 6 is troubling, but I'm not sure why it's happening (I reran the test run on Travis with the same result).

The 7.5.0 update seems to have broken my setup as I keep getting [Error: must specify version] when building. I then tried installing<EMAIL_ADDRESS>and used same configuration with no problems.
The item that seems to fix this issue would be having to define the version of electron I'd like to package with. Yet as of the docs I shouldn't have to.

If omitted, it will use the version of the nearest local installation of electron or electron-prebuilt, defined in package.json in either dependencies or devDependencies.

Can confirm this happens with both node 5.10.0 and 6.2.0 on macOS. Here is the full error message...
Error: must specify version
    at download (/Users/gregholguin/Desktop/testing123/node_modules/electron-download/index.js:16:27)
    at Array.<anonymous> (/Users/gregholguin/Desktop/testing123/node_modules/electron-packager/index.js:121:7)
    at each (/Users/gregholguin/Desktop/testing123/node_modules/run-series/index.js:17:24)
    at next (/Users/gregholguin/Desktop/testing123/node_modules/rimraf/rimraf.js:74:7)
    at FSReqWrap.CB [as oncomplete] (/Users/gregholguin/Desktop/testing123/node_modules/rimraf/rimraf.js:110:9)

@SimulatedGREG I'm in the middle of releasing a fix (see #439 and #440).

@malept was just noticing. Thanks for the awesome work! Scary how switching from get-package-info to lodash.get created such a problem 😁.

It basically means we need help with a better testsuite.


Mute/UnMute Audio/Video option enabled for "Video only"/"Audio only" call respectively

Description:
When the user makes an appRTC call with "audio=false&video=true" (or "audio=true&video=false") parameters, he/she sees that the "Mute audio" / "Mute Video" icon can be toggled on/off respectively.
Steps to reproduce:

Append "audio=false&video=true" (or "audio=true&video=false") parameters to an appRTC call and click Join.
Hover over the "Mute audio" / "Mute Video" icons and toggle them on/off
Sample Test page - https://apprtc.appspot.com/r/862942399?audio=false&video=true

Actual results:
The user is able to mute/unmute Audio on a "Video only" call (and/or) mute/unmute Video on an "Audio only" call
Expected results:

Audio icon should be disabled/invisible for a Video only call
Video icon should be disabled/invisible for an Audio only call

Thanks for reporting this!

@samdutton is this something you want to take a look at?

Sure — I can take a look.


MINOR: Explain need to escape backslashes in rules

Documentation only PR as subject
This contribution is my original work and I license the work to the project under the project's open source license

can anyone review this please?


Documentation feedback: /docs/sources/reference/components/loki.process.md

It's unclear to me how to use the replace argument of a stage.replace block.
My initial assumption looking at it is that it works against the entire captured expression, like GNU sed. I.e. the replace argument would replace the entire string matched by expression. After several re-readings and a couple of tests, I'm beginning to suspect my assumptions were wrong.
The problem I'm trying to solve starting with is that NGINX error_log cannot be customized, so I intend to use stage.regex to convert it to logfmt and then add a stage.logfmt block to parse it.
For example, NGINX logs this when it starts up:
2024/06/15 19:39:46 [notice] 1#1: using the "epoll" event method
2024/06/15 19:39:46 [notice] 1#1: nginx/1.27.0
2024/06/15 19:39:46 [notice] 1#1: built by gcc 12.2.0 (Debian 12.2.0-14)
2024/06/15 19:39:46 [notice] 1#1: OS: Linux 5.15.0-112-generic
2024/06/15 19:39:46 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 65536:65536
2024/06/15 19:39:46 [notice] 1#1: start worker processes
2024/06/15 19:39:46 [notice] 1#1: start worker process 29

This is how I'm attempting to use this block to reformat the log:
loki.process "logfmt" {
  forward_to = [loki.write.local_loki.receiver]

  // NGINX: Parse error_log lines, which cannot be reformatted to logfmt (see https://stackoverflow.com/questions/4246756/is-it-possible-to-specify-custom-error-log-format-on-nginx)
  stage.match {
    selector = "{instance=~\"production/nginx-.+\"}"
    stage.replace {
      expression = "(?P<year>\\d{4})/(?P<month>\\d{2})/(?P<day>\\d{2}) (?P<hour>\\d{2}):(?P<minute>\\d{2}):(?P<second>\\d{2}) \\[(?P<level>[a-z]+)\\] (?P<processId>\\d+)#(?P<threadId>\\d+): (?P<message>.+)"
      replace = "time={{.Values.year}}-{{.Values.month}}-{{.Values.day}}T{{.Values.hour}}:{{.Values.minute}}:{{.Values.second}}z level={{.Values.level}} processId={{.Values.processId}} processId={{.Values.processId}} message={{.Values.message}}"
    }
  }
}

My initial question is how to use the named capture groups. I'm unfamiliar with Go so I can only guess at the template language syntax. Should I be using top-level values like {{.year}} or should I use the "Values" prefix {{.Values.year}}?
In either case, I get log lines like this:
time=<no value>-<no value>-<no value>T<no value>:<no value>:<no value>z level=<no value> processId=<no value> processId=<no value> message=<no value>/time=<no value>-<no value>-<no value>T<no value>:<no value>:<no value>z level=<no value> processId=<no value> processId=<no value> message=<no value>/time=<no value>-<no value>-<no value>T<no value>:<no value>:<no value>z level=<no value> processId=<no value> processId=<no value> message=<no value> time=<no value>-<no value>-<no value>T<no value>:<no value>:<no value>z level=<no value> processId=<no value> processId=<no value> message=<no value>:time=<no value>-<no value>-<no value>T<no value>:<no value>:<no value>z level=<no value> processId=<no value> processId=<no value> message=<no value>:time=<no value>-<no value>-<no value>T<no value>:<no value>:<no value>z level=<no value> processId=<no value> processId=<no value> message=<no value> [time=<no value>-<no value>-<no value>T<no value>:<no value>:<no value>z level=<no value> processId=<no value> processId=<no value> message=<no value>] time=<no value>-<no value>-<no value>T<no value>:<no value>:<no value>z level=<no value> processId=<no value> processId=<no value> message=<no value>#time=<no value>-<no value>-<no value>T<no value>:<no value>:<no value>z level=<no value> processId=<no value> processId=<no value> message=<no value>: time=<no value>-<no value>-<no value>T<no value>:<no value>:<no value>z level=<no value> processId=<no value> processId=<no value> message=<no value>

Heyyyyy wait a minute...
After copying this text from Grafana to an editor with line wraps turned on, I finally see that the replace block seems to operate on each capture group individually. My template groups resolve to <no value> because .Value itself takes on the value of each named capture group! My expectation was incorrect.
Maybe I'm trying to shove square pegs into round holes here. Maybe I'm supposed to use stage.regex to parse these capture groups into labels. But doesn't Loki want me not to create too many labels? I don't know. I was hoping to at least get all my logs looking the same before I figured out what else I'm supposed to do with them.

Also: It would be useful to have a link to documentation for the functions supported in the replace argument.

The following list contains available functions with examples of more complex replace fields.
ToLower, ToUpper, Replace, Trim, TrimLeftTrimRight, TrimPrefix, TrimSuffix, TrimSpace, Hash, Sha2Hash, regexReplaceAll, regexReplaceAllLiteral

"{{ if eq .Value \"200\" }}{{ Replace .Value \"200\" \"HttpStatusOk\" -1 }}{{ else }}{{ .Value | ToUpper }}{{ end }}"
"*IP4*{{ .Value | Hash "salt" }}*"

Is this what they're referring to? https://coveooss.github.io/gotemplate/docs/functions_reference/sprig-regex/

I found the unholy abomination that accomplishes what I was looking to do:
loki.process "logfmt" {
  forward_to = [loki.write.local_loki.receiver]

  // NGINX: Parse error_log lines, which cannot be reformatted to logfmt (see https://stackoverflow.com/questions/4246756/is-it-possible-to-specify-custom-error-log-format-on-nginx)
  stage.match {
    selector = "{instance=~\"production/nginx-.+\"}"
    stage.replace {
      // Regex is processed with the RE2 library (https://github.com/google/re2/wiki/Syntax)
      // Must match the entire line with one capture group for the `replace` argument to rewrite it
      expression = "(^\\d{4}/\\d{2}/\\d{2} \\d{2}:\\d{2}:\\d{2} \\[[a-z]+\\] \\d+#\\d+: .+)"

      // Use `regexReplaceAll` to rewrite the log line in logfmt format
      // Docs (?): https://coveooss.github.io/gotemplate/docs/functions_reference/sprig-regex/
      replace = "{{ regexReplaceAll \"(\\\\d{4})/(\\\\d{2})/(\\\\d{2}) (\\\\d{2}):(\\\\d{2}):(\\\\d{2}) \\\\[([a-z]+)\\\\] (\\\\d+)#(\\\\d+): (.+)\" .Value \"time=${1}-${2}-${3}T${4}:${5}:${6}z level=${7} processId=${8} threadId=${9} message=${10}\" }}"
    }
  }
}

It has double-nested strings and the regular expression is repeated twice.
I also discovered that the $ replacements must be in curly braces when next to letters because otherwise "$3T4" gets interpreted as $4


Branch : quicklogic : Dump report file for the area utilization

uart.zip
Is there any option to dump the summary of the area utilization , as shown below:

Device Utilization for QLAL4S3/PU90

Resource                          Used    Avail   Utilization
IOs                               17      32        53.13%
LCs                               28      891       3.14%
DFFs                              47     891       5.27%
ClockCells                       2       5         40.00%
RAM8Ks                         0       8           0.00%
If not, supported can be provided ?

Yosys does not really have info about available resources. That kind of report should rather be generated in VPR.

Thank  you for the quick reply. What is the option to dump the console log ?

when you run the build the flow should write the logs from each stage into files. So in the pack.log file you can find:
Pb types usage...
  PB-SYN_VCC   : 1
  PB-SYN_GND   : 1
  outpad       : 4
  VCC          : 1
  INPUT        : 1
  bidir_buf    : 5
  OUTPUT       : 4
  inpad        : 1
  BIDIR        : 5
  LOGIC        : 38
  PB-LOGIC     : 38
  f_frag       : 1
  GND          : 1
  q_frag       : 24
  c_frag       : 38
  PB-SYN_IO    : 5
	EMPTY: # blocks: 0, average # input + clock pins used: 0, average # output pins used: 0
	PB-LOGIC: # blocks: 38, average # input + clock pins used: 16.8684, average # output pins used: 1.65789
	PB-ASSP: # blocks: 0, average # input + clock pins used: 0, average # output pins used: 0
	PB-SYN_IO: # blocks: 5, average # input + clock pins used: 2.8, average # output pins used: 0.2
	PB-SYN_VCC: # blocks: 1, average # input + clock pins used: 0, average # output pins used: 1
	PB-SYN_GND: # blocks: 1, average # input + clock pins used: 0, average # output pins used: 1
Absorbed logical nets 5 out of 71 nets, 66 nets not absorbed.
FPGA sized to 35 x 33 (ql-eos-s3)
Device Utilization: 0.04 (target 1.00)
	Block Utilization: 0.04 Type: PB-LOGIC
	Block Utilization: 0.16 Type: PB-SYN_IO
	Block Utilization: 1.00 Type: PB-SYN_VCC
	Block Utilization: 1.00 Type: PB-SYN_GND

The above example is for a simple 24 bit counter design. The Pb types usage ... counts usage of every block type used in the device. The Device Utilization section sums that up.


Bump cf-deployment to 12.29 based on CVEs

Is your feature request related to a problem? Please describe.
Cloud Foundry posted a security notification recommending bumping to cf-deployment 12.22. https://www.cloudfoundry.org/blog/mysql-security-updates-oct2019/ has the details pertaining to a dozen or so warnings about MySQL sploits tied to PXC (plus one Curl weakness).

New CVE posted today: https://www.cloudfoundry.org/blog/cve-2020-5399/
Credhub would also need to be bumped to 2.5.10 to ensure TLS is properly enabled for MySQL.

More CVEs that 12.29 will address:
https://www.cloudfoundry.org/blog/usn-4249-1/
https://www.cloudfoundry.org/blog/usn-4252-1/ which will also need cflinuxfs3 bumped to 0.158 though I see pipelines already with 0.164
https://www.cloudfoundry.org/blog/usn-4233-2/ (similar cflinuxfs3 bump to 0.159)
https://www.cloudfoundry.org/blog/usn-4256-1/ (similar cflinuxfs3 bump to 0.160)

12.29 contains a CAPI regression. This is address in 12.33: https://github.com/cloudfoundry/cf-deployment/releases/tag/v12.33.0

The release images for CF 12.33 are successfully built. Also, Initial changes for cf-deployment bump merged into master: https://github.com/SUSE/kubecf/pull/409
However, more changes are WIP: https://github.com/SUSE/kubecf/pull/413
CATs release bump to 12.33 also merged: https://github.com/SUSE/cf-acceptance-tests-release/pull/18

PR: https://github.com/cloudfoundry-incubator/kubecf/pull/413
Build + Deploy + Smokes on Diego and Eirini are passing.
CATs(on diego) are passing: https://concourse.suse.dev/teams/main/pipelines/kubecf/jobs/cf-acceptance-tests-diego/builds/4

closing it since pull request is merged.


Add support for server options (-s and -o)

We want to be able to pass a server ID or fully qualified hostname to the speedtest CLI command so that we can use a specific server to test against, rather than the default server that the CLI selects for us.
This change adds two new mutually exclusive environment variables which can be used to specify either:

SPEEDTEST_SERVER_ID - A server from the server list using its id
SPEEDTEST_HOSTNAME  - A server, from the server list, using its host's fully qualified domain name

Impact on the speedtest CLI command:

If SPEEDTEST_SERVER_ID is specified, the -s option will be added

If SPEEDTEST_HOSTNAME is specified, the -o option will be added

If both environment variables are specified, the container will error out and not run the speedtest CLI command

If neither of the new environment variables are not present, the CLI command will be called as usual with the default server

JSON=$(speedtest --accept-license --accept-gdpr -f json)

If you specify an invalid server ID/host name you will see an error:

{"type":"log","timestamp":"2024-06-17T03:39:08Z","message":"Configuration - No servers defined (NoServersException)","level":"error"}

To get the available ids, sponsors, and hostnames from speedtest run:

curl -s https://cli.speedtest.net/api/cli/config | jq -r '.servers[] | "id: \(.id), sponsor: \(.sponsor), host: \(.host)"'

and you will get a response like:
id: 32298, sponsor: SYNLINQ, host: speedtest1.synlinq.de:8080
id: 44081, sponsor: 23M GmbH, host: speedtest.23m.com:8080
id: 53257, sponsor: LWLcom GmbH, host: netservice01.fra02.lwlcom.net:8080
id: 40094, sponsor: PVDataNet, host: speedtestde.pvdatanet.com:8080
id: 55462, sponsor: Twerion.net | Minecraft Server, host: speedtest.twerion.net:8080
id: 37492, sponsor: Melbicom, host: speedtest-fra.melbicom.net:8080
id: 10010, sponsor: fdcservers.net, host: lg-fra.fdcservers.net:8080
id: 54504, sponsor: Deutsche Glasfaser, host: speed2.dg-sys.net:8080
id: 46569, sponsor: wirsNET, host: speed1.wirsnet.com:8080
id: 23712, sponsor: kko.me | avrx.de, host: speedtest.kko.me:8080

Remove the port number from the entry and run the container with the new environment variable, for example:

docker run -e SPEEDTEST_SERVER_ID=32298 robinmanuelthiel/speedtest:latest
docker run -e SPEEDTEST_HOSTNAME=speedtest1.synlinq.de robinmanuelthiel/speedtest:latest

Hi @robinmanuelthiel, I have tested the container with the following scenarios and they all behave as I expected.
Please do let me know if you would like any changes made, or indeed if we should carry out some more testing.

no server option environment variables specified (as-is behaviour)
SPEEDTEST_SERVER_ID environment variable specified
SPEEDTEST_HOSTNAME  environment variable specified
both SPEEDTEST_SERVER_ID and SPEEDTEST_HOSTNAME environment variables specified
invalid SPEEDTEST_SERVER_ID environment variable specified
invalid SPEEDTEST_HOSTNAME  environment variable specified

Snags attached below:

Looks lovely, thanks a lot!


Possible memory leak

Hi John,
We are currently investigating a memory leak, it can be caused by lua-vips (or libvips 8.7 RC1), another dependency, a leak in OpenResty or a combination of these factors.
To exclude things, I started checking lua-vips for possible memory leaks. I've found a number of things:

While checking for possible memory leaks, I used valgrind on our test environment from the openresty-valgrind RPM package provided by OpenResty. With this lua-script:
local vips = require "vips"

and this command:
valgrind --leak-check=full /usr/local/openresty-valgrind/luajit/bin/luajit-2.1.0-beta3 vips-init.lua &> output.txt

it generates this log: https://gist.github.com/kleisauke/57558977a2e31be0c809424078885196
Note: this is without any GLib / libvips suppression file. So it may be a false-positive.

In addition to valgrind we've made some stacktraces to potential memory leaks with memleax on the production server. See the memleax.txt attachment. We used memleax -e 400 to report all memory allocations  that haven't been freed after 400 seconds.

While attempting to fix this (see: https://github.com/kleisauke/lua-vips/commit/bb4971f3584362bcaed907bd5c61042ff2a55c7d), I found something odd:
Test image: https://images2.alphacoders.com/651/651450.jpg
pyvips:
python3.6 soak-test.py /home/651450.jpg
memory: high-water mark 80.83 MB

lua-vips master:
luajit-2.1.0-beta3 soak-test.lua -- /home/651450.jpg
memory: high-water mark 1.02 GB

This may be the cause of the GC collector of LuaJIT, but it seems odd that the high-water mark is much higher than that of pyvips / python. I thought this was due to caching, but the soak test from Python is also disabling that: https://github.com/jcupitt/pyvips/blob/master/examples/soak-test.py#L9

I will further debug this (in combination with OpenResty) in the next few days, but I thought I'd let you know in advance. Any help would be greatly appreciated.
memleax.txt

Hi Kleis,
I added vips.leak_set(true) to lua-vips and added a soak example:
https://github.com/jcupitt/lua-vips/blob/master/example/soak.lua
I can run like this:
$ luajit soak.lua ~/pics/k2.jpg 10
loop 	0
loop 	1
...
loop 	10
memory: high-water mark 13.40 MB

And like this:
$ luajit soak.lua ~/pics/k2.jpg 1000
loop 	0
loop 	1
loop 	2
...
loop 	999
loop 	1000
memory: high-water mark 13.42 MB

I watched the second case in top and memuse crept to ~150mb by loop 200, and then stabilised. So I think, in this case anyway, there's no leak.
It's higher than Python's 50mb, but luajit has a very different GC -- I think luajit 2.1 is generational mark-sweep, whereas Python has reference counting plus non-generational mark-sweep to break cycles. This means py can often release memory as soon as it is unused, but luajit has to wait for the next large GC cycle.
People usually say that GC roughly doubles memory consumption, so this seems in line with that.
Obviously there are lots of paths this soak tester is not testing :(

Oh, I just saw your improve-memory branch. I wonder why you see >1gb? git master lua-vips seems to run in less memory than that.

Sorry I keep forgetting things to add.
I have soak-tested lua-vips before and it's seemed OK (or the same as the other bindings), so I doubt if there are any problems directly in the code (plus it's rather simple).
I don't really understand how openresty manages resources or the life-cycle of a request, so I can easily imagine something strange with fork() messing up lua-vips's reference counts.
I'm sure you saw this old issue:
https://github.com/jcupitt/lua-vips/issues/19
it has some more memory benchmarks with openresty.

Indeed, I think this has something to do with GC. Explanation of >1gb:
It seems that I can't reproduce this with write_to_buffer:

Details k2.jpg
luajit-2.1.0-beta3 soak.lua /home/k2.jpg 1
loop    0
loop    1
memory: high-water mark 24.48 MB

luajit-2.1.0-beta3 soak.lua /home/k2.jpg 10
loop    0
loop    1
loop    2
loop    3
loop    4
loop    5
loop    6
loop    7
loop    8
loop    9
loop    10
memory: high-water mark 24.48 MB

luajit-2.1.0-beta3 soak.lua /home/k2.jpg 50
loop    0
loop    1
loop    2
loop    3
loop    4
loop    5
loop    6
loop    7
loop    8
loop    9
loop    10
...
loop    47
loop    48
loop    49
loop    50
memory: high-water mark 24.63 MB

luajit-2.1.0-beta3 soak.lua /home/k2.jpg 100
loop    0
loop    1
loop    2
loop    3
loop    4
loop    5
loop    6
loop    7
loop    8
loop    9
loop    10
...
loop    97
loop    98
loop    99
loop    100
memory: high-water mark 24.63 MB

luajit-2.1.0-beta3 soak.lua /home/k2.jpg 500
loop    0
loop    1
loop    2
loop    3
loop    4
loop    5
loop    6
loop    7
loop    8
loop    9
loop    10
...
loop    497
loop    498
loop    499
loop    500
memory: high-water mark 24.63 MB

luajit-2.1.0-beta3 soak.lua /home/k2.jpg 1000
loop    0
loop    1
loop    2
loop    3
loop    4
loop    5
loop    6
loop    7
loop    8
loop    9
loop    10
...
loop    997
loop    998
loop    999
loop    1000
memory: high-water mark 24.63 MB

(omitted 651450.jpg because I think that goes well too)
But as soon as I test with write_to_file:
diff --git a/example/soak.lua b/example/soak.lua
index 1111111..2222222 100644
--- a/example/soak.lua
+++ b/example/soak.lua
@@ -17,5 +17,5 @@

     local im = vips.Image.new_from_file(arg[1])
     im = im:embed(100, 100, 3000, 3000, { extend = "mirror" })
-    local buf = im:write_to_buffer(".jpg")
+    im:write_to_file("x.v")
 end

Details k2.jpg
luajit-2.1.0-beta3 soak.lua /home/k2.jpg 1
loop    0
loop    1
memory: high-water mark 41.42 MB

luajit-2.1.0-beta3 soak.lua /home/k2.jpg 10
loop    0
loop    1
loop    2
loop    3
loop    4
loop    5
loop    6
loop    7
loop    8
loop    9
loop    10
memory: high-water mark 161.99 MB

luajit-2.1.0-beta3 soak.lua /home/k2.jpg 50
loop    0
loop    1
loop    2
loop    3
loop    4
loop    5
loop    6
loop    7
loop    8
loop    9
loop    10
...
loop    47
loop    48
loop    49
loop    50
memory: high-water mark 265.18 MB

luajit-2.1.0-beta3 soak.lua /home/k2.jpg 100
loop    0
loop    1
loop    2
loop    3
loop    4
loop    5
loop    6
loop    7
loop    8
loop    9
loop    10
...
loop    97
loop    98
loop    99
loop    100
memory: high-water mark 265.26 MB

luajit-2.1.0-beta3 soak.lua /home/k2.jpg 500
loop    0
loop    1
loop    2
loop    3
loop    4
loop    5
loop    6
loop    7
loop    8
loop    9
loop    10
...
loop    497
loop    498
loop    499
loop    500
memory: high-water mark 265.26 MB

luajit-2.1.0-beta3 soak.lua /home/k2.jpg 1000
loop    0
loop    1
loop    2
loop    3
loop    4
loop    5
loop    6
loop    7
loop    8
loop    9
loop    10
...
loop    997
loop    998
loop    999
loop    1000
memory: high-water mark 265.41 MB

Notice when I reached the ±265 MB that it will not go any further.
With 651450.jpg it becomes more intense:

Details 651450.jpg
luajit-2.1.0-beta3 soak.lua /home/651450.jpg 1
loop    0
loop    1
memory: high-water mark 149.34 MB

luajit-2.1.0-beta3 soak.lua /home/651450.jpg 10
loop    0
loop    1
loop    2
loop    3
loop    4
loop    5
loop    6
loop    7
loop    8
loop    9
loop    10
memory: high-water mark 630.00 MB

luajit-2.1.0-beta3 soak.lua /home/651450.jpg 50
loop    0
loop    1
loop    2
loop    3
loop    4
loop    5
loop    6
loop    7
loop    8
loop    9
loop    10
...
loop    47
loop    48
loop    49
loop    50
memory: high-water mark 973.32 MB

luajit-2.1.0-beta3 soak.lua /home/651450.jpg 100
loop    0
loop    1
loop    2
loop    3
loop    4
loop    5
loop    6
loop    7
loop    8
loop    9
loop    10
...
loop    97
loop    98
loop    99
loop    100
memory: high-water mark 1.02 GB

luajit-2.1.0-beta3 soak.lua /home/651450.jpg 500
loop    0
loop    1
loop    2
loop    3
loop    4
loop    5
loop    6
loop    7
loop    8
loop    9
loop    10
...
loop    497
loop    498
loop    499
loop    500
memory: high-water mark 1.02 GB

luajit-2.1.0-beta3 soak.lua /home/651450.jpg 1000
loop    0
loop    1
loop    2
loop    3
loop    4
loop    5
loop    6
loop    7
loop    8
loop    9
loop    10
...
loop    997
loop    998
loop    999
loop    1000
memory: high-water mark 1.08 GB

Notice when I reached the ±1 GB that it will not go any further.
Where k2.jpg is: https://commons.wikimedia.org/wiki/File:Picture_of_K2.jpg and 651450.jpg is: https://images2.alphacoders.com/651/651450.jpg
I'll make a list of all libvips operations that we use, as we don't use write_to_file (we do use a lot of image operations + write_to_buffer).

Huh curious, I'm not sure why write_to_file should use more memory. I'll investigate.

I tried luajit soak again on lua-vips master:
jpg buffer output 250mb peak in top
jpg file output 500mb peak in top
vips file output 500mb peak in top

I added this to the end of the loop:
    im = nil
    collectgarbage()

And with vips output it runs in 250mb of memory.
So I think it's clearly the lua GC here. I doubt if you can insert many collectgarbage() calls without hurting performance.
Could you have something to trigger it if the server is idle for more than 100ms? Or perhaps once every 10 requests and no more than once per second?

Hi John,
I think that I've found the culprit, see:
local ffi = require "ffi"

local vips_lib
local glib_lib

if ffi.os == "Windows" then
    vips_lib = ffi.load("libvips-42.dll")
    glib_lib = ffi.load("libglib-2.0-0.dll")
else
    vips_lib = ffi.load("vips")
    glib_lib = vips_lib
end

ffi.cdef [[
    char* vips_filename_get_filename (const char* vips_filename);
    char* vips_filename_get_options (const char* vips_filename);

    void g_free(void* data);
]]

local vips_filename = 'document.pdf[page=1]'

local i = 1

while true do
    print("iteration ", i)

    local filename = vips_lib.vips_filename_get_filename(vips_filename)
    local options = vips_lib.vips_filename_get_options(vips_filename)

    local filename_str = ffi.string(filename)
    local options_str = ffi.string(options)

    print(filename_str)
    print(options_str)

    -- lua-vips isn't doing this:
    -- glib_lib.g_free(filename)
    -- glib_lib.g_free(options)

    i = i + 1
end

vips_filename_get_filename and vips_filename_get_options needs to be freed with g_free because the string is duplicated (with g_strdup), see:
https://github.com/jcupitt/libvips/blob/0b3565c04d7b3f491126433cd42edeb0618824b6/libvips/iofuncs/image.c#L1803
https://github.com/jcupitt/libvips/blob/0b3565c04d7b3f491126433cd42edeb0618824b6/libvips/iofuncs/image.c#L1827
Other libvips bindings may also need to be checked.

Oh yes! Well done!! I pushed a fix, what do you think?
soak probably isn't sensitive enough to spot this, but it seems to run OK.
I'll check the other bindings.

And pyvips (forgot to paste in the link to this issue)
https://github.com/jcupitt/pyvips/commit/52e218462af9261766f5aebf60b8a39ef0c3e2c2

OK, all done, I hope.

Thanks for the fix! This does solve something, but we still see high memory usage (GC seems to be the biggest problem). Can easily be produced with this:
Test image: https://t0.nl/undefined.jpg
lua-vips soak script (adapted for thumbnail processing):
local vips = require "vips"

vips.leak_set(true)
vips.cache_set_max(0)

if #arg ~= 2 then
    print("usage: luajit soak.lua image-file iterations")
    error()
end

local im

for i = 0, tonumber(arg[2]) do
    print("loop ", i)

    im = vips.Image.thumbnail(arg[1], 10, {
        height = 10000000,
        auto_rotate = false,
        linear = false,
        size = "down",
    })
    local buf = im:write_to_buffer(".jpg")
    im = nil

    --collectgarbage()
end

This time I've made a flame graph (with sample-bt-leaks) which can be found here: https://t0.nl/memleak.svg
The high-water mark memory from libvips reports: ±603.71 KB but with 200 iterations the RSS memory easily becomes ±1 GB. This is solved with collectgarbage(), but it feels a bit unnatural to do and indeed I doubt if you can insert many collectgarbage() calls without hurting performance (I still have to test that in production).
Could there be another bug that we have overlooked? The flame graph looks suspicious with those leaks(?) in vips_region_*.
Note that I didn't see this high memory usage in pyvips. Maybe I should wait for the New Garbage Collector in LuaJIT 3.0.

It seems that I've found a strange image again, when I first process it with vipsthumbnail:
vipsthumbnail undefined.jpg \
    --size "4032<" \
    -o %s_2.jpg[optimize_coding,strip] \
    --eprofile sRGB.icm

(which preserves the image size and move it to the sRGB colourspace, sRGB.icm can be found here)
Afterwards when I will process the normalized image (undefined_2.jpg) with the identical script above (so without collectgarbage()), the memory does not continue to rise. So I doubt if it's LuaJIT GC that creates this high memory usage. It looks like there is a leak somewhere in vips_icc_* or LittleCMS.

Your test image is interlaced, so the first time it's read the jpeg reader has to allocate a huge buffer to unpack it. The second time, it's a regular JPG, so memuse is low.
Python doesn't really have a GC -- it's reference-counted. This means it can free objects as soon as they go out of scope, so memuse will typically be much lower than LuaJIT. It has a GC as well, but it's just for breaking cycles and only runs occasionally.
Ruby has a generational GC, so it can GC very frequently without hurting performance. I think this is what LuaJIT is switching to.
Ruby used to have a simple mark-sweep GC, like LuaJIT. Back then, ruby-vips had this thing:
https://github.com/jcupitt/ruby-vips/blob/master/lib/vips/image.rb#L162
It triggered a GC after every 100 writes. The performance impact was small, and it did stop pathological mem growth.

I doubt if there can be a serious memory leak in libvips (though of course it's always possible!).
All large memory areas are linked to GObjects, and set_leak will find any object reference leaks. As long as you are not leaking references, memleaks can't be very significant.

Oh, and interlaced JPG images can't be processed with shrink-on-load either. They really are awful.

We will try to experiment with collectgarbage() on the production server, I'll let you know if this solves our problem.
In the meantime, I'll close this issue because this doesn't seem to be related to lua-vips / libvips. Thanks again for fixing the _get_filename() and _get_options() leaks, I've just fixed this thing in NetVips.
By the way,
When I tried to make further improvements to lua-vips (see this commit: https://github.com/kleisauke/lua-vips/commit/3cacde0268b5ac2bd6e07454b9aa3d48696eab1f), I noticed this line: https://github.com/jcupitt/lua-vips/blob/master/src/vips/Image.lua#L9
Hopefully that doesn't cause leaks, because usually this is done by providing 2 different variables. I tried to bypass that but had no luck, it seems that it's required for fixing the recursive requires. A neater way would be to extract Image.is_Image(x) into a utils file because that seems to be the only thing that needs to be shared accross the Image_methods and voperation files. (https://stackoverflow.com/a/13969886/1480019 is relevant here).

Yes, I remember that being a bit ugly. If you could cook up a PR with a better solution, I'd be very happy!

Oh dear, sorry about another rewrite :(
Perhaps you could make a tiny module that just does thumbnail and run that from nginx? If you see mem growth there too, perhaps the problem is something to do with openresty?

Don't worry, another rewrite is also good for learning a new programming language.
A tiny module that invokes thumbnail, is indeed the first step to start with. I'm going to experiment with this, I'll let you know if that doesn't cause memory leaks. Thanks!

I came across an interesting thread on the luajit email list:
https://www.freelists.org/post/luajit/LuaJIT-GC-Problem
tldr: luajit can keep compiled traces longer than you'd think, and they can fill memory. Moreover, with a trace, it must keep referenced objects. Calling jit.flush(); will dump them all.

@jcupitt: We've tried this solution for a few days in production together with gc.flush(); in the log_by_lua_file-directive, so that it will be executed at the end of every request, but it didn't seem to yield any difference. Additional suggestions are welcome!

Update:
On December 11, 2018 ImageMagick (v6.9.10-16) was temporarily unavailable on our production server (due to an incorrect update). Our users were not able to load / save (magickload / magicksave) images through ImageMagick (for e.g. .bmp, .ico and .jp2 images).
On the same moment we saw that our memory leak had disappeared:

It is therefore possible that the memory leak is located in magickload and/or magicksave or that it isn't located in libvips at all. I'll do some preliminary investigation with valgrind in February.

Wow, that's dramatic!

Gotcha! It seems that there is a leak in GetExceptionInfo (see valgrind output). Tested with this C code:
/* compile with:
 *      gcc -g -Wall test.c `pkg-config vips --cflags --libs` -o test
*/

#include <stdio.h>
#include <vips/vips.h>

int
main(int argc, char **argv) {
    VipsImage *in;

    if (VIPS_INIT(argv[0]))
        vips_error_exit(NULL);

    if (argc != 3)
        vips_error_exit("usage: %s infile outfile", argv[0]);

    if (!(in = vips_image_new_from_file(argv[1], NULL)))
        vips_error_exit(NULL);

    if (vips_image_write_to_file(in, argv[2], NULL))
        vips_error_exit(NULL);

    g_object_unref(in);

    vips_shutdown();

    return (0);
}

And this valgrind command:
valgrind --suppressions=libvips.supp \
         --leak-check=yes \
         --log-file=valgrind-out.txt \
         ./test favicon.ico[page=1,access=sequential] favicon.gif[format=gif]

(favicon.ico is located here)
For some reason the ExceptionInfo structure is not freed correctly. I've tried to fix this (see this commit), and the memory leak seems to be gone (see new valgrind output).
I'm not sure if this is the underlying cause of our memory leak. Also, AcquireExceptionInfo seems not available in GraphicsMagick so this may not be the most appropriate solution.

Oh nice! Looks like magick7load and magicksave already do this, so it's just magick2vips.
I'll add a configure test and a little wrapper to magick.c.

OK, there's a branch of 8.7 with a fix, could you test Kleis?
Also, good job on finding this!

Seems to work, tested with ImageMagick 6.9.10-23. Thanks for fixing this! I can't wait to test this on our production environment.


Incredibly Slow Network Performance

Why is the speed of the virtual AP that the app creates so slow??

Try to enable 802.11n with --ieee80211n, also make sure haveged is installed.

I have tried to enable 802.11n with --ieee80211n and  installed haveged  but cannot solve.


Add types for TypeScript

a lot of people are using TS nowadays,
it would be nice to have some type definitions for the library

v1.1.0


Incorrect Docs Link and subsequent GitHub Link

Describe the issue
No documentation is shown when forwarded to docs link, by clicking on docs button on:
https://klasa.js.org/#/
Then, when trying to go to the github page after clicking the docs link:
https://klasa.js.org/#/docs/klasa/stable/Getting Started/GettingStarted
I get a dead git hub url
https://github.com/dirigeants/klasa/tree/stable
Code or steps to reproduce

Go to https://klasa.js.org/#/
Click on Docs (first issue)
Click on Github Link (second issue)

Expected and actual behavior
Expected behavior:
Github link: https://github.com/dirigeants/klasa/tree/master
Docs link: maybe auto link it to v0.5.0 (stable and master show no documentation)
Further details

@klasa/core version:
node.js version:
Klasa version:
[ ] I have modified core files.
[ ] I have tested the issue on latest master. Commit hash:

They know the docs are broken for master and stable, they're currently working on rewriting the site from the ground up
To view @klasa/core docs: https://dirigeants.github.io/core/index.html
To view klasa master docs: https://dirigeants.github.io/klasa/index.html
To view klasa stable docs: https://klasa.js.org/#/docs/klasa/v0.5.0/Getting Started/GettingStarted

Ahh I apologize, just a new user to Klasa. I'll close out the issue. Thanks @KunoichiZ for the links and heads up


Update solution algorithm

This PR updates the SS and TPI solution algorithms to have r_p (not r) in the outer loop.  This is more natural because r_p is what is important for the household problem and trying to recover r_p from r takes a few steps, such as having to back out capital demand from the guess at output, solve for K_g and so on.
The changes in this PR should make for an easier transition to the multiple industry problem.

@rickecon Per our conversation, let me give a summary of what I've done here and where I'm spinning my wheels:
What I've done:

Replaced r with r_p in the outer loop of the SS and TPI solution algorithms

The advantage of this is that one needs r_p to solve the HH problem, but if you guess r, you need to find r_p, which requires knowing K, D, and K_g.

Confirmed that all tests in test_SS.py::test_run_SS pass

Where I'm stuck:

If my changes are correct, all tests in test_TPI.py::test_run_TPI_full_run should pass.
The following parameterizations pass: 'Baseline, balanced budget', 'Baseline', 'Reform'
The following parameterizations fail: 'Reform, baseline spending',  'Baseline, small open', 'Baseline, small open some periods', 'Baseline, delta_tau = 0', 'Baseline, Kg >0'
N.B. All other tests in test_TPI.py pass except for the test_run_TPI tests, which are expected to fail since the new algorithm will give different results after the two iterations of TPI in those tests (the new algorithm should return the same values once the model converges to eq'm).

I wish I saw a pattern in the test failures to pin point where the errors are, but it's not just a open economy thing or just a Kg > 0 thing...

@jdebacker. Brutal. I ran the full suite of tests on this PR and got the following 18 failures. I am working through the test_SS.py failures.
tests/test_SS.py ......F.FF.FFFF..............                                              [  6%]
tests/test_TPI.py .......FFFFFFFFFF                                      [ 10%]
tests/test_aggregates.py ................................                [ 18%]
tests/test_basic.py ....                                                 [ 19%]
tests/test_elliptical_u_est.py .......                                   [ 20%]
tests/test_execute.py .                                                  [ 21%]
tests/test_firm.py .......................................               [ 30%]
tests/test_fiscal.py ..................                                  [ 34%]
tests/test_household.py ........................................         [ 43%]
tests/test_output_plots.py ......................................        [ 52%]
tests/test_output_tables.py .............                                [ 55%]
tests/test_parameter_plots.py ...................................        [ 63%]
tests/test_parameter_tables.py .......                                   [ 65%]
tests/test_parameters.py .............                                   [ 68%]
tests/test_run_example.py .F                                             [ 69%]
tests/test_run_ogcore.py .                                               [ 69%]
tests/test_tax.py ...............................                        [ 76%]
tests/test_txfunc.py .........................                           [ 82%]
tests/test_user_inputs.py .........                                      [ 84%]
tests/test_utils.py ................................................................ [ 99%]
tests/test_wealth.py ..                                                  [100%]


Forward buttons

If the message contains the inline button, after forwarding it button is not forwarded, only text. It's possible to forward with buttons?
var (
	buyerBtn = tb.InlineButton{
		Unique: "buyer",
		Text:   "Buyer",
		Data:   "string",
	}

	sellerBtn = tb.InlineButton{
		Unique: "seller",
		Text:   "Seller",
		Data:   "string",
	}
        mainMenu = [][]tb.InlineButton{
		        []tb.InlineButton{buyerBtn},
		        []tb.InlineButton{sellerBtn},
        }
)

b.Send(m.Sender, msg_welcome_main_menu, &tb.ReplyMarkup{ InlineKeyboard: mainMenu})

It's impossible by design.

@x777 Unless it was you how asked this question on Stack Overflow - have a look at this: https://stackoverflow.com/questions/46756643/forward-inline-keyboard-along-message-in-telegram
It looks like you can only have the url button forwarded with the message.


Could not move temporary workspace ... to immutable location

Current Behavior
Reopening #27844, because it seems to not be fixed and has been broken since 8.5
When trying to build a freshly created project, I get the following error:
FAILURE: Build failed with an exception.

* What went wrong:
A build operation failed.
    Could not move temporary workspace (F:\local\applicationData\gradle\caches\8.9\transforms\e7fa29f876e87b2a15c2bc90d0781fdd-9307ed84-c3b0-4017-bce5-8de12a7efb13) to immutable location (F:\local\applicationData\gradle\caches\8.9\transforms\e7fa29f876e87b2a15c2bc90d0781fdd)
> Could not move temporary workspace (F:\local\applicationData\gradle\caches\8.9\transforms\e7fa29f876e87b2a15c2bc90d0781fdd-9307ed84-c3b0-4017-bce5-8de12a7efb13) to immutable location (F:\local\applicationData\gradle\caches\8.9\transforms\e7fa29f876e87b2a15c2bc90d0781fdd)

This was even after already having moved GRADLE_USER_HOME off of my C drive (which some suggested could fix the issue).
The issue only occurs in version 8.6 and newer. Version 8.5 doesn't have this issue.
Manually I actually am able to do the move, but only after killing the Gradle daemon process, since it's locking the files.
Workarounds:

Some people mention that uninstalling the virus scanner might help. This however is obviously not a preferred solution, since we don't want Gradle users to be vulnerable to viruses.
As mentioned, you can manually move/rename gradle cashes as the error is trying to do (So e7fa29f876e87b2a15c2bc90d0781fdd-9307ed84-c3b0-4017-bce5-8de12a7efb13 to e7fa29f876e87b2a15c2bc90d0781fdd. This works as a workaround, but is tedious (you need to do this at least 5-6 times before it works) and is only a temporary solution. Any Gradle version updates or cache wipes would require you to do this again.
Downgrade to version 8.5. This doesn't work for kotlin compose multiplatform projects though, since they rely on features from newer versions.

Expected Behavior
I expect a simple new gradle project to just build without issue.
Context (optional)
This behavior is not tied to a specific project. It was tested with a simple kotlin jvm application and a project generated by the compose multiplatform wizard.
Self-contained Reproducer Project
The problem is not caused by the project itself (considering it happens with any Gradle project).
Here is the gradle project that I used in the build scan: https://github.com/CC007/test-gradle
Gradle version
any version between 8.6 and 8.11.1
Build scan URL (optional)
https://scans.gradle.com/s/n3yylnhnckwuy
Your Environment (optional)
OS: Windows 10 (22H2 build 19045.5131)
Virus scanner: Bitdefender

This issue needs a decision from the team responsible for that area. They have been informed. Response time may vary.

Related:

#31392

Ive been having a similar issue, i noticed that you are also using bitdefender, could these be linked?

Ive been having a similar issue, i noticed that you are also using bitdefender, could these be linked?

It might be related to Antivirus activity, but as said: it didn't happen in Gradle version 8.5. Something got changed in a way that triggers the write permission issues.

Ive been having a similar issue, i noticed that you are also using bitdefender, could these be linked?

My ".gradle" folder is already in bitdefender exceptions since ages ago.
Gradle 8.5 is working fine. Problem started with 8.6.
Manually rename the folder, e.g. "ren 81991bfd1b4ed8debe429166c1ad789e-6b703885-e467-4284-bbc7-e01c819eb8cd 81991bfd1b4ed8debe429166c1ad789e" helps.

Hello, are there any plans to fix the problem? I struggle with that for 2 months at least

I just found out the easiest way to workaround this problem. All you have to do is delete the .\gradle folder altogether and let Android Studio rebuilt it for you.


"Laravel PHPUnit routes not found

Laravel Version
8
PHP Version
8.2
Database Driver & Version
No response
Description
The application functions correctly in the production environment, and I can access all routes without any issues.
However, when running tests using PHPUnit, I receive an error indicating that certain routes cannot be found. This error only occurs in the production environment; all tests pass without any issues on my local development machine.
Configuration:
I have created a .env.testing file and set the APP_URL variable to http://localhost.
I have also cleared the application cache, route cache, and config cache in the production environment.
Testing Code:
The tests in question attempt to access routes defined in the application. The following is an example of a test that fails due to the "route not found" error:
php
Copiar código
/** @test */
public function itCanCreateSymptomType()
{
$user = User::factory()->create(['profile' => 1]);
$this->actingAs($user)->withSession(['_token' => csrf_token()]);
$data = ['description' => 'Test Symptom Type Description'];
$response = $this->call('POST', '/symptomtype', $data);

$this->assertDatabaseHas('symptom_types', $data);

}
Steps Taken:
I have tried using processIsolation in my phpunit.xml configuration, but the issue persists.
I have verified that the routes are defined correctly and accessible during normal application execution.
Request for Assistance: I would appreciate any guidance or troubleshooting steps to resolve this issue. It appears to be related to route accessibility during tests in the production environment, but I cannot identify the underlying cause.
Steps To Reproduce
php artisan test

Hey there,
Unfortunately we don't support this version of the library anymore. Please check out our support policy on which versions we are currently supporting. Can you please try to upgrade to the latest version and see if your problem persists? If so, please open up a new issue and we'll help you out.
Thanks!


cache retrieval check for server remark

If a cached server response includes a "remark" (often indicating a server warning), don't retrieve it from the cache by default.

Coverage decreased (-0.07%) to 92.214% when pulling 5a5b4ff01c4285512e9e211d2067c147844f3306 on feature into db60d205e3b68216a3d53b4fb1cbbf268b89138b on master.


Support for oJDK17

Dear @kevin-ortega and other buildpack contributors,
is there any plan for enabling support for oJDK17?
We did open a case with IBM (TS008791818) Seems like colleagues assigned to the ticket are not really familiar with the Buildpack itself. Could you please get in touch with them and sort this out? They do suspect that liberty is wrongly getting started with java17. However, this is handled by the Buildpack, right?
Basically 'trade' test app is unable to get started with java17
2022-03-21 14:25:32 [APP/PROC/WEB/0] OUT [INFO ] FFDC1015I: An FFDC Incident has been created: "java.lang.IllegalAccessException: class com.ibm.ws.kernel.pseudo.internal.PseudoContextFactory cannot access class com.sun.jndi.ldap.LdapCtxFactory (in module java.naming) because module java.naming does not export com.sun.jndi.ldap to unnamed module @f630ab39 com.ibm.ws.kernel.pseudo.internal.PseudoContextFactory 48" at ffdc_22.03.21_<IP_ADDRESS>.log
We would really appreciate your help and support :)
Best regards,
K.M.

@krismarc Thanks for reaching out.   I read through the SF case.   I will reach out to the Liberty team.
You are correct, the buildpack is providing Java 17 to the application because it has been configured:
-----> Downloading IBM 17.0.2_001 JRE from https://ojdk-j9-17-0-2-0-01.appcloudtest-np.swissre.com/linux/sr-openj9-java-jre-<IP_ADDRESS>_01-linux.tar.gz ... (0.7s)

At this time, we have no specific plans to enable Java 17 in the buildpack.

@kevin-ortega Thanks for quick answer, highly appreciated. Indeed, that's our so called offline repo and we did put new java there as usual. As I understand this requires some extra actions in order to get it started properly using the Buildpack. Manually using a standalone Liberty we are able to get all things up and running without any issues. Even if we figure this out, we would like to avoid having the Buildpack customized by ourselves.
btw, are there any differences between Buildpacks posted here in the github and IBM's downloads center ?
ibm-websphere-liberty-buildpack-v<IP_ADDRESS>.zip (222.7 MB)
Best regards,
K.M.

Can you ssh into the app and navigate to ~/app/.java and provide output of ls?   I pushed an app using openjdk v17 and I don't see a jre directory.

@krismarc I pushed an app using both semeru openj9 v17 JRE and JDK.  In both cases, JAVA_HOME="$PWD/.java" and
java.home = /home/vcap/app/.java are set correctly.
The liberty team believes JAVA_HOME is not set correctly based on where the JRE is actually installed.

@kevin-ortega yes, there's jre directory inside .java but this never changed in our setup. We use the same with java8 and 11 without any issues in starting any app using your Buildpack. Should we change it just for 17? And yes, we are using forked buildpack, however without any code changes. We just put our offline repo urls there and we maintain our java/liberty versions via index.ymls. Keeping the source code is important for us to avoid maintenance of customized changes in the future.
As I understand, JAVA_HOME and java.home should be the same? I am still not sure where we could have messed this up . As said, we never had issues with this until now.
I'll take closer look at this tomorrow. Thanks for your input, I'll keep you posted.

Yes, JAVA_HOME and java.home should be the same.  But the real issue is JAVA_HOME is not pointing to the JRE, it's pointing one directory up.  The buildpack is not expecting ~/app/.java/jre and sets JAVA_HOME to ~/app/.java.  I don't know if the particular java17 you're downloading is different from the semeru v17 JDKs and JRE I tried but neither had a jre directory.
This is the semeru jdk I used:
 -----> Downloading OpenJDK jdk-17.0.2+8_openj9-0.30.0 from https://github.com/AdoptOpenJDK/semeru17-binaries/releases/download/jdk-17.0.2%2B8_openj9-0.30.0/ibm-semeru-open-jdk_x64_linux_17.0.2_8_openj9-0.30.0.tar.gz

I am out of the office until Monday April 18th.

Hey @kevin-ortega,
so you all were right. It's indeed JAVA_HOME problem. The one which is in use during the staging phase.

"web": ".liberty/create_vars.rb wlp/usr/servers/defaultServer/runtime-vars.xml && .liberty/calculate_memory.rb && WLP_SKIP_MAXPERMSIZE=true JAVA_HOME="$PWD/.java/jre" WLP_USER_DIR="$PWD/wlp/usr" exec .liberty/bin/server run defaultServer"

..so, it means we never really had this JAVA_HOME set properly and this never caused any issue to us :) Applications always used to start properly without it.
In order to resolve our initial problem with the lack of accessibility between Liberty classes and Internal java ones.
Java changes described here.

"java.lang.IllegalAccessException: class com.ibm.ws.kernel.pseudo.internal.PseudoContextFactory cannot access class com.sun.jndi.ldap.LdapCtxFactory (in module java.naming) because module java.naming does not export com.sun.jndi.ldap to unnamed module @79c12353 com.ibm.ws.kernel.pseudo.internal.PseudoContextFactory 48"

We had to add exports manually or make a use of default java9.options file.
..and this file is enabled only if you meet some conditions in the script which runs liberty (server) (which is a check of proper JAVA_HOME set)
  if [ -z "${JAVA_HOME}" ]; then
    JAVA_PATH_FROM_PATH=$(command -v java)
    if [ -n "${JAVA_PATH_FROM_PATH}" ] ; then
      JPMS_MODULE_FILE_LOCATION=$(dirname $(dirname $JAVA_PATH_FROM_PATH))/lib/modules
    fi
  else
    JPMS_MODULE_FILE_LOCATION="${JAVA_HOME}/lib/modules"
  fi

  # If we are running on Java 9, apply Liberty's built-in java 9 options
  if [ -f "${JPMS_MODULE_FILE_LOCATION}" ]; then
    mergeJVMOptions "${WLP_INSTALL_DIR}/lib/platform/java/java9.options"
  fi

Thanks for the right hint!
One more question comes to my mind.. what is also going to get enabled or disabled by turning on the right JAVA_HOME? I am afraid this might be not enabling the java9.options file only. I am worried about unexpected side effects of some new configurations enabled/disabled.
Best regards,
K.M.


Error bar text spacing fix

What

Add 0.3rem padding to the top of the fluent message bar
Why

creates even spacing in the errorbar component between the text and the edges of the bar.

https://skype.visualstudio.com/SPOOL/_workitems/edit/2896682
How Tested

Tested in storybook

Looks like the "X" is now not centered:

yeah this is something that wasn't reflected in the story book example. removed it for now and will do some more testing here in the actual app to see if this could be something with the test.

Took a quick look, still looks like the Dismiss button X is still not centered. Will check back later today :)


5.0.0

Coverage remained the same at 100.0% when pulling 119c7e78968e43d907fd9502acdc7c26619a08d7 on frenzzy:v5.0.0 into 126c970f569848088b757af8e31dcd2fb7d48397 on kriasoft:master.