Support for opt-in network isolation in build/test sandboxes

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[x] Have you written new tests for your changes? Here's an example.

[x] Have you successfully run brew typecheck with your changes locally?
[x] Have you successfully run brew tests with your changes locally?

This is probably going to need a lot of iterating, but I think it's in a state that's ready for some eyes and feedback now.
Overview
This PR introduces opt-in (disabled by default) support for network isolation during build, test, and postinstall. A formula can opt-in like so:
class Hello < Formula
  url "[...]"
  sha256 "[...]"
  [...]
  # For enabling on a single phase:
  enable_offline_phase :build
  # For enabling on multiple phases:
  enable_offline_phase [:build, :postinstall]
  # Without arguments, enables on all phases:
  enable_offline_phase

  bottle do
    [...]
  end
  [...]

Suppose we opt-in a formula for offline build phase like so:
class Hello < Formula
  desc "Program providing model for GNU coding standards and practices"
  homepage "https://www.gnu.org/software/hello/"
  url "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz"
  sha256 "8d99142afd92576f30b0cd7cb42a8dc6809998bc5d607d88761f512e26c7db20"
  license "GPL-3.0-or-later"
  enable_offline_phase :build
  [...]

and perhaps there's a pesky curl call in the install block:
def install
    system "curl", "example.org"
    ENV.append "LDFLAGS", "-liconv" if OS.mac?
[...]

When we attempt to install this formula:
==> curl example.org
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: example.org
[...]
Error: hello 2.12.1 did not build
Logs:
     /Users/caleb/Library/Logs/Homebrew/hello/00.options.out
     /Users/caleb/Library/Logs/Homebrew/hello/01.curl
READ THIS: https://docs.brew.sh/Troubleshooting

Things still to be (re-)considered

I refer to "phases" here since it was the first term that came to mind. Any opinions on what to call "build", "test", "postinstall", etc.? This naming would be reflected in the method naming.

Open generally to feedback on the method/DSL naming.

Going in a different direction, is the option to choose phases for which to enable this too granular? Maybe we just settle for one big on/off toggle. But then there's a lot of network-related tools/libraries that probably don't need connectivity to build but might need it for a meaningful test.

Compromise: combine :build and :postinstall into :build, while leaving :test as its own
Maybe we can open some exceptions for localhost in the sandbox, at least when running test block

Running enable_offline_phase without arguments currently enables for all. Would it be less surprising for the no-arguments case to enable for build only? build + postinstall? Or maybe require an argument

Consider providing :all as an easy way to enable for everything, if we decide to require an argument

     What about having:

An 'offline do' block for use when only part(s) of an 'install' (build,) 'postinstall,' or 'test do' block need network isolation;

An ':offline' 'Symbol' argument that formula authors can pass to Formula's 'install,' 'postinstall,' and 'test' methods before their block argument that causes the entire block to get evaluated as if all of its author-sipplied contents were written inside of such an 'offline do' block; and

some kind of DSL to specify that all of a formula's 'install,' 'postinstall,' and 'test' methods are to be evaluated as if they were all passed such an 'offline' argument?
     This could be by either:

Calling 'Formula' method, like the 'enable_offline_phase!' method that this PR introduces in its current form, or
Inheritance from a new 'Formula' subclass (maybe call it OfflineFormula.'

An 'offline do' block for use when only part(s) of an 'install' (build,) 'postinstall,' or 'test do' block need network isolation;

For this point in particular, to achieve this would require more significant re-design of the formula installation/postinstall/test sandbox setup code. Currently, the sandbox is set up before the respective block is run, so the decision/logic to enable this has to be outside the install/test/postinstall blocks; trying to change it inside the block is too late since everything inside e.g. def install is already inside the sandbox.

An ':offline' 'Symbol' argument that formula authors can pass to Formula's 'install,' 'postinstall,' and 'test' methods before their block argument that causes the entire block to get evaluated as if all of its author-supplied contents were written inside of such an 'offline do' block

I personally like this idea but am not sure how it could be achieved syntactically.

An 'offline do' block for use when only part(s) of an 'install' (build,) 'postinstall,' or 'test do' block need network isolation;

For this point in particular, to achieve this would require more significant re-design of the formula installation/postinstall/test sandbox setup code. Currently, the sandbox is set up before the respective block is run, so the decision/logic to enable this has to be outside the install/test/postinstall blocks; trying to change it inside the block is too late since everything inside e.g. def install is already inside the sandbox.
⋮

     Ah, right; definitely fair enough.  Never mind that idea, then, at least for the foreseeable future.

⋮

An ':offline' 'Symbol' argument that formula authors can pass to Formula's '‘install,' 'postinstall,' and 'test' methods before their block argument that causes the entire block to get evaluated as if all of its author-supplied contents were written inside of such an 'offline do`' block

I personally like this idea but am not sure how it (def install :offline) could be achieved syntactically.

     Maybe look at how 'it "", :needs_network do' and 'RSpec.describe klass_or_string, :needs_network do' are implemented in/for 'Library/Homebrew/test/,' though I don't know if that helps since those are part of the test suite helpers.  I think they're in:
https://github.com/Homebrew/brew/blob/master/Library/Homebrew/test/spec_helper.rb#L116-L126
https://github.com/Homebrew/brew/blob/master/Library/Homebrew/test/spec_helper.rb#L158-L169
That's what gave me that idea.

This PR introduces opt-in (disabled by default) support for network isolation during build, test, and postinstall. A formula can opt-in like so:

@alebcay AMAZING work here 👏🏻. Seen many attempts over the years and this seems by far the most likely to actually get adopted.

I refer to "phases" here since it was the first term that came to mind. Any opinions on what to call "build", "test", "postinstall", etc.? This naming would be reflected in the method naming.

Open generally to feedback on the method/DSL naming.

Given there's only three: I wonder about just having them all be separate calls? I guess "methods" would work instead of "phases". I don't think we really describe these to users/contributors anywhere 🤔

But then there's a lot of network-related tools/libraries that probably don't need connectivity to build but might need it for a meaningful test.

Yeh, I think different phases makes sense.

Maybe we can open some exceptions for localhost in the sandbox, at least when running test block

This makes sense to me 👍🏻

The main question I have here isn't actually about the implementation but about formulae: how many formulae have you tried this on already? I'd be interested to see at least 3+ where it's working for (each of) build, postinstall, test before this gets merged.
Also I think it'd make sense pretty quickly to have some sort of environment variable to enable this without the DSL. It'd be great to be able to play around locally or with test-bot to find out which formulae can have this DSL used. We also probably want to e.g. require it unconditionally for new formulae at some point.
Relatedly, based on similar DSL addition work like this, it might be nice to have effectively a no-op reverse DSL e.g. disable_offline_phase. This might seem odd but: if this merges, eventually we'll end up flipping the default on formulae to disabling network access unless specifically enabled (and maybe in ~5-10 years: we remove the ability to disable here entirely).
This DSL would also make it easier to have a triage process (perhaps with the environment variable above) for formulae where:

neither DSL used: formula hasn't been checked yet
enable_* used: formula has been checked and offline works (for at least one method)
disable_* used: formula has been checked and offline is broken (for at least one method)

Again: great work here and looking forward to seeing how this evolves! ❤

@RandomDSdevel I appreciate the enthusiasm but I'm not sure your code review on PRs like this is helpful given your knowledge of Homebrew internals and Ruby, sorry ❤

Thanks for the feedback everyone, made some adjustments:

DSL is now {allow,deny}_network_access_for_{build,postinstall,test} - total 6 methods
Added environment variables HOMEBREW_FORMULA_{BUILD,POSTINSTALL,TEST}_NETWORK. Set to allow or deny to force allow or deny, respectively; otherwise if unset will default to the value(s) specified via DSL in the formula(e). Open to further bikeshedding on the name of these env vars.

One thing I noticed - tests that use resources currently cannot run offline because, unlike formula build time, the resource fetch seems to be executed in the sandbox (vs before the sandbox).
Todo - consider opening up localhost in test sandbox when network access is denied.

Some formulae that already seem to be working offline - or not - with this. Might update with some more in the next day or two.

Formula
Build succeeds?
Postinstall succeeds?
Test succeeds?

cmake
✅
✅
✅

ca-certificates
✅
✅
✅

xz
✅
✅
✅

gh
❌1
✅
✅

1 - go build needs network access to pull modules/dependencies

In the long term, I think there will be formulae (like gh above) that will never be able to build offline with our current processes/policies - we would have to do a lot of vendoring to control downloading stuff ourselves (kind of like we do for some Python formulae).

I don't think we should ever do that vendoring. I think better to make some DSL changes/additions. Most of these dependency download systems use checksummed and/or version pinned downloads here and have their own (used at comparable scale to Homebrew) security around this. Instead we should isolate what they do so they aren't able to e.g. arbitrarily write and perhaps restrict which hosts they can download from.
I think there's probably a place for a preinstall do or fetch do and/or pretest do that has different sandboxing:

it can download arbitrary stuff off the internet
it can only write to the sandboxed temporary directory
maybe even: it can only read from the sandboxed temporary directory
this temporary directory is then copied into the install directory to be used by an offline-only installation

To be clear: this should not block this PR being merged/DSL being added but will be worth thinking about as time goes on and we want to get 100% of homebrew-core formulae offline.

Alternatively, for tests: there was also talk in other issues about having resources assigned as "test resources" (much like how "normal" resources are implicitly "build resources") and then they could be fetched by brew fetch --test or similar. Again: not in this PR but an approach that will let us move forwards here.

@alebcay specifically re:  the second part of https://github.com/Homebrew/brew/pull/17081#issuecomment-2058179803, with the beginnings of a formula triage list:
     I may know of some formulas that would fail to build with network isolation, but it's been a while since I've looked at them, so they may need checking to confirm that this is still the case first.  I can list them either here in this thread or in a separate Homebrew/homebrew-core triage issue thread if(/when?) one gets started.

I don't think we should ever do that vendoring.  … Most of these dependency download systems use checksummed and/or version pinned downloads here and have their own (used at comparable scale to Homebrew) security around this.

     I could see users potentially wanting to do this for at least some formulas in their own, third-party taps, but that's indeed outside the scope claimed by the core tap.

… I think better to make some DSL changes/additions.  …
⁝
I think there's probably a place for a preinstall do or fetch do and/or pretest do that has different sandboxing:

it can download arbitrary stuff off the internet
it can only write to the sandboxed temporary directory
maybe even: it can only read from the sandboxed temporary directory
this temporary directory is then copied into the install directory to be used by an offline-only installation

To be clear: this should not block this PR being merged/DSL being added but will be worth thinking about as time goes on and we want to get 100% of homebrew-core formulae offline.
⁝

     Build-time fetch steps confined to this new block's scope would happen after:

formula source download and resource fetch steps finish and
their results get staged.

Maybe another potential name for the new formula stanza might be 'poststage,' then?

     That's the last feedback I have to add here.

That's the last feedback I have to add here.

@RandomDSdevel I asked you earlier so I'll be a little more firm here: until you're willing to step up contributions such that you're asking to be a Homebrew maintainer: please stop commenting on PRs, thanks.

In the long term, I think there will be formulae (like gh above) that will never be able to build offline with our current processes/policies - we would have to do a lot of vendoring to control downloading stuff ourselves (kind of like we do for some Python formulae).

I don't think we should ever do that vendoring. I think better to make some DSL changes/additions.

Agreed. It would probably be better to run something like go mod vendor (perhaps automatically?) in the source directory before sandboxing happens or something like that. (Which admittedly may be complex, since I'm not sure whether source extraction happens inside or outside the sandbox).

Made some adjustments:

DSL is now reduced back to two methods, {allow,deny}_network_access, accepting a single symbol or array of symbols. If no argument is passed, the method applies to all phases.
Added typing for the new methods in sandbox. It's a bit messy since I tried to limit changes to only the new methods, when it would probably be less jarring once some of the existing methods are also typed. I have another branch with more of those changes, if we would prefer to have that in a PR first and then add this.

@alebcay Would love to see an example PR in homebrew-core!

@alebcay Would love to see an example PR in homebrew-core!

Opened https://github.com/Homebrew/homebrew-core/pull/169720. A few more things I thought of as I opened that:

Do we need to enforce block/statement order for this? (If so I need to figure out how to write a cop for this). I intuitively, without any particular reason, inserted it before def install in this case but open to thoughts on if there's some place that we think makes more sense.
Any considerations needed for this to work when building from source using API? Not something I have tested (and not quite sure how to test this).

Do we need to enforce block/statement order for this? (If so I need to figure out how to write a cop for this).

There's already an existing cop so you should be able to add it to that 👍🏻

inserted it before def install in this case

This makes sense to me.

Any considerations needed for this to work when building from source using API?

Feels like it should be the same building from source/building bottles with/without API.


Add support for custom helm options

Sometimes I need to include --force because of a failed deployment on the first deployment, or include the --timeout option to helm. When I need to do that I hardcode these, and use my version of hubploy. I'm wondering if we could include an option for custom helm options instead? @yuvipanda, I'm happy to make these changes and PR if you suggest a preferred way of doing this. Thanks!

@tjcrone yeah, I think a '--helm-options' flag maybe? That could just get passed in to helm.

Okay. I'm going to have to think about how to pass a list of flags and values for another program through the hubploy argparse with a --helm-options flag. Maybe a list of strings. Maybe a dict. I have not passed a set of options for another program to argparse before. If you have a preferred method, please let me know.

Kindof need a way to get **kwargs working with argparse and the CLI. Googling ...

I'd suggest taking it in as a single string, splitting it with shlex.split, and then passing it to helm.

Another option would be to add "force=False" and "timeout=300" as separate argparse options to hubploy, and then pass them on to helm. I kindof like this idea better. What do you think? If you do not like this, I will do it as a string like you suggest.


Fix tx/rx bytes counters

Boringtun was not reporting handshake bytes and tx keepalive bytes in tunnel stats.

Regarding HANDSHAKE_INIT_SZ I confirmed with Tomasz that it's like this:

type 4 bytes
sender index 4 bytes
unencrypted eph 32 bytes
encrypted static (32 + 16) - > decrypted 32
encrypted timestamp (12 + 16) -> 12 decrypted
MACs (16 + 16) - this are the macs over all messages until this point of the connection

It's correct and it sums up to 148.


Provide list of filters automatically

More details are in the commit message of the first commit.

:+1:
I like this documentation automatically

I merged this so that it does not block further work on doc automation. We can modify it later anyway.


[feature-request] Introduce chart:// url scheme for loading extra values files from inside the chart

Although only the file values.yaml within the chart has any special meaning, it is possible to include other arbitrary files inside the packaged chart (and indeed by default, in the absence of a .helmignore file, all files in the chart directory are included in the packaged chart). By default, these extra files have no purpose, but they can be extracted by end users who helm pull the packaged chart and then untar it.
Among other things, this mechanism can be used to provide extra values files that group several related changes together. For example, the linkerd-control-plane chart includes a file, values-ha.yaml, that includes a variety of non-default values which, when used together, configure Linkerd in a highly-available manner. However, to make use of values-ha.yaml, the end user has to helm pull the chart, untar it, and then manually pass the local path to values-ha.yaml to their helm install or helm upgrade invocation.
I propose that Helm be adjusted to accept URLs with the scheme chart:// when passed to --values/-f flag of helm install, helm upgrade, helm template, and other similar commands, to allow passing as a values file a file which is bundled within the chart itself; for example, helm install linkerd-control-plane linkerd/linkerd-control-plane -f chart://values-ha.yaml would use the values from the values-ha.yaml file inside of the linkerd-control-plane chart.
This option would have several advantages over the currently available methods:

It would play more nicely with other tools such as Helmfile, which work with Helm charts but don't necessarily expose easy ways to run arbitrary shell commands in the midst of their Helm automation.
It would be easier for chart authors to document for end users.
It would simply be more convenient than doing everything manually.

closing as duplicate of https://github.com/helm/helm/issues/4782


Implement Channels CRUD Functionality

Description:
AutoEye aims to improve vehicle organization and management by implementing Channels CRUD functionality. Channels will be used to categorize vehicles based on their usage and configuration. This enhancement will pave the way for future Role-Based Access Control (RBAC) implementation, where access permissions will be managed channel-wise.
Objective:
The primary objective of this issue is to implement Channels CRUD functionality to allow users to add, retrieve, update, and delete channels. Each channel will serve as a grouping mechanism for vehicles, enabling better organization and management.
Tasks:

Database Schema Update:

Add a Channels table to the database schema with fields for ID, name, and any other relevant attributes.

Backend Implementation:

Implement GraphQL endpoints for creating, retrieving, updating, and deleting channels.
Implement necessary validation and error handling for channel operations.
Ensure proper integration with existing backend services.

Frontend Integration:

Update the frontend interface to include UI elements for managing channels.
Implement functionality to display existing channels, add new channels, edit channel details, and delete channels.

Testing:

Write unit tests to validate the functionality of the Channels CRUD operations.
Conduct integration testing to ensure seamless interaction between frontend and backend components.
Perform thorough testing to identify and resolve any potential issues or bugs.

Acceptance Criteria:

Users can successfully perform CRUD operations on channels via the web interface.
Channels are properly stored and retrieved from the database.
Frontend and backend components are seamlessly integrated.

Additional Information:

Channels will serve as a crucial component for organizing vehicles and implementing RBAC in the future.

Committed enhancement to develop branch.

Manage Channels

Create/Edit Channels


Consider AES overhead when testing encrypted folder entries

refs my last comment in #317 - TestArchive doesn't seem to consider AES encryption when testing encrypted directory entries (it always uses ZipConstants.CryptoHeaderSize).
This is just a simple change to use the newly added EncryptionOverheadSize  instead.
I'm not sure what the '+ 2' is allowing for? (empty compressed data overhead? I'm not sure what the comment in the non-encrypted case about not knowing the true size refers to, when directories don't really have a size?)
I certify that I own, and have sufficient rights to contribute, all source code and related material intended to be compiled or integrated with the source code for the SharpZipLib open source product (the "Contribution"). My Contribution is licensed under the MIT License.

(tests presently fail on master because of #353 - might need to finish that one first for testing purposes, or merge the PRs together)


CobbDouglasEconomy - only allows default parameter values

Describe the bug
For HARK/ConsumptionSaving/ConsAggShockModel.py, the CobbDouglasEconomy subclass does not seem to solve using my custom parameter values.
To Reproduce
Steps to reproduce the behavior:
I assign parameters and solve for the AggShockConsumerType agents, and then solve the CobbDouglasEconomy with the AggShockConsumerType agents. For example, I can assign a CRRA value of 4.0 for CobbDouglasEconomy.
Expected behavior
For example, when I solve for the aggregate rental rate and aggregate wage rate using "calc_R_and_W()", I expect to calculate using my CRRA value of 4.0. Instead, it always lists the default parameters, including the default parameter value of CRRA = 2.0.
Desktop (please complete the following information):

Econ-ARK and Python version [eg. 0.10.6 and python 3.8.8]

Can you provide a code snippet rather than a description? The CRRA value
given to a CobbDouglasEconomy isn't used during the iterative solution
method, only in choosing an initial state for the simulation. If you look
at the code for calc_R_and_W, it neither directly nor indirectly calls on
self.CRRA; it only uses the parameters CapShare and DeprFunc. The CRRA
parameter of the economy itself is only used in the update method when
computing the steady state capital-to-labor ratio in a perfect foresight
(riskless) version of the model. That's only used for initializing the
wealth of agents in the simulation, so the economy starts out (reasonably)
near the "right" level for models with risk. Changing CRRA on a
CobbDouglasEconomy should only change the solution to the model due to
numeric error (different initial conditions can lead to a very marginally
different solution).
On Thu, Apr 14, 2022 at 2:39 PM kwkak37 @.***> wrote:

Describe the bug
For HARK/ConsumptionSaving/ConsAggShockModel.py, the CobbDouglasEconomy
subclass does not seem to solve using my custom parameter values.
To Reproduce
Steps to reproduce the behavior:
I assign parameters and solve for the AggShockConsumerType agents, and
then solve the CobbDouglasEconomy with the AggShockConsumerType agents. For
example, I can assign a CRRA value of 4.0 for CobbDouglasEconomy.
Expected behavior
For example, when I solve for the aggregate rental rate and aggregate wage
rate using "calc_R_and_W()", I expect to calculate using my CRRA value of
4.0. Instead, it always lists the default parameters, including the default
parameter value of CRRA = 2.0.
Desktop (please complete the following information):

Econ-ARK and Python version [eg. 0.10.6 and python 3.8.8]

—
Reply to this email directly, view it on GitHub
https://github.com/econ-ark/HARK/issues/1135, or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADKRAFPZMI7VRZSZZ2HVOMLVFBQ5PANCNFSM5TOSF63A
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>


Idiomatic way to do deep target mapping

Have you read the project documentation?

[X] Yes, but it does not include related information regarding my question.
[ ] Yes, but the steps described do not work.
[ ] Yes, but I am having difficulty understanding it and want clarification.

Describe your question
This question is half "how do I do this" and half "what are the nature of the limitations".
I am trying to write a converter that describes how to map one field to another. The nature of the mapping is that a field on my source object drives a field on another object that is embedded more than one layer deep.
I know the goverter mapping does not allow targets as paths but I am struggling to understand what the proper solution here is.
Per the documentation, it seems the case I'm in is "un-flattening" and the suggestion is to use map DOT TARGET; but unless I’m greatly misunderstanding something, that only works when the target fields are nested exactly one level deep and have the same names.
Source code
/** My source type */
type Ingestable struct {
  EntityLocationDesignationShortCode string
  EntityLocationDesignationLongCode string
}

/** My target type - layer 1 */
type MyEntity struct {
  EntityLocation EntityLocation
}

/** My target type - layer 2 */
type EntityLocation struct {
  LocationDesignator LocationDesignator
}

/** My target type - layer 3 */
type LocationDesignator struct {
  LocationShortCode string // Ingested field "EntityLocationDesignationShortCode" belongs here
    LocationLongCode string // Ingested field "EntityLocationDesignationLongCode" belongs here
}

// goverter:converter
type Converter interface {
  // Can't define the conversion from here...
  ConvertIngestableToMyEntity(source Ingestable) MyEntity

  // I could use map . LocationDesignator here except that the fields have different names...
  ConvertIngestableToEntityLocation(source Ingestable) EntityLocation

  // I can specify the mapping here, but how to get ConvertIngestableToMyEntity to then use it?
  // goverter:map EntityLocationDesignationShortCode LocationShortCode
  // goverter:map EntityLocationDesignationLongCode LocationLongCode
  ConvertIngestableToLocationDesignator(source Ingestable) LocationDesignator
}

I know I can solve my problem by specifying a mapping function, but:

In my use-case, the goverter annotations are driving a documentation artifact, so it's desirable to capture as much of the schema definition as possible;
It seems like I shouldn't have to, and I'm trying to understand what limitation I am running up against. Is this something that isn't implemented but could be, or something that should basically be considered insurmountable under the limitations of go?

You could currently solve it like this:
// goverter:converter
type Converter interface {
	// goverter:map . EntityLocation
	ConvertIngestableToMyEntity(source Ingestable) MyEntity

	// goverter:map . LocationDesignator
	ConvertIngestableToEntityLocation(source Ingestable) EntityLocation

	// goverter:map EntityLocationDesignationShortCode LocationShortCode
	// goverter:map EntityLocationDesignationLongCode LocationLongCode
	ConvertIngestableToLocationDesignator(source Ingestable) LocationDesignator
}

With
	// goverter:map . EntityLocation
	ConvertIngestableToMyEntity(source Ingestable) MyEntity

you basically say, may Ingestable to tho EntityLocation property of type EntityLocation. Now goverter tries to convert Ingestable -> EntityLocation, sees another user defined method ConvertIngestableToEntityLocation(source Ingestable) EntityLocation and uses that one.
It should be possible to support nested path on as TARGET in goverter:map, but there are some edge-cases that are difficult and need thinking how they should be implemented. The feature is requested in #80.


patch for reverse lazy url in wagtailadmin menu fixes #1906

Added test as mentioned

Squashed and merged in dee427713e63b7014358e2135ccfff75dc6f1149 thanks!


[FEATURE]Is there a limit to api, is 3.5 ok?

What
A clear and concise description of what the feature is. Please include any specific functionality and why this feature would be beneficial to the project.
Why
Discuss the impact of this feature on current users, system performance, and any other areas of the application. Indicate if this feature would be beneficial to other users.
Implementation guidance
Suggest how this feature might be implemented. Include details like API changes, database schema modifications, architectural considerations, and any other relevant implementation details.

Hi @happywinder, the rate limit will be based on the account limits for the OPENAI_API_KEY you set in the .env file or the config. You can also change model in those files too.


Proposal: Add Windows PE support for Unpackaged Desktop Apps

Proposal: Add Windows PE support for Unpackaged Desktop Apps
Summary
Windows PE (WinPE) is a small operating system used to install, deploy, and repair Windows 10 for desktop editions (Home, Pro, Enterprise, and Education), Windows Server, and other Windows operating systems.

Set up your hard drive before installing Windows.
Install Windows by using apps or scripts from a network or a local drive.
Capture and apply Windows images.
Modify the Windows operating system while it's not running.
Set up automatic recovery tools.
Recover data from unbootable devices.
Add your own custom shell or GUI to automate these kinds of tasks.

For more information, please read https://docs.microsoft.com/en-us/windows-hardware/manufacture/desktop/winpe-intro.
Rationale
Add Windows PE support for Unpackaged Desktop Apps will bring several benefits, for example:

The Windows system utilities developers can build their apps with the modern UI technology. I think it meets the goal for creating the Project Reunion. (Many Windows system utilities developers only use Win32 API to build their apps because they need to support Windows PE.)
Microsoft can modernize their implementations for Windows 10 installation.
Port Windows Terminal to Windows PE for improving the system maintenance experience.
It can help developers include Microsoft to unify the Windows experience.

Scope

Capability
Priority

Add Windows PE support for Unpackaged Desktop Apps
Must

I think this is the common demand of equipment manufacturers.

I hope that MS will not pass this request  to the Feedback Hub and let it rot there forever; ( as first time in the last 9 years, windows Desktop development in general just has slowly started to become exciting again because of Project Reunion.)

+1

The problem with Windows PE is that the base installation needs to fit on a DVD so adding more stuff will not help even if it's also installed on internal hard disk by OEMs for Windows Recovery Environment just causes duplicated files on the same PC. Windows PE doesn't even support anything after .NET Framework 4.5 and even then WPF isn't supported. Since the WinUI framework could be distributed with each app package separately instead of a global install, so it could work

Windows Forms support would be more than enough here IMO. No need to create anything heavyweight when a lightweight, (presumably) compatible approach already exists. Being able to run a WinForms app written targeting .NET 5 on WinPE would tick all the feature boxes I could ask for.

Being able to run a WinForms app written targeting .NET 5 on WinPE would tick all the feature boxes I could ask for.

But we can't use the xml-based XAML layout in a standard WinForms app.


Use assertInternalType

I've refactored some tests using assertInternalType method.

Coverage remained the same at 97.071% when pulling d73f41034fc36294b3655e986d925da46582a0d5 on carusogabriel:refactoring-tests into 3ab1a154e45546fd8654f87708577096391b50f7 on slimphp:3.x.

Thanks @carusogabriel


Panic printing if with folded instructions where parameter is from block param

The fuzzer today discovered that this input:
(module
  (func
    i32.const 0
    try_table (param i32)
      if
      end
    end
  )
)

will panic wasm-tools:
$ cargo run print -f test1.wat
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.04s
     Running `target/debug/wasm-tools print -f test1.wat`
(module
  (type (;0;) (func))
  (type (;1;) (func (param i32)))
thread 'main' panicked at crates/wasmprinter/src/operator.rs:1708:14:
no predicate
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

@keithw would you be interested in looking into this? If you're otherwise busy I'm also happy to dig in further.

Glad it's catching things like that! I think #1840 will fix the panic.
It could be interesting to start testing and fuzzing that every well-formed Wasm module (valid or not) successfully roundtrips through wasmprinter -> wasmparser (with or without folding and/or name-unnamed). This would probably require fixing #1671; I'm not sure if it would be worth the effort and churn given that invalid (but well-formed) Wasm modules probably aren't very important in practice.

Agreed! In https://github.com/bytecodealliance/wasm-tools/pull/1839 I've updated the print fuzzer to frob configuration knobs (folding/name-unnamed/etc) and it just has random bytes thrown at it. In the same PR the validate fuzzer, which has a "mostly valid" module but functions may have random bytes, also tests printing with all the knobs. In https://github.com/bytecodealliance/wasm-tools/pull/1831 I updated the validate_valid_module to ensure that print-then-parse for both folded/non-folded produces the same binary out the other end.
Our coverage for "mostly valid" modules isn't that great though, we're relying relatively heavily on the fuzzer doing a good job for that (aka running for a long time).
For #1671 I think we should be good though? That one's just a difference with the spec interpreter for when an error shows up, not that it shows up at all I think.


Supported Parquet writer tests

Testing different compressions kinds of native parquet writer.

@majetideepak  Please review, thanks !

With the recent changes in the ThriftCodecToCompressionKind() function in PageReader.h , an argument can no longer be passed into the ThriftCodecToCompressionKind() function, meaning that the compressionKind needs to be extracted in a different way.

One workaround could be adding the ThriftCodecToCompressionKind() function  as a helper function to the ParquetTestBase.h as done so here.

@czentgr Please take a look.

Hello  @nmahadevuni,
I am trying to test the parquet writer tests with the supported compression kinds.
I am pulling in the compression codec used to write the file from the file metadata, my goal is to make sure that the file was written and read in the compression kind specified.
For converting the compression Codec obtained from the file MetaData to the CompressionKind, I need to use the thriftCodecToCompressionKind() which is currently a member function of the PageReader class.
Any suggestions on how I can tackle this without duplicating the thriftCodecToCompressionKind() helper function as is done in this PR?
Thank you  for your time.

@majetideepak Can you please review this PR when you get the chance , need your inputs here.

Hi @majetideepak, can you finish the review?

@yzhang1991 This PR (https://github.com/facebookincubator/velox/pull/7643) needs to be reviewed first.

@pdabre12 can you please address the existing comments? Thanks.

@pdabre12 can you please address the existing comments? Thanks.

Hi @majetideepak , I opened a new PR to address the comments. (https://github.com/facebookincubator/velox/pull/7643)

@majetideepak  @czentgr Please review again.

@czentgr Can you take another pass?

@majetideepak Addressed the comments, PTAL, thanks!

Addressed the comments, using an in-memory writer instead.
Keeping the git log incase changes need to be rolled back. If changes are approved, will clean the git log.
Please review,
@majetideepak @pedroerp

@majetideepak Added column-level codec , PTAL.
Thanks.

@pedroerp, can you take another look? Thanks.

@pdabre12 Can you rebase with main? I added new API here https://github.com/facebookincubator/velox/pull/8486
to get the Column Compression. We should use that instead.


Add support for external inputs in Filebeat

As part of application log ingestion into TOP (i.e. Cassandra and Couchbase), it would be great to have the ability of adding external inputs in Filebeat.
Ref: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration-reloading.html
I therefore created this branch with a small addition in the filebeat template config/templates/etc/filebeat/filebeat.yml.j2:
https://github.com/sky-uk/clusterverse/tree/filebeat_ext_inputs

I also found an issue on line 33 of the filebeat template:
{% if beats_config.filebeat.extra_logs_paths %}

should be:
{% if beats_config.filebeat.extra_logs_paths is defined %}

So that the variable beats_config.filebeat.extra_logs_paths can be optional.

@flfernandez it would be a great addition, please don't hesitate to raise a PR when your tests are ok :)

Thanks @antoineserrano
Created PR: https://github.com/sky-uk/clusterverse/pull/110

PR merged, closing this issue.


Bug DMLFusedNode_0_0 on second token in 0.5.2 (DML) (Wrong tensor shape)

I updated to 0.5.2 DirectML mode. Quadro P5000 GPU. Windows. C# DirectML 1.15.4. Model: microsoft/Phi-3-mini-4k-instruct-onnx
I get the following bug. (Bug only comes up in DML mode not CPU mode. It worked in version 0.4.0).
For any prompt e.g.
"<|user|>Hello <|end|><|assistant|>"
(tokens:)
1,32010,15043,29871,32007,32001
It outputs one token but then when trying to output the second token it errors out with:
OnnxRuntimeGenAIException: Non-zero status code returned while running DmlFusedNode_0_0 node. Name:'DmlFusedNode_0_0' Status Message: D:\a\_work\1\s\onnxruntime\core\framework\execution_frame.cc:173 onnxruntime::IExecutionFrame::GetOrCreateNodeOutputMLValue shape && tensor.Shape() == *shape was false. OrtValue shape verification failed. Current shape:{1,32,7,96} Requested shape:{1,32,2048,96}

Microsoft.ML.OnnxRuntimeGenAI.Result.VerifySuccess (System.IntPtr nativeResult) (at D:/a/_work/1/onnxruntime-genai/src/csharp/Result.cs:25)
Microsoft.ML.OnnxRuntimeGenAI.Generator.ComputeLogits () (at D:/a/_work/1/onnxruntime-genai/src/csharp/Generator.cs:25)
Main.Generate () (at Assets/Main.cs:202)

The relevant part seems to be: Current shape:{1,32,7,96} Requested shape:{1,32,2048,96}

I have "solved" this bug by commenting out this line in my code:
generatorParams.SetSearchOption("past_present_share_buffer", false);
I don't know if that is important or not.


Running a computation graph in C++ with custom ops

The current C API does not seem to support loading custom op library, which is tf.load_op_library for python.
Because of this, I cannot run a computation graph with custom ops.
Is there any way I can register custom ops in C?

@jhseu checking what the correct route is

It does support it:
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/c/c_api.h#L973
Python's load_op_library calls it underneath:
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/framework/load_library.py#L55

Closing, but feel free to reopen if I misinterpreted the issue.

What is the usage for C API?


Automate SPFx version upgrade

Category

[x] Question
[ ] Typo
[ ] Bug
[ ] Additional article idea

Is there a way to automate SPFx version uppgrade on a sharepoint project? Thank you!

Isn't Office 365 CLI to automatically detect necessary changes for upgrading SPFx version? It generates a report with steps of commands to run for upgrade. Please let me know if I am wrong. I am curious to know if there is a command which on running would automatically upgrade sharepoint project to a newer version.

You are correct: Office 365 CLI generates a report with the list of steps that need to be applied manually to upgrade the project to a newer version. You could of course automate this process yourself, by executing the suggested commands and applying the recommended changes, but it's not trivial and you should ensure you have a backup of your project in case you need to revert the changes.

Thank you @waldekmastykarz

You're welcome!


fix: Add missing hitbox parameters

Description

This PR adds the position parameter to PolygonHitbox and isSolid parameters to all hitbox classes' .relative named constructors.
Also returns PolygonComponent's internal topLeft parameter to private _topLeft as no longer required outside of the class file.
Checklist

[x] The title of my PR starts with a Conventional Commit prefix (fix:, feat:, docs: etc).
[x] I have followed the Contributor Guide when preparing my PR.
[ ] I have updated/added tests for ALL new/updated/fixed functionality.
[ ] I have updated/added relevant documentation in docs and added dartdoc comments with ///.
[ ] I have updated/added relevant examples in examples or docs.

Breaking Change?

[ ] Yes, this PR is a breaking change.
[x] No, this PR is not a breaking change.

Related Issues

Adding the position parameter to PolygonHitbox also means the same vertices list used for (an optional) parent PolygonComponent can be reused in the PolygonHitbox constructor with position: Vector2.zero(). Previously position would be fixed at null and the hitbox would be offset by topLeft, with no way of correcting except to clone and modify the original vertices list.


Compiling source

What is the best approach for cross-platform compiling a C library for node-ffi? Is it possible to use node-gyp?

Ya you can use node-gyp. It's probably not the best approach but it works. See the test case.

What's the best approach?

It depends on your application, there is no clear generic answer. There's meta build systems like cmake, Ninja, etc. If node-gyp works for you then use it!

I'm wanting my c source code to build on when npm install is run.

And it needs to work for Linux, Mac, and Windows

Node-gyp gives me a dynamic linking error.

See something like this:

https://github.com/TooTallNate/node-dlfcn/blob/master/test/binding.gyp
https://github.com/TooTallNate/node-dlfcn/blob/2db2bcd409c60afe3f028331e212409120a3a5b5/package.json#L7

Gonna close since this isn't a node-ffi issue.


[WIP] Update upstream tests for Istanbul

What was wrong?
Please ignore this for now. I'm just experimenting. I will ping the team once it's ready or if I need help
How was it fixed?
not fixed yet.
TODO:

[x] Blockchain Tests are not yet included for Istanbul yet. Update the commit when they get included.
[x] blake2 tests are still WIP. Update them when they are ready.
[x] Find out what's wrong with the blockchain tests for older VMs and fix them.
[x] Update CircleCI to include Istanbul tests
[ ] Fix Istanbul blockchain tests

To-Do

[ ] Clean up commit history

[ ] Add entry to the release notes

Cute Animal Picture

Test failures are related to this issue: https://github.com/ethereum/tests/issues/637.
The format of tests has changed. I'll have to wait until the format is finalized.

Tests are crashing on CircleCi. I ssh'd into the machine and monitored the memory usage using htop. When the tests are about to crash the memory usage shoots to 4gb. CircleCI by default allocates 4gb of memory which is the reason for the crash IMO. I tried increasing the machine size but currently, the paid tier doesn't support large instances.
cc @pipermerriam

UPDATE: I got tests to turn green by marking the tests slow that were consuming a lot of memory.
These tests have the following comment:
test takes a lot for time because of sha3 calculation on a huge range.
Is it okay to mark these tests as slow?

upstream tests now include blockchain tests for Istanbul. 11 tests are failing at the moment. I will at these test failures later.
Below are some stats for Istanbul blockchain tests:
11 failed, 10607 passed, 32 skipped

Semi-OT: may I also note that in the test durations, there's
42.96s call     tests/json-fixtures/test_blockchain.py::test_blockchain_fixtures[/home/circleci/repo/fixtures/BlockchainTests/GeneralStateTests/stCreate2/Create2Recursive.json:Create2Recursive_d0g0v0_Istanbul:Istanbul]
41.59s call     tests/json-fixtures/test_blockchain.py::test_blockchain_fixtures[/home/circleci/repo/fixtures/BlockchainTests/GeneralStateTests/stCreate2/Create2Recursive.json:Create2Recursive_d0g1v0_Istanbul:Istanbul]
26.76s call     tests/json-fixtures/test_blockchain.py::test_blockchain_fixtures[/home/circleci/repo/fixtures/BlockchainTests/GeneralStateTests/stStaticCall/static_Call50000.json:static_Call50000_d0g0v0_Istanbul:Istanbul]

Perhaps these should be marked SLOW_TESTS as well.

OTOH, the entire "run tox" section is 02:26, so maybe that's fine.

Yeah CI runs them in parallel. 2:26 doesn't seem too long to me.
@veox I could borrow a little help from you. This is regarding the failing tests in Istanbul. I did not get time to look closely into the issue but I thought that I'd gather some information before I'd start digging. Until now I have figured out that the state_root for the computed block header and the one from the tests fixtures don't match. This happens when the storage is empty. I remember that you faced a similar issue while updating tests for Constantinople. Can you describe that issue in brief to me or do you have any hunch if the two are related?

@voith I don't remember the specifics (poor memory). I think it was Trinity not applying new EIP-1283 gas refund rules to pre-compiles (as it rightfully shouldn't); but again, not even sure on that.

Until now I have figured out that the state_root for the computed block header and the one from the tests fixtures don't match. This happens when the storage is empty.

This can happen for any reason. Different gas use/stipend if storage is empty, or a different value being written to storage (0 instead of 1, say - resulting in, again, different gas usage)... Need to dig into specific tests that fail.

@voith You pinged me for a review. I skimmed over this and left a view comments. I noticed the failing tests but didn't get into debugging those. Do you need help debugging them? Do you know anything about the failures yet? I know upgrading these tests is more like an art than a craft so I'm trying to gather as much info from you as I can before I start diving in ;)

@cburgdorf I triggered the review button by mistake. I've been meaning to sit and debug these tests but haven't got time.

Do you need help debugging them?

Actually, I wanted to do this by myself. On Saturday I tried debugging for a while and I ended up learning a lot about the VM. But I only ended up learning about the VM, didn't debug anything. But I will be unable to get back to this anytime soon.
But I understand very well that this PR is quite urgent, so please feel free to continue this. If no one takes it up then I'll surely complete it the following week after Devcon(Too late, I know).

trying to gather as much info from you as I can before I start diving in

So I didn't get too far with debugging this. But here are a few pointers:

For now, I have marked  5 tests as incorrect. These tests are of the same type that @veox marked as Incorrect while trying to update tests for Constantinople.
The RevertPrecompiledTouch failing tests need to debugged. Here's a comment from the ethereum/tests gitter channel that might help you.
Istanbul has some tests for blake, some of which are very very slow. These need to be marked slow. But I haven't spent time trying to identify which of these are slow.

I triggered the review button by mistake.

Ah, no problem!

I've been meaning to sit and debug these tests but haven't got time.
But I understand very well that this PR is quite urgent, so please feel free to continue this. If no one takes it up then I'll surely complete it the following week after Devcon(Too late, I know).

To be honest, I'm not sure if I could complete it any earlier. I'm knee deep in adding Görli support and would rather like to avoid the context switch if I can :sweat_smile:
So, carry on and if it's helpful we can also stick heads together at devcon or the days right after :+1:

FTR, I started looking into this, too.
Failing:
test_blockchain_fixtures[/home/veox/src/py-evm/fixtures/BlockchainTests/GeneralStateTests/stRevertTest/RevertPrecompiledTouch.json:RevertPrecompiledTouch_d0g0v0_Istanbul:Istanbul]
test_blockchain_fixtures[/home/veox/src/py-evm/fixtures/BlockchainTests/GeneralStateTests/stRevertTest/RevertPrecompiledTouchExactOOG.json:RevertPrecompiledTouchExactOOG_d31g2v0_Istanbul:Istanbul]
test_blockchain_fixtures[/home/veox/src/py-evm/fixtures/BlockchainTests/GeneralStateTests/stRevertTest/RevertPrecompiledTouch_storage.json:RevertPrecompiledTouch_storage_d0g0v0_Istanbul:Istanbul]
test_blockchain_fixtures[/home/veox/src/py-evm/fixtures/BlockchainTests/GeneralStateTests/stRevertTest/RevertPrecompiledTouch.json:RevertPrecompiledTouch_d3g0v0_Istanbul:Istanbul]
test_blockchain_fixtures[/home/veox/src/py-evm/fixtures/BlockchainTests/GeneralStateTests/stRevertTest/RevertPrecompiledTouchExactOOG.json:RevertPrecompiledTouchExactOOG_d7g2v0_Istanbul:Istanbul]
test_blockchain_fixtures[/home/veox/src/py-evm/fixtures/BlockchainTests/GeneralStateTests/stRevertTest/RevertPrecompiledTouch_storage.json:RevertPrecompiledTouch_storage_d3g0v0_Istanbul:Istanbul]

Those look suspiciously like the existing cases where "synthetic" state stays cleared even after a REVERT.
I vaguely remember someone from the go-ethereum arguing that "this is valid because EIP-158 ("State clearing")", but that didn't seem like valid logic to me, since EIP-158 was long before the introduction of REVERT. Or maybe that's just a faulty memory chip.
Anyway, I hope to look into EVM stack traces tomorrow.

superseded by #1858.


Added aria support to angular-toggle-switch

Hi I was using this very handy plugin in my project, turned out i had to add aria support to my web app, hence ended up modifying the component.
I have made a small change which allows the user to tab to this control and change the state with the space key

Still need to remove the tabindex and also don't build the main file.  I do that on release of the library.

tabIndex removed

In release 1.1.0


v1.4

Codecov Report

Merging #229 (1446f0b) into master (b4cd444) will increase coverage by 0.99%.
The diff coverage is 35.48%.

@@             Coverage Diff              @@
##             master     #229      +/-   ##
============================================
+ Coverage     66.16%   67.16%   +0.99%
- Complexity      625      640      +15
============================================
  Files           111      111
  Lines          2143     2156      +13
  Branches        235      236       +1
============================================
+ Hits           1418     1448      +30
+ Misses          643      628      -15
+ Partials         82       80       -2

Impacted Files
Coverage Δ

src/main/java/seedu/address/model/CsBook.java
85.71% <ø> (-4.09%)
:arrow_down:

src/main/java/seedu/address/model/group/Group.java
80.95% <0.00%> (-6.23%)
:arrow_down:

...c/main/java/seedu/address/model/student/Email.java
80.00% <ø> (ø)

...ain/java/seedu/address/ui/DetailedStudentCard.java
0.00% <0.00%> (ø)

src/main/java/seedu/address/ui/MainWindow.java
0.00% <ø> (ø)

...java/seedu/address/logic/commands/NoteCommand.java
60.00% <33.33%> (+60.00%)
:arrow_up:

...java/seedu/address/logic/commands/EditCommand.java
96.66% <100.00%> (+10.09%)
:arrow_up:

.../seedu/address/logic/parser/EditCommandParser.java
100.00% <100.00%> (+36.36%)
:arrow_up:

...rc/main/java/seedu/address/model/ModelManager.java
70.07% <100.00%> (-1.93%)
:arrow_down:

...rc/main/java/seedu/address/model/student/Note.java
88.88% <100.00%> (+26.38%)
:arrow_up:

... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b4cd444...1446f0b. Read the comment docs.


TEST, DO NOT MERGE

🤖 AutoRest automatic publish job 🤖
> will publish once PR gets merged

🤖 AutoRest automatic feature coverage report 🤖
feature set version 2.4.3
❌ General: 97%
12 out of 535 features are not covered by tests
additionalPropertiesInProperties, additionalPropertiesInPropertiesWithAPTypeString, additionalPropertiesTrue, additionalPropertiesTypeObject, additionalPropertiesTypeString, animalNotFoundError, expectedNoErrors, expectedPetHungryError, expectedPetSadError, intError, linkNotFoundError, stringError
✔ Azure: 100%

🤖 AutoRest automatic feature coverage report 🤖
feature set version 2.4.3
❌ General: 97%
12 out of 535 features are not covered by tests
additionalPropertiesInProperties, additionalPropertiesInPropertiesWithAPTypeString, additionalPropertiesTrue, additionalPropertiesTypeObject, additionalPropertiesTypeString, animalNotFoundError, expectedNoErrors, expectedPetHungryError, expectedPetSadError, intError, linkNotFoundError, stringError
❌ Azure: 0%
no tests were run for this category

win32-x64 restart

🤖 AutoRest automatic publish job 🤖
> will publish once PR gets merged

🤖 AutoRest automatic feature coverage report 🤖
feature set version 2.4.3
❌ General: 97%
12 out of 535 features are not covered by tests
additionalPropertiesInProperties, additionalPropertiesInPropertiesWithAPTypeString, additionalPropertiesTrue, additionalPropertiesTypeObject, additionalPropertiesTypeString, animalNotFoundError, expectedNoErrors, expectedPetHungryError, expectedPetSadError, intError, linkNotFoundError, stringError
✔ Azure: 100%

🤖 AutoRest automatic publish job 🤖
> will publish once PR gets merged


Cannot use localhost as base url

Describe the bug
Unable to use 'localhost' as base url.
To Reproduce
Steps to reproduce the behavior:

Go to https://liyasthomas.github.io/postwoman/
Click on URL field
Type in: http://localhost:3002 (any port will do, I just happen to have a Node.js server running on that port)
Send button is disabled

Expected behavior
I expect the Send button to be enabled
Desktop (please complete the following information):

OS: Arch Linux (64-bit)
Browser: Chromium
Version: 75.0.3770.142 (Official Build)

Additional context
I cloned the repo and I think the issue is related to pages/index.vue line 247 on urlNotValid's validation pattern, I changed the return statement to always be false, and the problem was solved.

This might be a possibile duplicate of #2
Its a restriction due to the CORS (Cross Origin Request Policy), there isn't a client side work around to this unless the host your trying to access should respond with the appropriate CORS headers.
Please refer to docs here: https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS
Quick solution is to use a HTTPS API host, which supports CORS

I first thought of that, but my Node.js API is configured to support CORS requests. I'm sure it's just the URL validation's pattern which expects an URL with a domain at the end (any_url + .com .co .ar .jp ...). I'll see if I can open a PR to solve the problem tomorrow.

If that's the case, I'll surely look into it. You're welcomed to open a PR to fix it.

Ah yes, I know the issue, I had to mess with the regex to get websockets to work. I'll see if I can find some more permissive ones - and I'll add in IP addresses.

You could've opted to use two regex for URLValidation()  🤭

Yes, that was probably how I'd do it.
That said - is validation even necessary? Perhaps it would be ideal to just show an error if it fails to connect, like the websockets do, then to handle each edge case in regexes?

Sooner or later, URL validation will be necessary. Anyway let's discuss.

Okay. I'll update the url validation to include the two regex checks in the mean time.

@NBTX sendRequest() isn't firing. Can you check changes on URL validation

Solved by #52


[Fluent] Play button and icon bug

Describe the bug
The play button is all white and folder / playlist icons don't load
To Reproduce
Steps to reproduce the behavior:
open spotify
Expected behavior
Play button is visible, icons for folders/playlists load
Screenshots

Logs
fluent.js:28 Uncaught TypeError: Cannot read properties of undefined (reading 'get')
    at replacePlaylistIcons (fluent.js:28:31)
    at fluent.js:47:5
    at waitForElement (fluent.js:6:7)

Specifics (please complete the following information):

OS: Windows 10
Spotify version <IP_ADDRESS>3.gc2965cdf
Spicetify version 2.9.0
Fluent

fixed on spicetify 2.9.1 update!


Android Studio 导入libdobby.a静态库，编译时，链接阶段报错

在github上下载最新的android版本静态库，在Android Studio项目中使用。
使用的ndk版本:android-ndk-r25b
ld: error: relocation R_AARCH64_ADR_PREL_PG_HI21 cannot be used against symbol 'regions'; recompile with -fPIC

defined in D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a(ProcessRuntimeUtility.cc.o)
referenced by ProcessRuntimeUtility.cc
ProcessRuntimeUtility.cc.o:(ProcessRuntimeUtility::GetProcessMemoryLayout()) in archive D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a

ld: error: relocation R_AARCH64_ADD_ABS_LO12_NC cannot be used against symbol 'regions'; recompile with -fPIC

defined in D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a(ProcessRuntimeUtility.cc.o)
referenced by ProcessRuntimeUtility.cc
ProcessRuntimeUtility.cc.o:(ProcessRuntimeUtility::GetProcessMemoryLayout()) in archive D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a

ld: error: relocation R_AARCH64_ADR_PREL_PG_HI21 cannot be used against symbol 'regions'; recompile with -fPIC

defined in D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a(ProcessRuntimeUtility.cc.o)
referenced by ProcessRuntimeUtility.cc
ProcessRuntimeUtility.cc.o:(ProcessRuntimeUtility::GetProcessMemoryLayout()) in archive D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a

ld: error: relocation R_AARCH64_ADD_ABS_LO12_NC cannot be used against symbol 'regions'; recompile with -fPIC

defined in D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a(ProcessRuntimeUtility.cc.o)
referenced by ProcessRuntimeUtility.cc
ProcessRuntimeUtility.cc.o:(ProcessRuntimeUtility::GetProcessMemoryLayout()) in archive D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a

ld: error: relocation R_AARCH64_ADR_PREL_PG_HI21 cannot be used against symbol 'regions'; recompile with -fPIC

defined in D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a(ProcessRuntimeUtility.cc.o)
referenced by ProcessRuntimeUtility.cc
ProcessRuntimeUtility.cc.o:(ProcessRuntimeUtility::GetProcessMemoryLayout()) in archive D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a

ld: error: relocation R_AARCH64_ADD_ABS_LO12_NC cannot be used against symbol 'regions'; recompile with -fPIC

defined in D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a(ProcessRuntimeUtility.cc.o)
referenced by ProcessRuntimeUtility.cc
ProcessRuntimeUtility.cc.o:(ProcessRuntimeUtility::GetProcessMemoryLayout()) in archive D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a

ld: error: relocation R_AARCH64_ADR_PREL_PG_HI21 cannot be used against symbol 'regions'; recompile with -fPIC

defined in D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a(ProcessRuntimeUtility.cc.o)
referenced by ProcessRuntimeUtility.cc
ProcessRuntimeUtility.cc.o:(ProcessRuntimeUtility::GetProcessMemoryLayout()) in archive D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a

ld: error: relocation R_AARCH64_LDST64_ABS_LO12_NC cannot be used against symbol 'regions'; recompile with -fPIC

defined in D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a(ProcessRuntimeUtility.cc.o)
referenced by ProcessRuntimeUtility.cc
ProcessRuntimeUtility.cc.o:(ProcessRuntimeUtility::GetProcessMemoryLayout()) in archive D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a

ld: error: relocation R_AARCH64_ADR_PREL_PG_HI21 cannot be used against symbol 'regions'; recompile with -fPIC

defined in D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a(ProcessRuntimeUtility.cc.o)
referenced by ProcessRuntimeUtility.cc
ProcessRuntimeUtility.cc.o:(ProcessRuntimeUtility::GetProcessMemoryLayout()) in archive D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a

ld: error: relocation R_AARCH64_LDST64_ABS_LO12_NC cannot be used against symbol 'regions'; recompile with -fPIC

defined in D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a(ProcessRuntimeUtility.cc.o)
referenced by ProcessRuntimeUtility.cc
ProcessRuntimeUtility.cc.o:(ProcessRuntimeUtility::GetProcessMemoryLayout()) in archive D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a

ld: error: relocation R_AARCH64_ADR_PREL_PG_HI21 cannot be used against symbol 'regions'; recompile with -fPIC

defined in D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a(ProcessRuntimeUtility.cc.o)
referenced by ProcessRuntimeUtility.cc
ProcessRuntimeUtility.cc.o:(ProcessRuntimeUtility::GetProcessMemoryLayout()) in archive D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a

ld: error: relocation R_AARCH64_ADD_ABS_LO12_NC cannot be used against symbol 'regions'; recompile with -fPIC

defined in D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a(ProcessRuntimeUtility.cc.o)
referenced by ProcessRuntimeUtility.cc
ProcessRuntimeUtility.cc.o:(ProcessRuntimeUtility::GetProcessMemoryLayout()) in archive D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a

ld: error: relocation R_AARCH64_ADR_PREL_PG_HI21 cannot be used against symbol 'regions'; recompile with -fPIC

defined in D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a(ProcessRuntimeUtility.cc.o)
referenced by ProcessRuntimeUtility.cc
ProcessRuntimeUtility.cc.o:(ProcessRuntimeUtility::GetProcessMemoryLayout()) in archive D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a

ld: error: relocation R_AARCH64_ADD_ABS_LO12_NC cannot be used against symbol 'regions'; recompile with -fPIC

defined in D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a(ProcessRuntimeUtility.cc.o)
referenced by ProcessRuntimeUtility.cc
ProcessRuntimeUtility.cc.o:(ProcessRuntimeUtility::GetProcessMemoryLayout()) in archive D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a

ld: error: relocation R_AARCH64_ADR_PREL_PG_HI21 cannot be used against symbol 'regions'; recompile with -fPIC

defined in D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a(ProcessRuntimeUtility.cc.o)
referenced by ProcessRuntimeUtility.cc
ProcessRuntimeUtility.cc.o:(_GLOBAL__sub_I_ProcessRuntimeUtility.cc) in archive D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a

ld: error: relocation R_AARCH64_ADD_ABS_LO12_NC cannot be used against symbol 'regions'; recompile with -fPIC

defined in D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a(ProcessRuntimeUtility.cc.o)
referenced by ProcessRuntimeUtility.cc
ProcessRuntimeUtility.cc.o:(_GLOBAL__sub_I_ProcessRuntimeUtility.cc) in archive D:/WorkSpace/XposedNative/app/src/main/cpp/jniLibs/arm64-v8a/libdobby.a
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.

我也遇到了个问题，用的mk文件编译的，百度说加个LOCAL_CFLAGS  := -fPIC就可以，但是我加了还是不行

已找到解决方法，Android.mk文件加上这一行就可以通过：LOCAL_LDFLAGS += -Wl,-Bsymbolic


Git cheatsheet

Git commands added.

Thanks for your guide too. can you please give me an idea of the type of topics to add.

@chidsss try to add docs related to new topics like ML, cloud (AWS,GCP ,etc), DevOps, BigData Processing(Spark, Flink, etc), ML Tools/lib(TensorFlow, Theano, etc), ..
Or you may also go with Docs for Languages like Go, Ruby, etc which are relatively new.


C-like enum nested inside of an internally tagged enum fails to deserialize

Trying to roundtrip this type Rust-RON-Rust:

#[derive(Debug, Serialize, Deserialize)]
enum InnerEnum {
    UnitVariant,
}

#[derive(Debug, Serialize, Deserialize)]
struct Container {
    field: InnerEnum,
}

#[derive(Debug, Serialize, Deserialize)]
#[serde(tag = "tag")]
enum OuterEnum {  // <- this type
    Variant(Container),
}

gives the following RON:
(
    tag: "Variant",
    field: UnitVariant,
)

which produces the following error when trying to deserialize it:
4:2: Expected string or map but found a unit value instead

(playground-ish)
Some notes:

Deserialization to ron::Value reveals that the variant of the InnerEnum is lost and replaced with Unit.
The error location is particularly unhelpful: it points to the end of the file.

Full repro code duplicated here for resillience
/*
[dependencies]
serde = { version = "1", features = ["derive"] }
ron = "=0.8.0"
*/

use serde::{Deserialize, Serialize};

#[derive(Debug, Serialize, Deserialize)]
enum InnerEnum {
    UnitVariant,
}

#[derive(Debug, Serialize, Deserialize)]
struct Container {
    field: InnerEnum,
}

#[derive(Debug, Serialize, Deserialize)]
#[serde(tag = "tag")]
enum OuterEnum {
    Variant(Container),
}

fn main() -> Result<(), ron::Error> {
    let value = OuterEnum::Variant(Container {
        field: InnerEnum::UnitVariant,
    });
    let serialized = ron::ser::to_string_pretty(&value, <_>::default())?;
    println!("--- serialized ---\n{serialized}\n");
    let ron_value: ron::Value = ron::from_str(&serialized)?;
    println!("--- deserialized as ron::Value ---\n{ron_value:#?}\n");
    let err = ron::from_str::<OuterEnum>(&serialized).unwrap_err();
    println!("--- deserialization error ---\n{err}");

    Ok(())
}

Possibly related to #217, but doesn’t use neither untagged nor externally tagged enums.

Thank you for the detailed report @GoldsteinE :) I've expanded it a bit to cover all enum cases:
use serde_derive::{Serialize, Deserialize};

#[derive(Debug, Serialize, Deserialize, PartialEq, Eq)]
enum InnerEnum {
    UnitVariant,
}

#[derive(Debug, Serialize, Deserialize, PartialEq, Eq)]
struct Container {
    field: InnerEnum,
}

#[derive(Debug, Serialize, Deserialize, PartialEq, Eq)]
enum OuterEnum {
    Variant(Container),
}

#[derive(Debug, Serialize, Deserialize, PartialEq, Eq)]
#[serde(tag = "tag")]
enum OuterEnumInternal {
    Variant(Container),
}

#[derive(Debug, Serialize, Deserialize, PartialEq, Eq)]
#[serde(tag = "tag", content = "c")]
enum OuterEnumAdjacent {
    Variant(Container),
}

#[derive(Debug, Serialize, Deserialize, PartialEq, Eq)]
#[serde(untagged)]
enum OuterEnumUntagged {
    Variant(Container),
}

#[test]
fn test_enum_in_enum_roundtrip() {
    let outer = OuterEnum::Variant(Container { field: InnerEnum::UnitVariant });

    let ron = ron::to_string(&outer).unwrap();

    assert_eq!(ron, "Variant((field:UnitVariant))");

    let de = ron::from_str::<OuterEnum>(&ron);

    assert_eq!(de, Ok(outer));
}

#[test]
fn test_enum_in_internally_tagged_roundtrip() {
    let outer = OuterEnumInternal::Variant(Container { field: InnerEnum::UnitVariant });

    let ron = ron::to_string(&outer).unwrap();

    assert_eq!(ron, "(tag:\"Variant\",field:UnitVariant)");

    // Wrong JSONy RON would correctly deserialise here
    assert_eq!(ron::from_str::<OuterEnumInternal>("(tag:\"Variant\",field:\"UnitVariant\")").as_ref(), Ok(&outer));

    let de = ron::from_str::<OuterEnumInternal>(&ron);

    // FAILS
    assert_eq!(de, Ok(outer));
}

#[test]
fn test_enum_in_adjacently_tagged_roundtrip() {
    let outer = OuterEnumAdjacent::Variant(Container { field: InnerEnum::UnitVariant });

    let ron = ron::to_string(&outer).unwrap();

    assert_eq!(ron, "(tag:\"Variant\",c:(field:UnitVariant))");

    let de = ron::from_str::<OuterEnumAdjacent>(&ron);

    assert_eq!(de, Ok(outer));
}

#[test]
fn test_enum_in_untagged_roundtrip() {
    let outer = OuterEnumUntagged::Variant(Container { field: InnerEnum::UnitVariant });

    let ron = ron::to_string(&outer).unwrap();

    assert_eq!(ron, "(field:UnitVariant)");

    // Wrong JSONy RON would correctly deserialise here
    assert_eq!(ron::from_str::<OuterEnumUntagged>("(field:\"UnitVariant\")").as_ref(), Ok(&outer));

    let de = ron::from_str::<OuterEnumUntagged>(&ron);

    // FAILS
    assert_eq!(de, Ok(outer));
}

Normal enums and adjacently tagged enums work here, but untagged enums and internally tagged enums fail. In both cases, the problem is that serde is built for JSON and deserialises the input into an effectively-JSON Value, which does not preserve the identity of the UnitVariant enum variant. #409 would fix this issue but comes with its own problems. For now the RON stance has been that RON does not fully support using RON as a self-describing format, i.e. anything that needs to go through deserialize_any, like untagged and internally tagged enums and struct flattening. I'm trying to slowly change that though. However, until serde gives the formats more control over how these special functionalities are implemented (e.g. though a format-specific Value type), supporting them in RON requires nasty hacks that are difficult to maintain since RON just is not JSON.

I think that “RON does not support untagged and internally tagged enums (and also flattening)” should be documented in some visible place, because it’s features that are normally expected from a serde format, and people use enums a lot in “Rusty” code. It’s especially important given that README says:

It's designed to support all of Serde's data model, so structs, enums, tuples, arrays, generic maps, and primitive values.

and also

Note the following advantages of RON over JSON: ... enums are supported (and less verbose than their JSON representation)

which is really misleading, given that serde_json actually supports enums more fully (while they’re obviously not a first-class feature in JSON).
The (only?) way to get the information about RONs limitations now is to read through the issues, which the user isn’t supposed to do IMO. I think a “Limitations / known bugs” section in the README and/or the docs.rs/ron page would be really helpful to the new users.
The other thing that I think is worth exploring is a way to make the error message more useful. Currently it doesn’t even point to the correct span, so the issue takes some effort to minimize and debug.

I've opened #450 - does that limitations section look good to you?


[Bug]: SHOPIFY_CLI_TTY behavior inverted when CLI crashes

Please confirm that you have:

[X] Searched existing issues to see if your issue is a duplicate. (If you’ve found a duplicate issue, feel free to add additional information in a comment on it.)
[X] Reproduced the issue in the latest CLI version.

In which of these areas are you experiencing a problem?
Theme
Expected behavior
CLI Errors immediately without prompting for user input.
Actual behavior
CLI prompts for use input despite SHOPIFY_CLI_TTY being 0
Verbose output
% shopify theme push --theme=neiltest --unpublished --json
✗ An unexpected error occurred.
	To submit an issue include the stack trace.
	To print the stack trace, add the environment variable SHOPIFY_CLI_STACKTRACE=1.

neil@Neils-Air dist % export SHOPIFY_CLI_TTY=0
neil@Neils-Air dist % shopify theme push --theme=neiltest --unpublished --json
✗ An unexpected error occurred.
	To submit an issue include the stack trace.
	To print the stack trace, add the environment variable SHOPIFY_CLI_STACKTRACE=1.

? Send an anonymized error report to Shopify? (Choose with ↑ ↓ ⏎, filter with 'f')
> 1. Yes, send
  2. No, don't send

### Reproduction steps

1. Populate a store with 100 themes
2. Do a theme push to the store - Will error and die.
3. export SHOPIFY_CLI_TTY=0
4. Repeat a theme push - CLI will error, but then prompt for input about sending a bug report.

### Operating System

Ubuntu

### Shopify CLI version (check your project's `package.json` if you're not sure)

3.55.4

### Shell

Bash

### Node version (run `node -v` if you're not sure)

v20.10.0

### What language and version are you using in your application?

Node 20

same issue. using BitBucket Pipelines.
      - export SHOPIFY_CLI_TTY=$SHOPIFY_CLI_TTY (set to 0)


Support Kubernetes and Knative cluster setup for arm devices

Summary
Support stock-only k8s and knative cluster setup for arm64/amd64 ubuntu 18.04 devices.
Implementation Notes :hammer_and_pick:

Removed restrictions in setup scripts for arm architecture
Edit installation paths to support arm devices

External Dependencies :four_leaf_clover:

N/A

Breaking API Changes :warning:

N/A

@yulinzou Since go version of vHive has been downgraded, please rebase your current branch on master branch, and check again if it breaks the unit tests


Links in Content Block

HI, I just put a site together on my development server and the client entered all the content into the site while it was on my server.  When I moved the site over to their server all the links in Content Bocks were pointing at my development server and not to the site it was now installed on.  No other links seem to be effected, just the Content Block, they used the redactor link button to add in the links.

Can anyone else verify this? If you clear cache does the problem persist? Content blocks cache their output, likely including the links.

I don't have develop branch in front of me right this second but in <IP_ADDRESS> it appears to be the case, the full URL is being used when you use the sitemap picker

As with other releases, please verify that the full URL actually gets
inserted into the database. There are many routines that should be run on
save that replace URLs with placeholders which, when executed, should
automatically insert the proper values in the output content.
On Wed, Oct 7, 2015 at 6:14 AM, Joe Meyer<EMAIL_ADDRESS>wrote:

[image: image]
https://cloud.githubusercontent.com/assets/1292766/10338364/aa950048-6cd3-11e5-86ce-98a3ebb70580.png
I don't have develop branch in front of me right this second but in
<IP_ADDRESS> it appears to be the case, the full URL is being used when you use
the sitemap picker
—
Reply to this email directly or view it on GitHub
https://github.com/concrete5/concrete5/issues/3013#issuecomment-146192904
.

Ah ya you're right, the database does have the place holders on it.

clearing the cache did not change the URL, tried that first, we had to go in and do the links over

There's definitely an issue here. It's been giving me some grief. When I move the contents of the database between a staging and local server (via SQL export/import), page links within content blocks break because the URLs are absolute.
-Steve

This is going to require some debugging then, as URLs are definitely not
intended to be absolute – they're intended to be replaced by placeholders
using the LinkAbstractor class.
On Wed, Oct 7, 2015 at 8:55 AM, Shotster<EMAIL_ADDRESS>wrote:

There's definitely an issue here. It's been giving me some grief. When I
move the contents of the database between a staging and local server (via
SQL export/import), page links within content blocks break because the URLs
are absolute.
-Steve
—
Reply to this email directly or view it on GitHub
https://github.com/concrete5/concrete5/issues/3013#issuecomment-146242429
.

@pvernaglia / @Shotster can either of you verify you're on the latest version of concrete5, and that you can replicate the bug there? If you can, exact steps to reproduce would be helpful. If I use the sitemap picker, and check my content data in the database it's definitely not appearing to be hard coded.

@joe-meyer, this issue does not seem to exist in the latest develop branch. However, there does seem to be a similar issue for Feature blocks. See https://github.com/concrete5/concrete5/issues/3070.

I don't think the site I had a problem with was started with the latest version but it was upgraded at some point, I don't remember the timing  of upgrades vs problems.  I'll be doing a similar job soon and see if it happens again.


How can I export the Wavesurfer-generated peaks? (rather than generate them server-side)

Wavesurfer.js version(s):
1.3.2
Browser version(s):
chrome 56
I have tried calling wavesurfer.exportImage() but it just tells me "this.drawer.getImage is not a function".
I have tried modifying the code / moving around some functions but I am getting similar undefined errors, and I feel like I shouldn't have to do much modifying to get this work.
Is there a way to easily grab the array of generated peaks so that I can export them to my server?  Am i initializing something incorrectly?
note i am having no problems loading audio, unloading it, playing it, etc...

also, using exportPCM works, but it gives me a long array full of null's and two 0's.

I have tried calling wavesurfer.exportImage() but it just tells me "this.drawer.getImage is not a function".

It probably uses the default MultiCanvas backend and that doesn't implement getImage yet :(
exportImage will give you a bitmap normally, and I'm not sure how to export the points to the server.

@thijstriemstra do you know of a way to use something other than MultiCanvas that does implement getImage?

ended up using backend.getPeaks()

I had the same problem. You can solve it by firing the function inside the wavesurfer.on('ready', function) event.


[release/1.7] Fix panic due to nil dereference cgroups v2

This PR ports #11069 to release/1.7 (with minor modification to use logrus).
It's not cherry picked because runtime/v2/runc/container.go was moved to cmd/containerd-shim-runc-v2/runc/container.go, and also 1.7 is still using logrus instead of containerd/log pkg.

/test pull-containerd-node-e2e-1-7


Implement auto merging in importers

For the Activity Insight, Pure, Law School, and the Web of Science importers we want to add our auto merging methods in after the grouping logic runs.  For the AI and Pure importers, running the AI-Pure auto merger right after the grouping could help to reduce the duplicate groups we get across AI and Pure.  We should be able to get these benefits with the way the grouping logic currently works, however, consider if it would be helpful to separate the doi grouping from the title/pub date grouping, and then run the AI-Pure auto merge right after the title/pub date grouping but before the doi grouping.
Ideally, we want to be able to auto merge the AI-Pure duplicates before we end up with a duplicate group containing more than 2 publications.  For the non AI/Pure importers, auto merging on import will keep our auto merging more up-to-date.  We probably don't need to run the AI-Pure auto merger for these.

It also probably doesn't matter when we run the DOI grouping.  We would just like to run the AI-Pure automerge right after the grouping is done on import, so we clean up duplicates faster.

Only do this for new imports.  We've experienced issues with the AI imports trying to do grouping logic on every import (it dramatically slows down the imports).  Not sure how much automerging will slow things down, but it's not necessary to do this on existing publications anyway, since our rake task will handle those.


extractPaper not working with cluttered background

I'm very impressed by jscanify and I really appreciate that you open sourced your work.
I have two examples where a cluttered background breaks the extractPaper logic to find the paper in the image.   I think this is an edge case (no pun intended) but perhaps these can help you future improve the algorithm.

Thanks for reaching out from Reddit!
BTW I just released a debugging tool that will make it much quicker to test the detection on specific images. The screenshots that I've added below were of the debugging tool.
For the second image, I'm not seeing any issue :). Here's a screenshot:

The lib does have some trouble detecting the first one though:

If I were to guess the problem there would be two:

the glare is extending the boundary of the contour
that, compounded with the fact that there are lots of other white objects in the frame

This is a tough problem to solve. If you're building a scanner app for the users, I would suggest you simply ask them to place the document in a better spot without glare and a solid background.
If you're using this to process these specific images, you may need to modify the thresholds a little bit.
If the path you choose to take is the latter, let me know and I'll guide you through it.

Thanks for you comments.  The debugger is fantastic!


Fixes Issue #66: ConcurrentTimedRotatingFileHandler rotating gzip fil…

…es fails
The gzip extension is already present in the return value of getFilesToDelete in Python's TimedRotatingFileHandler and doesn't needed to be added. This was causing it to fail to rollover properly in Timed mode.

Hi @Preston-Landers ,
I am trying to use this code from concurrent_log_handlers-0.9.25 version.I am unable to find the setup.py from the tar package and so my build creation is failing for python pakcages.
Where can I find the setup.py for same? Also is there some reason behind not including the setup.py and setup.cfg file in the tar ball?

@SukeerthiBJ that's a good question. I'm not sure. I'll check into it later. I never use the .tar.gz files myself, just the whl. If the whl file doesn't work for you for some reason, you could always Git clone the repo and install it from there running the setup.py script.


chore: bdd-spring-1573379059 to 0.0.1

chore: Promote bdd-spring-1573379059 to version 0.0.1

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approvers:
If they are not already assigned, you can assign the PR to them by writing /assign  in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


[Request] Possibility to mix light and dark theme

It would be nice if we can combine light and dark theme together.
For example light for file explorer and dark for code.
I think you cannot do that properly right now.
IMO both feels more separated.
It's just a thought.

This will be possible once we add more support to theme the workbench and you can also define the background color of the sidebar.

Is there any idea when the broad support for theme will be added ? is there any milestone, I'm curious about it.

We do support a lot of colors now in our theming story, however I see two missing things:

we have no support to set the foreground color individually per component, only across the workbench
our icons are not scoped to the background color per component but only globally (either dark or light theme)


Add ln to +sh-builtin-keywords

Removed duplicate "cat" and added missing "ln", which IMO is common enough to be included.

Might as well add the whole collection of gnu coreutils:
b2sum
base32
base64
basename
basenc
cat
chcon
chgrp
chmod
chown
chroot
cksum
comm
cp
csplit
cut
date
dd
df
dir
dircolors
dirname
du
echo
env
expand
expr
factor
false
fmt
fold
head
hostid
id
install
join
link
ln
logname
ls
md5sum
mkdir
mkfifo
mknod
mktemp
mv
nice
nl
nohup
nproc
numfmt
od
paste
pathchk
pinky
pr
printenv
printf
ptx
pwd
readlink
realpath
rm
rmdir
runcon
seq
sha1sum
sha224sum
sha256sum
sha384sum
sha512sum
shred
shuf
sleep
sort
split
stat
stdbuf
stty
sum
sync
tac
tail
tee
test
timeout
touch
tr
true
truncate
tsort
tty
uname
unexpand
uniq
unlink
users
vdir
wc
who
whoami
yes

Thanks again for your help! I originally wanted to keep this command list short, but I'll consider the rest of the GNU coreutils to it soon.

Maybe there is some way to infer it from $PATH?

There is:
(setq +sh-builtin-keywords
      (delete-dups
       (mapcar #'file-name-nondirectory
               (doom-files-in exec-path
                              :depth 0
                              :filter (lambda (f) (not (file-executable-p f)))))))

However, most PATHs contain thousands of binaries. That would be hell on Emacs' fontifier, so I'd rather restrict it to a much smaller subset of these utilities.

Ah okay. Didn't know that.


Add workflow builder package for v1alpha2 api

The workflow package provides functions for building an Argo Workflow
from a Kubebench Job v1alpha2.

This change is

/assign @Akado2009


RIOT supports some builds on stable

Issue with details: https://github.com/RIOT-OS/RIOT/issues/17526

Wrong target branch...


No implementation of map_cpus on non-Linux OSes

Currently, all other OSes revert to the old sync.RWMutex behaviour of sharing a single lock. Implementing map_cpus for other OSes in cpus_GOOS.go will fix this, and should be relatively straightforward if you have a machine running that OS.
Patches are welcome.

On BSD, I suspect something like dmesg | grep -i cpu can be used based on the example output given in this post.

For Windows (and potentially also a fallback for Linux/BSD) would be to use the affinity trick that drwmutex used previously. Example code for Windows can be seen here.

OpenBSD:
$ dmesg | grep -i cpu | grep apid | awk '{print $1" "$5}' | sort -u
cpu0 0
cpu1 1
cpu2 2
cpu3 3

I'd need confirmation that this command indeed gives the correct results on machines with more cores and on other BSDs though.

That has two main problems. The first is that it'll run forever if the default cpu() implementation is used (rather than cpu_amd64), since that one always returns 0, which means we'll never get NumCPUS() entries. The second is that it might take a very long time to run, especially on boxes with many cores, because there's no guarantee that the task will end up getting run on every CPU in any reasonable amount of time.

First point makes sense. For the second do you mean because of the change that cpu() returns the same cpu during the map_cpus()? Thanks!

Yes, exactly. cpu() may (will) end up returning the same value over and over and over, possibly forever. The correctness (or rather, termination) of the implementation depends entirely on the OS/Go's runtime constantly moving the task to different CPUs, but in practice schedulers tend to try to do exactly the opposite.


handle VGM commands 0x70..0x7F on GB

Some VGMs in the wild seem to rely on them. (Unfortunately, they also seem to play a bit slow...)

i added that in a bit different way, for the other targets as well. i think it is better just skip those? or maybe you are right and it should force the playback? also i added reporting of the unsupported commands to the error message.

I'm not sure, honestly; but if you've added this, I think the PR can be closed.

i have no compatible hardware to test on the game boy, so i can't tell.

i followed your approach, please check. i close this now. (i also unified a bit with the SMS/GG version)


Request to Add Issue Templates: Bug Report, Feature Request, and Styling

I propose adding three important issue templates to our project: Bug Report, Feature Request, and Styling.
These templates will enhance our project management and collaboration by providing standardized formats for reporting bugs, submitting feature requests, and addressing styling-related issues. With these templates, we can streamline communication, improve issue tracking, and ensure consistency in our development and design processes.
I would like to work on this issue under the GSSoC'23. @MonalikaPatnaik

adding issue template is already been assigned!
Thanks!

Can I close my issue since it has already been raised by someone else as you said?


Changed Spin Powerup

Does this now...
https://giphy.com/gifs/5C0Enzn0IBLPe4y5sQ
How to use it

If you have the spin powerup, hit the ball into the wall
It will start curving / wallbouncing towards the opposite ends
It stops spin-bouncing if it hits anything along the way

cc @secretrobotron - imma merge.

Renamed to wall-spin


Add Support for Preallocating Entry Slice in Batch Verifier

Before:
goos: darwin
goarch: arm64
pkg: github.com/hdevalence/ed25519consensus
BenchmarkCreateBatch/1-12  	 5439469	       214.8 ns/op	      96 B/op	       2 allocs/op
BenchmarkCreateBatch/8-12  	  704911	      1714 ns/op	    1104 B/op	      12 allocs/op
BenchmarkCreateBatch/64-12 	   90453	     13074 ns/op	    9680 B/op	      71 allocs/op
BenchmarkCreateBatch/1024-12         	    5697	    204843 ns/op	  155089 B/op	    1035 allocs/op
BenchmarkCreateBatch/4096-12         	    1401	    859972 ns/op	  892369 B/op	    4111 allocs/op
BenchmarkCreateBatch/16384-12        	     316	   3727873 ns/op	 5094872 B/op	   16405 allocs/op
PASS
ok  	github.com/hdevalence/ed25519consensus	10.901s

After (~71% reduction in bytes allocated, ~13% reduction in runtime on batch of 16384):
goos: darwin
goarch: arm64
pkg: github.com/hdevalence/ed25519consensus
BenchmarkCreatePreallocatedBatch/1-12  	 5431094	       213.9 ns/op	      96 B/op	       2 allocs/op
BenchmarkCreatePreallocatedBatch/8-12  	  762730	      1585 ns/op	     704 B/op	       9 allocs/op
BenchmarkCreatePreallocatedBatch/64-12 	   95875	     12557 ns/op	    6144 B/op	      65 allocs/op
BenchmarkCreatePreallocatedBatch/1024-12         	    5967	    199347 ns/op	   90112 B/op	    1025 allocs/op
BenchmarkCreatePreallocatedBatch/4096-12         	    1501	    805202 ns/op	  360449 B/op	    4097 allocs/op
BenchmarkCreatePreallocatedBatch/16384-12        	     374	   3237073 ns/op	 1441795 B/op	   16385 allocs/op
PASS
ok  	github.com/hdevalence/ed25519consensus	10.878s

Codecov Report
Attention: 3 lines in your changes are missing coverage. Please review.

Comparison is base (7232f88) 77.52% compared to head (f937c8c) 72.82%.

:exclamation: Current head f937c8c differs from pull request most recent head d8a485d. Consider uploading reports for the commit d8a485d to get more accurate results

Files
Patch %
Lines

batch.go
0.00%
3 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #16      +/-   ##
==========================================
- Coverage   77.52%   72.82%   -4.71%
==========================================
  Files           2        2
  Lines          89       92       +3
==========================================
- Hits           69       67       -2
- Misses         10       14       +4
- Partials       10       11       +1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Thank you for your patience! I pushed a few further changes that cut the allocations down and let the Add inputs avoid escaping to the heap as well.  I tried to tread as lightly as I could on my pass but maybe I should've been more ambitious lol...thanks for taking the time!

Only nit, I preferred to see the batch creation time separate from the batch verification time (just to get a better sense of where time was spent but eh).

I generally prefer to make benchmarks for whole operations that consumers are likely to perform discretely. That avoids over-indexing on a sub-benchmark once it's fast enough and it doesn't move the overall needle much. Different parts that contribute to the running time can be visualized with flame graphs. In this case, I don't think any user will do batch creation without batch verification, or reuse the batch for multiple verifications, so I think an overall benchmark is the right metric.


[ODS-4581] Merge Web.config and Web.release.config

@jamessmckay can you add the change to merge contents of Web.Release.config to Web.config and remove Web.Release.config

I don't see that \Ed-Fi-ODS-Implementation\Application\EdFi.Ods.WebApi\Web.Release.config into Web.config


Add aws-secretsmanager-r2dbc

Discussed in https://github.com/awspring/spring-cloud-aws/discussions/14

Originally posted by eddumelendez January  6, 2021
Similar to aws-secretsmanager-jdbc, a new project can be created to support secretsmanager for r2dbc.
Hi, i have idea for using r2dbc with AwsSecretsManager and shared it in attached discussion
NOTICE: example big just because i didn't access to internal spring-cloud-r2dbc code from my self program.
if you agree with this idea i can will try implement it self

At this stage there are no plans to add r2dbc support.

@IRuzhnikov I suggest creating this a separate OSS project. Ideally, AWS themselves could provide R2DBC compatible variant of Secrets Manager integration.


Unknown parameter env while using varlink

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description

When using varlink to create container, parameter env is not recognized.
Steps to reproduce the issue:

sudo varlink call unix:/run/podman/io.podman/io.podman.CreateContainer '{"create":{"image":"docker.io/library/fedora:30","env":{}}}'

Describe the results you received:
Call failed with error: org.varlink.service.InvalidParameter
{
  "parameter": "parameters"
}

Describe the results you expected:
A new container would be created, such as:
{
  "container": "7f2d8b78648be33b09510246eec50fc0fe918e646d2a3cfc323a4651a5ad3cc0"
}

Additional information you deem important (e.g. issue happens only occasionally):
This is regression. It works with podman-1.2.0 but not with podman-1.3.0 (of Fedora 30).
Omitting var works (well, "works", https://github.com/containers/libpod/issues/3183, but at least gets beyond parsing of the parameters), such as:
sudo varlink call unix:/run/podman/io.podman/io.podman.CreateContainer '{"create":{"image":"docker.io/library/fedora:30"}}'
Output of podman version:
Version:            1.3.1
RemoteAPI Version:  1
Go Version:         go1.12.2
OS/Arch:            linux/amd64

Output of podman info --debug:
debug:
  compiler: gc
  git commit: ""
  go version: go1.12.2
  podman version: 1.3.1
host:
  BuildahVersion: 1.8.2
  Conmon:
    package: podman-1.3.1-1.git7210727.fc30.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.12.0-dev, commit: c9a4c48d1bff85033b7fc9b62d25961dd5048689'
  Distribution:
    distribution: fedora
    version: "30"
  MemFree:<PHONE_NUMBER>
  MemTotal:<PHONE_NUMBER>6
  OCIRuntime:
    package: containerd.io-1.2.1-2.0.rc.0.1.fc29.x86_64
    path: /usr/sbin/runc
    version: |-
      runc version 1.0.0-rc5+dev
      commit: 10d38b660a77168360df3522881e2dc2be5056bd
      spec: 1.0.1-dev
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 4
  hostname: localhost.localdomain
  kernel: 5.0.11-300.fc30.x86_64
  os: linux
  rootless: true
  uptime: 5h 33m 46.8s (Approximately 0.21 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/mmarusak/.config/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /home/mmarusak/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 15
  RunRoot: /run/user/1000
  VolumePath: /home/mmarusak/.local/share/containers/storage/volumes

After discussion with @baude it was resolved that API changed. Now this command should be as:
sudo varlink call unix:/run/podman/io.podman/io.podman.CreateContainer '{"create":{"args":["docker.io/library/fedora:30"],"env":[]}}'
Small issue was that documentation was not correctly created and this information was not shown. See https://github.com/containers/libpod/pull/3186


Make the path change only temporary

Currently the batch script alters the user's path (Windows environment) variable permanently, which might not be best practice. Change this so it only changes that variable for time that the batch script is running.

Is it true that "conda for general use on your machine" after this change, as it states in the readme currently?


[AutoPR hanaonazure/resource-manager] Added HANA instance restart API

Created to sync https://github.com/Azure/azure-rest-api-specs/pull/3761

(message created by the CI based on PR content)

This PR has been merged into https://github.com/Azure/azure-sdk-for-python/pull/3251


Regularize crawl listing to match input for corpus

Using a consistent set of directories will make it clearer that the 2015-11 corpus was derived from the 2015-48 crawl.

The 2015-11 output was actually performed on CC-MAIN-2015-27, so we need to

rename output folders in the S3 bucked
update the readme appropriately


Your documentation is useless!

My Environment

ArangoDB Version:
Storage Engine:
Deployment Mode:
Deployment Strategy:
Configuration:
Infrastructure:
Operating System:
Total RAM in your machine:
Disks in use:
Used Package:

Component, Query & Data
Affected feature:

AQL query (if applicable):
AQL explain (if applicable):

Dataset:

Size of your Dataset on disk:

Replication Factor & Number of Shards (Cluster only):

Steps to reproduce

Problem:
Expected result:

Without any comment whats the problem this issue is at least as useless.
If you care to comment what your actual problem is - reopen the ticket.