Add checks for postprocessing operations

          Can we add in a check operation here that will validate expected keys are present?

Originally posted by @chrisiacovella in https://github.com/choderalab/modelforge/pull/169#discussion_r1661574225

I'm going to use this issue as a means to discuss ways we might modify the new approach to readout and post processing operation.  We obviously want to figure out a way to maintain flexibility, while not obfuscating, losing documentation, or introducing sources of error.
This is going to be long, but I'm trying to just get lots of thoughts on the paper for us to discuss.
Let us start by considering the processing step defined in the .toml file:
processing_operation = [
  {
   in = ["E_i",],
   out = "E_i",
   step = "normalization",
   mean = "E_i_mean",
   stddev = "E_i_mean"
 },
  {
   in = ["atomic_numbers", 'atomic_subsystem_indices' ],
   out = "ase",
   step = "calculate_ase"
  },

]

Right now, all of the .toml files look the same, the only differences is some have the calculate_ase step and some do not.  I think having the processing steps defined in the .toml file is very useful as it makes it very clear what is being done and easier to toggle steps on and off.  However, I'm not sure we need to explicitly define inputs and such.
I think it might be clearer to just provide bool triggers for functions, e.g.,:
[normalize_energy = True, calculate_ase = True]
Right now, I can't think of any cases, where we could use another key for the mean and stddev or any case where we would be applying this to anything but the atomic energy E_i. In reality, does the stats store anything else that could be used in place of the mean and stdev of the atomic energies...since this specifically references the stats information, it seems like giving it another key in there would cause a failure, since it is explicitly looking at that dictionary.  This might just introduce more chances for error, (note the committed files actually have an error where stddev is being set to the key for the mean E_i_mean).   Same thing with ASE, there isn't much need to require something different here.  If someone did modify the in it would be wrong (and wrong if the order is swapped).
And while we can argue a user might never want to modify this, presenting this with internal dictionary keys adds an extra level of possible confusion (especially since something like E_i_mean is not part of the outputs dictionary, but rather the dataset_stats, and is buried is not in models.py where this processing occurs).  Hiding all of internals seems like a better abstraction.
Especially for the processing part, I feel like if we want to do something different, we would still need to implement code.   The loops to process this only support normalization and calculate_ase; if we gave it some other operations it would do nothing at this point.
I do like the idea where it is clearly defined the name of the outputs, so we know how to access these parameters e.g.:
[{step = normalize_energy , output = 'E_i'}, {step =calculate_ase, output = ‘E_ase']
So basically, I’m not sure what we specifically gain in having this extra level of flexibility here, that could otherwise be accomplished with some more simple bools or switching toggles.
Looking at the readout,  right now there is:
readout_operation = [
  { step = "from_atom_to_molecule", mode = 'sum', in = 'E_i', index_key = 'atomic_subsystem_indices', out = 'E' },
  { step = "from_atom_to_molecule", mode = 'sum', in = 'ase', index_key = 'atomic_subsystem_indices', out = 'mse' },

]

The readout seems like it might actually benefit more from flexible operations, but I think it might be a lot clearer just to have many of the standard operations already coded in and just be able to toggle them on and off and set their output name.
Again, my concern here is that it could be confusing because one needs to know a lot about internal data representations and key names.   So following the approach above:
readout_operation = [
{step=‘energy_of_molecule’, out='E_molecule'}
{step=‘self_energy_of_molecule`, out='E_mse'}
]
although `E_mse` might not be the best choice, since don’t want anyone thinking mean squared error. We could probably be more descriptive here as well, say `E_molecule_self_energy` is clearer.
There is probably a reasonably small subset of operations we’d think to calculate for readout, so it shouldn’t be too hard to enumerate those.
I understand there could be some cases where we would want some “new” readouts, but I also think any custom readouts we implement would be of general interest, and thus worth simply adding into the code.
I think we could easy to allow custom operations here in a similar way you’ve done (easily toggled by adding in some key like “custom_readout’ to know to parse it), but I feel like trying to document fully how to do these custom function could be more problematic than just adding in a new function in the code.
I like the idea of making the functions general (e.g., like from_atom_to_molecule) and just having some specific wrappers for say energy or charge.
I suppose we could put these in as separate python file like “readout.py”  that contains all the possible readout function we have defined, so it is obvious what is available, how they are structured, where to add more, and have clear docstrings for each of these.
That last part, about docstrings, I think is a good thing.  Making this all flexible means we do not have a nice human readable docstrings of what was done.
Defining readout operations that depend upon other operations, e.g., like something that takes E_molecule, might be trickier with pre-defined functions, as we would need to know the dictionary key defined for the output, but that could be something like an optional argument to the definition.
readout_operation = [
{step=‘energy_of_molecule’, out='E_molecule'}
{step=‘self_energy_of_molecule`, out='E_mse'}
{step='sum_energy_and_self_energy', args=['E_molecule', 'E_mse'], out='E_summed'}
]
We could just have an internal dictionary that associates a user defined "out" to an internal name,  {'E_mse' = 'E_molecule_self_energy'}, so one does not need to define arguments, or we just do not allow user specified output names, and instead of predefined names.
I actually think I prefer predefined names as it makes it easier for us to document.

I think it is a great idea to refactor the postprocessing module. The PR that introduced this (PR 169) was mainly concerned with partitioning the postprocessing from the actual models and moving it outside the core model. I think it is clear that there is much room to improve this, make it more general, and safeguard the operations.
The current structure of a given model is as outlined here:
https://github.com/choderalab/modelforge/blob/ref-docs/docs/image/overview_network_v2.png
The postprocessing model gets a dictionary of per-atom properties as input and should perform operations on these to obtain model-specific (and user-specific) output properties. Obvious operations are reduction operations (obtaining per-molecule properties from per-atom properties). The reduction operations depend on the properties a neural network returns and the properties the user requests.
I like that version of the toml file (we might want to improve the variable names):

processing_operation=[normalize_energy = True, calculate_ase = True]

readout_operation = [
{step=‘energy_of_molecule’, out='E_molecule'}
{step=‘self_energy_of_molecule`, out='E_mse'}
]

it will hide much of the complexity. I think we can avoid the complexity for the args and dependent operations for now. Even for more complex operations (like calculating the electronic energies) we can predefined the keys for now.

I would say that if we are going to have dependent operations, we should just create new functions for those in the code that wraps the existing functions.
The variable out could be an array (with order specified by the function itself).
So if we had a dependent function total_molecule_energy that adds self energy to the energy of the molecule, it could look something like this:
 readout_operation = [
{step=`total_molecule_energy`, out=['E_molecule', 'E_mse', 'E_total']}
]

I would say ultimately, if we want more flexibility in our readout definitions, they should just be encoded as a user-defined python module (rather than trying to effectively implement our own scripting processor).

During our discussion yesterday, we made some progress on this topic. I will outline the structure of the toml file first and then discuss the actual implementation:
[potential]
model_name = "SchNet"

[potential.potential_parameter]

max_Z = 101
number_of_atom_features = 32
number_of_radial_basis_functions = 20
cutoff = "5.0 angstrom"
number_of_interaction_modules = 3
number_of_filters = 32
shared_interactions = false

[postprocessing]
[energy]
normalize = True
from_atom_to_molecule_reduction = True

[partial_charge]
equalize = True
from_atom_to_molecule_reduction = True

We pass the dictionary directly to EnergyPostprocessing and PartialChargePostprocessing classes that are then responsible for performing the postprocessing/reduction operations. This is straightforward and should minimize possible errors.

It would be good to define in the toml file what properties should be calculated in the model and/or expected in training.
We can either define this in the model toml file and make sure that the model can calculate the requested properties at runtime, or we can calculate all possible properties and only train on a (sub)set of them and define the properties in the training toml file (where we need to initialize loss functions for them). I am heavily leaning towards the first option.
A possible toml structure would look like this:

[potential]
model_name = "SchNet"
calculate = ['energy', 'force', 'partial_charge']

[potential.potential_parameter]

max_Z = 101
number_of_atom_features = 32
number_of_radial_basis_functions = 20
cutoff = "5.0 angstrom"
number_of_interaction_modules = 3
number_of_filters = 32
shared_interactions = false

[potential.postprocessing]
[potential.postprocessing.energy]
normalize = true
from_atom_to_molecule_reduction = true

[potential.postprocessing.partial_charge]
equalize = true
from_atom_to_molecule_reduction = true

The toml file for training must match this then (relevant changes are in [training.training_parameter.loss_parameter]:
[training]
nr_of_epochs = 5
save_dir = "test"
experiment_name = "your_experiment_name"
accelerator = "cpu"
num_nodes = 1
devices = 1                              # [0,1,2,3]
remove_self_energies = true
batch_size = 128

[training.training_parameter]
lr = 1e-3

[training.training_parameter.lr_scheduler_config]
frequency = 1
mode = "min"
factor = 0.1
patience = 10
cooldown = 5
min_lr = 1e-8
threshold = 0.1
threshold_mode = "abs"
monitor = "val/energy/rmse"
interval = "epoch"

[training.training_parameter.loss_parameter]
loss_property = ['energy',  'force', 'charge']
weight = [0.998, 0.001, 0.001]

[training.early_stopping]
verbose = true
monitor = "val/energy/rmse"
min_delta = 0.001
patience = 50

This looks great.
Would it be better to use a dictionary for the loss_parameter definition?
[training.training_parameter.loss_parameter]
loss_property = {'energy': 0.998,  'force': 0.001, 'charge': 0.001}

I don't think it matters too much, but this might remove needing to check if the length of loss_property and weight are the same (ensure we do provide a weight for each).


Bug with Multiple QEE genome templates

Describe the bug
Created QEE genome sequence templates (but only multiples) throw errors when merging or when opening trader inventories about neither having IComparable implemented.  Breaks the entire Hauling Job and breaks all trading interactions.
Expected behavior
No errors and trade windows try to open and/or when things get hauled
Here's my HugsLog for proof and any details:
https://gist.github.com/6a2ea712a01ef124a4d523969b1747b8

Changing OgreStack to ignore the Genome Template has allowed Hauling jobs to mostly resume, but the Trade window is still throwing errors.

I suspect this is a mod conflict somewhere, as this bug does not appear with just vanilla + QEE enabled. I'm going to close this issue for now, as OgreStack hasn't been updated to Rimworld v1.2 and it's not clear what mod from your list is causing this behavior.

Yeah, this was before 1.2 was available and may have since been resolved.
I have moved on from Rimworld at the moment so I cannot give more info.
<EMAIL_ADDRESS>On Wed, Aug 12, 2020 at 8:04 AM KongMD<EMAIL_ADDRESS>wrote:

Closed #107
https://github.com/KongMD-Steam/QuestionableEthicsEnhanced/issues/107.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/KongMD-Steam/QuestionableEthicsEnhanced/issues/107#event-3647350158,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AKP5YPHIMDNUPUOMSGNLU4TSAKALFANCNFSM4M5A43GQ
.


Focus/Blur & Open/Close

Hi, is possible have a way to open the select when it gets the focus, and close it when it's blur.
I tried with timeouts but not lucky. It seems there are some "colition" between focus and click andkeydown events.
Also, in IE11 is not working as expected. When the select is open, you press shift + tab and nothing happens.

You could try to call the 'open' method when the focus output event is triggered, but I haven't tried that myself.
For me shift+tab seems to work as expected in IE11.


chore: update dependencies

Updated development dependencies to fix dependency issues identified by npm audit

Pull Request Test Coverage Report for Build 196

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build 193:
0.0%

Covered Lines:
129

Relevant Lines:
129

💛  - Coveralls


⚠ ClamAV has degraded performance

In 9256eab, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 452 ms

Resolved: ClamAV performance has improved in 9acb195.


Website linux download link not working

Hello, sorry if it's not the right location to report this, but the linux download file is missing on the website:

#799 😅


Retry different disk when bootstrap fails in Full auto mode

Retry different disk when bootstrap fails

Codecov Report
Attention: 28 lines in your changes are missing coverage. Please review.

Comparison is base (be37869) 69.18% compared to head (09fbe46) 19.54%.
Report is 8 commits behind head on master.

Files
Patch %
Lines

...in/java/com/github/ambry/store/StorageManager.java
0.00%
17 Missing :warning:

...m/github/ambry/clustermap/HelixClusterManager.java
0.00%
11 Missing :warning:

Additional details and impacted files
@@              Coverage Diff              @@
##             master    #2659       +/-   ##
=============================================
- Coverage     69.18%   19.54%   -49.65%
+ Complexity    11010     2788     -8222
=============================================
  Files           806      809        +3
  Lines         65526    65560       +34
  Branches       8006     8001        -5
=============================================
- Hits          45337    12813    -32524
- Misses        17620    51566    +33946
+ Partials       2569     1181     -1388

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


[Q&A]: Open source version of drizzle-kit

Describe want to want
We are a startup evaluating to use drizzle in our project. Evaluating in depth our team identified that drizzle-kit is not open source, unlike https://kysely.dev or prisma. Please clarify the roadmap of drizzle-kit and open source status of it. Most of drizzle packages has no clear licensing. We can see license as MIT in npm, but can't find the source code, which seems misleading.

Hi @thenewyork!
Thank you for your question
First of all, drizzle-kit is going to be open source, and it will happen very soon.
There are several reasons why we haven't opened it yet. Before gaining popularity, we were using drizzle-kit locally for our projects, and the codebase was not developed in an open-source manner. There are sensitive files and database structures from our projects that should not be revealed to the public. We also have many comments written in Ukrainian that may not be understandable to the community.
Furthermore, after encountering numerous people attempting to use drizzle, we focused primarily on fixing and improving the ORM part and didn't have enough time to clean everything up and make it accessible to everyone.
This is why the MIT license was left on npm, which I completely agree is not ideal. I appreciate you pointing out these issues and providing explanations.
These are essentially all the reasons for keeping it closed. I intend to start the cleanup process at the end of this week and then force push the entire codebase (due to sensitive parts of the code in the Git history).
Feel free to reach out to me via DM if you have more questions about drizzle-kit and if getting early access to the codebase is important for your company.

Thanks you @AndriiSherman for clarifying. Closing this issue.

These are essentially all the reasons for keeping it closed. I intend to start the cleanup process at the end of this week and then force push the entire codebase (due to sensitive parts of the code in the Git history).

Hi Andrii, what is the status on this?

@AndriiSherman - would it make sense to reopen this issue to indicate that it's still on the roadmap?

I'm going to bump this thread, because I suspect this question must've been skipped.
Personally I completely agree that for us, this issue should be re-opened.

@AndriiSherman - would it make sense to reopen this issue to indicate that it's still on the roadmap?

I just have spend a while trying to debug the drizzle-kit (why it doesn't support ESM and TypeScript), then I realised I couldn't access the source code. I was surprised, as I've been misled by the statements "fully open-source".
I still love Drizzle and what you did for us ❤ , so please no hard feelings on this bump, which brings nothing to the table. I would love to help contribute to the issues I have now.

The linked project items says that this was planned for Q4 of last year, or more precise December 20th. That obviously has not happened yet. Would it be possible to get more insight into this process and/or update the estimates?

The linked project item still mentions December 20th 2023 (the dates load after a few seconds, after opening the item), but it has no associated github issue, and is not placed on the main board.

https://github.com/drizzle-team/drizzle-kit-mirror/issues/501
How do deal with an eventually trivial situation like this meanwhile ?
You're not the first one who creates scheme updates.  I had it written for Haxe and PHP some years ago.
Which is why I felt int love with library instantly :-)

I think this can now be closed with #2711


Expand the range of supported setuptools.

We now just exclude the island where setuptools ran a disruptive
de-vendoring experiment.

I manually tested pex --pre with setuptools versions inside and outside the excluded range, only getting failures in the excluded range. I used https://github.com/pypa/setuptools/blob/master/CHANGES.rst to find the exclusion range in the first place.
The motivation here is to let Pants use a newer version of setuptools so that it can run under python 3.7. See: https://github.com/pantsbuild/pants/issues/6363


Trailing period (.) in record value makes the record always act like it's out-of-date

CNAME records (and possibly other records) whose value ends with a . will always show as "will be updated in-place," even if no update is actually necessary.
This appears to be because Cloudflare strips the . when the record is added, which causes it to always look out-of-sync with the Terraform configuration.
Terraform Version
Terraform v0.13.3
Affected Resource(s)

cloudflare_record

Terraform Configuration Files
terraform {
  required_providers {
    cloudflare = {
      source  = "cloudflare/cloudflare"
      version = "~> 2.11"
    }
  }
}

provider "cloudflare" {
  api_token = "[redacted]"
}

resource "cloudflare_zone" "my_zone" {
  zone = "example.com"
  paused = "true"
}

resource "cloudflare_record" "my_cname" {
  zone_id = cloudflare_zone.my_zone.id
  type = "CNAME"
  name = "foo"
  value = "example.org."
}

Debug Output
https://gist.github.com/SyntaxColoring/291d72cb0f8f1058020dbe06d9ca3fde
Expected Behavior
If you run two terraform applys in a row, the second one should be a no-op, showing no changes.
Actual Behavior
The second terraform apply, and all terraform applys thereafter, show this:
  ~ resource "cloudflare_record" "my_cname" {
        created_on  = "2020-10-10T07:48:44.035567Z"
        data        = {}
        hostname    = "foo.example.com"
        id          = "d19f451a92593e19c771b2266f022abd"
        metadata    = {
            "auto_added"             = "false"
            "managed_by_apps"        = "false"
            "managed_by_argo_tunnel" = "false"
            "source"                 = "primary"
        }
        modified_on = "2020-10-10T07:48:44.035567Z"
        name        = "foo"
        priority    = 0
        proxiable   = true
        proxied     = false
        ttl         = 1
        type        = "CNAME"
      ~ value       = "example.org" -> "example.org."
        zone_id     = "c2d9cadc42c798f9bf3ef0158cdbdab6"
    }

Steps to Reproduce

terraform apply and confirm.
terraform apply again.

Important Factoids
None.
References
None.

Thanks for the report!
This is a duplicate of #184 and I’ve expanded upon the reasoning a bit in #154
Do let me know if you have any thoughts though!


Fixed response format on verification functions

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 4 (0.0%)  changed or added relevant lines in 4 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 52.158%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/controllers/internal/didcomm_generate_plaintext.rs
0
1
0.0%

src/controllers/internal/didcomm_verify_encrypted.rs
0
1
0.0%

src/controllers/internal/didcomm_verify_plaintext.rs
0
1
0.0%

src/controllers/internal/didcomm_verify_signed.rs
0
1
0.0%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
1668

Relevant Lines:
3198

💛  - Coveralls


Experiment with caching */target/

We may be able to aggressively cache target/ in Travis for PR builds only (cleaning it out on master).  This should, in theory, allow us to have much faster (incremental) PR builds while still maintaining reproducible and trustworthy master builds.

Done a while ago.  Ticket never got closed.

Closing this ticket, since this experiment was done and implemented and we have feedback from Quasar developers around feedback from Travis being a "lot more instantaneous" now. Basically, there is now intelligence to detect if PR or Master Build. If PR build, then we cache target directory, If MASTER build, then we go for safety first to hedge against any compiler issues and the possibility of a corrupt artifact.


add import/export genesis e2e

Summary
Add an e2e test for init and export genesis.
Proposal

startup chains
create channel/send packets
halt chain, export genesis, write file
restart chain
send packet over existing channel

For Admin Use

[ ] Not duplicate issue
[ ] Appropriate labels applied
[ ] Appropriate contributors tagged/assigned

can I handle this ser?

Hi! Yes you can! That would be wonderful!
I've started looking into this so here's some more information:
I think we should add the test into the e2e/tests/upgrades directory under a genesis_test.go file. We should create a new TestGenesisTestSuite as the upgrades one is skipped on regular pr's to ibc-go.
func TestGenesisTestSuite(t *testing.T) {
    suite.Run(t, new(GenesisTestSuite))
}

type GenesisTestSuite struct {
    testsuite.E2ETestSuite
}

We will need to make modifications to interchain test to complete this issue. I've opened this issue https://github.com/strangelove-ventures/interchaintest/issues/418. You are welcome to take that on if you'd like, but I had planned to give it a shot tomorrow on my fork as I know the changes I'd like to do so I don't think it'll take me very long.
The changes will add a function to the CosmosChain struct which will allow us to set the halt-height parameter in the app.toml. This would be the function we set on the cosmos chains we create. I think we could set the function by adding it as a field to testconfig.ChainOptionConfiguration and then calling GetChains(opts) before SetupChainsRelayersAndChannel
func setHaltHeight(node *cosmos.ChainNode) error {
    appToml := make(testutil.Toml)

    appToml["halt-height"] = haltHeight

    return testutil.ModifyTomlConfigFile(
        ctx,
        node.logger(),
        node.DockerClient,
        node.TestName,
        node.VolumeName,
        "config/app.toml",
        appToml,
    )
}

see similar code
For the genesis import/export test we would set the halt height to X so all nodes would halt at X. Once all the nodes are halted, we should export state from each validator. We can use exec cmd <bin> export. I'm not sure if there's an rpc we could call instead?
After exporting. Stop all nodes (remove containers). Write the exported genesis file to the genesis.json file location (replacing the old), have the unsafe-reset-all be ran on each node. Start all nodes.
Let me know if you have any questions I can answer. There should be some reusable logic in the upgrade_test.go

Hi again @ThanhNhann. A couple updates.
You can see here on how to modify the halt height config.
I tested this manually with @chatton, we ran into one issue on updating the client post export/restart. Since the data is wiped on the chain, relayers may have difficulty getting a trusted header to submit for a client update. Here's an issue I wrote on hermes a while back. I'm not sure if the same issue exists on the go relayer, but if it does we can open an issue there. I'd be okay with merging a test which simply sends a second packet, but doesn't attempt to relay the second packet. We can address that second functionality once relayers support genesis restarts as a valid upgrade

thank @colin-axner for the update, love it

hey @colin-axner, it seems like I also face the problem as you said - the data is wiped on the chain, can you plz check my draft pull request to see if it had set up as you above


[BUG] Map - Zooming makes portions of map disappear

Describe the bug
Zooming in and out of the map in both MAP and SATELLITE view makes sections of the map disappear.
To Reproduce
Steps to reproduce the behaviour:

Create an account, and land of the map screen
Zoom in and out of the map in both MAP and SATELLITE views
Notice that certain sections of the map disappear.
(This is not permanent. Often clicking on the map or zooming again will reset everything)

Expected behavior
When a user is zooming, the full map should load and be visible.
Screenshots
Screen recording here: https://drive.google.com/file/d/1j0J7QRvDwLqMt8j0B8R-dtT1cgGnV20G/view?usp=drivesdk
Smartphone (please complete the following information):
iPhone 7s
Chrome Mobile Browser

ZenHub Information
This information is updated automatically. To modify it, please use ZenHub.

This pretty much looks like the map is just loading slowly.
So when you zoom out, new areas of the map needs to load, this behavior should be observable in Google Maps as well.
It's the same case with Zooming in as well, because when you zoom in, more details needs to be loaded to show more content in the screen.
This is probably due to one of:

slow network
slow device
just random unfortunate timing 😅

@CorporateInvesther @ashwinkjoseph @sabind Can we close this issue?

Yeah seems like its not being reproduced today! Closing


SSLH GUI Update

Important notices
Before you add a new report, we ask you kindly to acknowledge the following:

[x] I have read the contributing guide lines at https://github.com/opnsense/plugins/blob/master/CONTRIBUTING.md
[x] I have searched the existing issues, open and closed, and I'm convinced that mine is new.
[x] When the request is meant for an existing plugin, I've added its name to the title.

Is your feature request related to a problem? Please describe.
Freshports is pulling the latest version of SSLH (2.1.2) in and it should be available soon.  That version adds some more configuration options, including UDP.  The interface doesn't currently support those.
https://www.freshports.org/net/sslh/
Describe the solution you'd like
Upgrade the GUI to expose the new options and/or the ability to have a custom condition or edit the cfg directly.  The new version allows UDP, custom regex based filters, etc...  See the example config at
https://github.com/yrutschle/sslh/blob/master/test.cfg
And
https://github.com/yrutschle/sslh/blob/master/udp.cfg
For just UDP.
Describe alternatives you've considered
manually editing the config file in the terminal, but it's possible to accidentally override it.
Additional context
Add any other context or screenshots about the feature request here.

Haven't seen the maintainer in a long time. You will have more luck contributing the changes yourself.

👍.


Fix typo in rhosak-devsandbox-quarkus-bind-cli-toolscontainer-quickstart.yaml

Fix typo.

FYI @sterobin.

@DuncanDoyle This can be applied to Sandbox if not there.

Thanks so much @merlante !

This have been already applied (but files were not updated). Closing


Error with nper function

I am using the nper function and works fine most of the times.
But for this specific values:
Exonio.nper(0.0575 / 12.0, -76.0, 943183.21)
I am having an error:
Math::DomainError Exception: Numerical argument is out of domain - "log"
Any idea about what can be the problem? Thanks.

@cleicar and I reviewed this issue and it seems that problem is with the data that we are providing.
The payment amount should not be lower than the interest amount.
Here is an idea: Maybe we should raise an error with more information about this? It could avoid future issues reports.


CPU Balloons on Endpoint Reuse

I realise this is a rather generic title and I'm am still investigating it - however:
I create a single Endpoint.
For every test iteration:

I have a new Streamer which I make a StreamerSession from (with audio going through).
I hook the StreamerSession up to a new Transport from the Endpoint from above. (transport connected)
Wait 2 seconds
I tear down the Streamer, StreamerSession and Transport

I repeat this iteration many times over the course of an hour or more and I see that the CPU is getting extremely high (up to 50% so far) for just the single connection. To make things more interesting if I stop everything there is no CPU load. As soon as I add another iteration (connection) the CPU goes back up to the same level for the connections duration.
Is there a list being iterated over in the C++ code thats not being cleared? Possibly for routing packets? Or stream information. Maybe you can point me in the right direction.
If I recreate the Endpoint for every new iteration/connection the CPU stays within 3-4% (which is about the level the code starts at before ballooning over time). So it's seemingly something in the Endpoint not being cleaned/cleared. The fact that it applies for the duration of the connection routing I presume means its somewhere near stream/socket level?
Recreating the endpoint certainly feels like the wrong thing to do although it does work around my issue...

Could you use https://github.com/brendangregg/FlameGraph to get cpu usage graph and check where is the cpu being used?
# perf record -F 99 -p 181 -g -- sleep 60
# perf script > out.perf
# ./stackcollapse-perf.pl out.perf > out.folded
# ./flamegraph.pl out.folded > kernel.svg

Sure, I'll give it a shot

2 minutes at 99Hz.
https://drive.google.com/file/d/1oosm8ahE8fx_AnNHqaQpRWufqD0DO4xl/view?usp=sharing

The cpu is been used on the event loop. The StreamerSessions have two, but they are stopped on RTPTransport::Stop() called by  RTPTransport::End()  which is called by js stop().
Can you enable logs and confirm if you see this trace?
https://github.com/medooze/media-server/blob/master/src/RTPTransport.cpp#L613

Can confirm I can see that trace yes

the only other objects that uses even loops are the EmulatedTransport, Transport and the MP4 player, are you using any of them too?

Just Endpoint

are you stopping them properly?

They don't get stopped - they are reused for the transports and an remain for the life of the application

Is correct practice in your view to have an endpoint per StreamerSession and transport grouping then?

How were you reusing the endpoint? can you put a code snippet here?
Attaching transport belonging to different Endpoints is not as efficient as attaching between transports of the same Endpoint (due to inter thread locking). But adding too many transports on an Endpoint can cause load to not being distributed evenly between cpu processors.

The transport is only attaching to the StreamerSession (which doesn't have an Endpoint). All transports are being made on the same endpoint but each has its own StreamerSession.
Everything has stop() called except the endpoint because I was hoping to reuse that as I only create as many endpoints as I have cores on the box.
I'll try to get an example for you since I can't paste my project.

I haven't easily managed to reproduce this at present and have worked around the issue through recreation of the Endpoint.


picom-next: init at unstable-2021-10-31

Motivation for this change
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
21.11 Release Notes (or backporting 21.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

https://github.com/NixOS/nixpkgs/pull/144696/commits/4b5557d9ae09f5a70388f94d5e8dabc99b81b78a
Why picom-tryone :thinking: ?

4b5557d
Why picom-tryone thinking ?

Ugh did I get the name wrong

4b5557d
Why picom-tryone instead of picom-next thinking ?

Renamed the commit

@GKasparov Thanks for your contribution!


Clarify extensions should live in specified namespaces

I noticed our newly introduced FFT stubs live in fourier_transform_functions.py, which might implicitly communicate those extensions should live in the xp.fourier_transform_functions namespace (xp being the top-level Array API namespace), where presumably we want adopters to use xp.fft like say NumPy has it.
So that can be renamed, but this brings up that there's no explicit clarification in the spec that the linalg and fft extensions should indeed live in those respective namespaces. I think a note could live in all extension pages (currently just Linear Algebra and Fourier transform Function) explaining that the respective functions should be present in these specific namespaces.

In todays meeting we were in agreement for both points (fourier_transform_functions.py -> fft.py and the notes), so I'll submit a PR.

I completely missed #470, which aims to do the latter point (note that these extensions must live in specified namespaces).

Fixed by gh-470


Add help command

Displays the list of commands and what they do

added in https://github.com/inttter/mdbadges-cli/commit/82f356ea248ac9ef9fe8d34ffa93810459025fc5


plot_vars does not show value edges if var has dist

Currently, the plotting function plot_vars does not show edges that determine the value of a node if that node has a distribution.
This potential fix draws edges in red if they influence value as well as distribution.
def _draw_edges(graph, axis, pos, is_var):
    """Adds edges to the figure."""

    edges = list(graph.edges)

    if is_var:
        dist_edges = []
        value_edges = []

        for edge in edges:

            # find distribution edges
            if edge[1].has_dist:
                edge_0_output_nodes = set(edge[0].all_output_nodes())
                edge_0_nodes = edge[0].nodes
                edge_1_input_nodes = set(edge[1].dist_node.all_input_nodes())

                if bool(edge_0_output_nodes.union(edge_0_nodes) & edge_1_input_nodes):
                    dist_edges.append(edge)

            # find value edges
            edge_0_output_nodes = set(edge[0].all_output_nodes())
            edge_0_nodes = edge[0].nodes
            edge_1_input_nodes = set(edge[1].value_node.all_input_nodes())

            if bool(edge_0_output_nodes.union(edge_0_nodes) & edge_1_input_nodes):
                value_edges.append(edge)

        edges_in_both = set(dist_edges) & set(value_edges)
        dist_edges = set(dist_edges) - edges_in_both
        value_edges = set(value_edges) - edges_in_both

        # assigns value_edges to edges to make it comparible with is_var=False
        edges = value_edges

        nx.draw_networkx_edges(
            graph,
            pos,
            edgelist=edges_in_both,
            edge_color="#FF0000",
            arrows=True,
            ax=axis,
            node_size=500,
        )

        nx.draw_networkx_edges(
            graph,
            pos,
            edgelist=dist_edges,
            edge_color="#aaaaaa",
            arrows=True,
            ax=axis,
            node_size=500,
        )

    nx.draw_networkx_edges(
        graph,
        pos,
        edgelist=edges,
        edge_color="#111111",
        arrows=True,
        ax=axis,
        node_size=500,
    )

I'm not sure if I understand the scenario. When would a node's value be a function of another node, if the first node has a distribution?

for example when using batching. Imagine three nodes:

node a hold the batch index
node b holds the variable
node c is a calculator selecting from a the elements indicated by b: a[b]
node c is also associated with a distribution for a[b]. this distribution may depend on a, b, or other nodes.

See #231 for an example.


[RF] Use niworkflows upstreamed interfaces and utilities

Left reports generation for another PR.
Updating with SignalExtraction as well (poldracklab/niworkflows#265)

Still need to investigate why ds054 does not generate the errored report anymore and why masks don't show green contours at all (regression tests)

@effigies I'll fix the indentation problem
ds054 issue is now resolved.
Okay, I run the test_pytest locally and the masks being generated are exactly the same as the originals, and for that reason the green contour is invisible. To double-check, I opened one report with inkscape and removed layers until I got to the green contour. They are there, right below the red contours.
So, in principle, this PR is ready to be merged. Thumbs-up?

I'll try to have a look through tonight.

Good to go?

Oh, sorry. Let me have a look through right now.


GradeBlock() unit test

As requested by @PaulBernier in https://github.com/pegnet/pegnet/issues/51
Creation of these tests is a bit painstaking but I tried to include as many edge cases as I could. I simplified the input using the same currency value for every single currency, since this is just testing GradeBlock which is about sorting, not Avg() or CalculateGrade().
The tests require at least 10 gradeEntry instances per test. e1 creates an OPR with difficulty, currencyValue, and auto-generates miner IDs starting at "1". e2 adds an additional parameter to set custom miner IDs, to use for duplicate testing further down the road, or non-sequential input.

Man!  Wonderful work!  Painstaking is right!

Brilliant, thank you @WhoSoup!


Addition of configuring Databricks to use a proxy

Many organizations for all outbound, internet-bound traffic to use a proxy server. Can Databricks be configured (either with environment variables or init scripts) to use a proxy? Can just pip install, R, Maven be specifically configured that they use a proxy when connecting with external repositories but no other outbound traffic use a proxy?
It would be very useful functionality to be able to route repository traffic through a proxy which can perform malware and virus inspection.
Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 93bd5c16-e0ec-5064-9bc0-1c527c328a7e
Version Independent ID: 5342d01e-c2da-f7a8-d22e-86995a17af79
Content: Deploy Azure Databricks in your Azure virtual network (VNet injection) - Azure Databricks
Content Source: databricks/administration-guide/cloud-configurations/azure/vnet-inject.md
Service: azure-databricks
GitHub Login: @mamccrea
Microsoft Alias: mamccrea

Hi @BDunTG - thanks for your feedback!
For routing and whitelisting all traffic through a firewall or proxy, please reference this blog.
I am working to get additional details for routing only library repo traffic through a proxy.

It may be possible for Python and R libraries as those are installed from the data plane, but not for Maven libraries as those are directly installed from the control plane.
For python, it’s possible to configure a proxy at the pip config level which could be done via init scripts - but the proxy server must be reachable from Databricks subnets (if proxy server is in cloud VNET then it’s easy, but if it’s on-premises that would require routing based changes too).
For R, you might be able to use the CRAN library API and UI that allows specifying repository URL.
There’s a generic solution like setting http_proxy and https_proxy via init script which should work for both Python and R. But it will impact all HTTP(S) traffic from the clusters including traffic to Azure Data Services. So even though it’s an easier one, but it has wider impact and not just for installing libraries.

I will now proceed to close this issue. If you have any additional questions, please reopen the issue.
#please-close


FQN needed to Any

this is going to sound really weird, but I am experimenting with flag -Yno-imports (stop laughing at the back!) and my (drone-dynamic-agents) tests fail with
[error] Handler not found to transform `logic.Deps.Op` to `cats.Id`

unless I import scala.Any. It would be good if everything was FQN in the macro code.

Makes sense, we'll take a look, thanks @fommil for reporting this. We are also interested in supporting alternative predefs.

sorry for being this guy

and AnyRef by the looks of it


 Documenting this so 1.) newcomers can see what's up, and 2.) serve as additional reference for future PR's.

[ ] Guides at https://vx-demo.now.sh/ links to outdated articles at https://medium.com/vx-code which does not work with the current version anymore.
[ ] https://vx-demo.now.sh/areas

[ ] Embedded code returns an error of undefined margin object
[ ] Could be improved by providing example with react-hooks + proper prop-type checks
[ ] Error: <line> attribute y2: Expected length, "NaN".

@davalapar gallery example source lives here: https://github.com/hshoff/vx/tree/master/packages/vx-demo/src/components/tiles
We've had some trouble syncing the source with the example code on the gallery site. As mentioned in https://github.com/hshoff/vx/issues/601#issuecomment-578267408 this will get addressed in the coming months with v1.
Going to close this issue for now as @williaster and I are tracking this already and working on improvements. Happy to re-open or feel to create other issues for improvements you'd like to see.

FYI duplicate demo + demo source code was fixed with #637, and the code sandox refactor is being tracked in #624


Reply early (rebased, fixed and refactored)

Rebased #52 against #53. Fixed #52 there.

Changes look really good! :)


Make algorithm capable of forks and preventing enemy forks

Part 4(end) of #20
If the algorithm sees no winning opportunities for either side, it should attempt to construct a fork whereby it will guarantee a win for itself.
Otherwise it should prevent the opponent from creating a similar condition.

There is extra depth to this problem.
I thought that the moves x:4 o:2? x:1 o:9 was the only condition xox was failing to exploit a fork, but I failed to anticipate board transpositions: x:4 o:9 x:1 o:2? transposes into the same position with the error to exploit one pile deeper. By nature of the rest of the algorithm, the game already forks some times. This will just fill out the other cases.
I'll need to assure these moves:

[ ] x:4 o:2? x:1 o:9 --> x:7!
[ ] x:4 o:9 x:1 o:2? --> x:7!
[ ] x:4 o:7 x:3 o:2? --> x:9!
[ ] x:4 o:3 x:7 o:4? --> x:9!
[ ] x:4 o:1 x:9 o:6? --> x:7!


Automated cherry pick of #4801: kvm elb listener update bugfix

Cherry pick of #4801 on release/2.12.
#4801: kvm elb listener update bugfix

Codecov Report

:exclamation: No coverage uploaded for pull request base (release/2.12@3c442a2). Click here to learn what that means.
The diff coverage is 0%.

@@              Coverage Diff               @@
##             release/2.12   #4803   +/-   ##
==============================================
  Coverage                ?   6.86%
==============================================
  Files                   ?     502
  Lines                   ?   85645
  Branches                ?       0
==============================================
  Hits                    ?    5880
  Misses                  ?   79270
  Partials                ?     495

Flag
Coverage Δ

#aFlag
6.86% <0%> (?)

Impacted Files
Coverage Δ

pkg/compute/models/loadbalancerlisteners.go
1.48% <0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3c442a2...713274c. Read the comment docs.

/lgtm

/approve


Faster introspection

Experimental change for faster introspection by avoiding some of the GraphQL Java execution overhead.
Old
    // Benchmark                                    Mode  Cnt    Score   Error  Units
    // IntrospectionBenchmarkJava.benchMarkAvgTime  avgt   20  134.819 ± 5.109  ms/op

New
    // Benchmark                                    Mode  Cnt   Score   Error  Units
    // IntrospectionBenchmarkJava.benchMarkAvgTime  avgt   20  30.656 ± 0.949  ms/op

If we like this approach I'll clean up the code and add an execution hint/feature flag.

We ended up caching this inside the Gateway.


Code organization

I was thinking about a better way to organize the growing main file and tests. It's getting bigger and bigger after each of your great contributions, which is great. But it's also getting harder and harder to read and follow up.
I was thinking about splitting each file in multiples one but I don't know yet the splitting logic. Do you have any idea ?

I agree that this is a must. Currently I'm not aware of any way to split the class function definitions into several files (I tried a basic test but that didn't work out). I've posted on Gitter but not gotten any replies, I'll perhaps try StackOverflow once I'm done with the categories.
My current thought is that we should move all helper functions to a utils-file, e.g. the entire utils section, the _to_html and my new __to_string could probably be moved. There is also some double documentation at the moment - I'll post a new issue regarding this.

How to split class into several files
I got a suggestion from @andreaskoepf to simply use the return value. Here's an example of an approach we could use:
main.lua
test = require("test")
assert(loadfile("test2.lua"))(test)

local a = test.new()
a:a()
a:b()
print("end")

The test.lua file
Here we create the class and return it to the main for passing onto the subsections:
test = torch.class('test')
function test:__init()
  self.data = {}
end

function test:a()
  print("a")
end

return test

The test2.lua
As we pass unnamed parameters to the file we need to extract the first and set the test class:
local params = {...}
local test = params[1]
function test:b()
  print("b")
end

This gives the desired output:

$ th rtest.lua
a
b
end

Possible structure
I haven't thought that thoroughly about this but the main themes seem to be:

Core
Load data (csv + table)
Save data
Print to screen
Print to itorch
Subset data (head/tail/sub/where)
Manipulate data
Utilities

Another thing that may be worth considering is if we should imitate Pandas' Series. This would be nice but I'm not sure its worth the effort. Although I'm currently trying to understand a strange bug in the categorical implementation where I can't create a new class and call the insert() of the same type once I have added a categorical column. Moving the column functionality into a separate class may be a solution to the problem...

There was some additional tweaking necessary in the init.lua/rockspec for everything to work (see my misc-branch for how I've implemented this). I've summarized everything in a SO-question as this information was surprisingly hard to get by.
The current subfiles that I have are:

load_data
save_data
load_batch
missing_data
output
select_set_update
statistics

I'm thinking of adding a categorical file with the specific categorical functions. I've tried to keep main to core functions but we should probably split it a little further now that the infrastructure is in place.

*Mind blown*

I'm thinking of adding a categorical file with the specific categorical functions.

I agree, categorical functions would be better in a specific file.

I've tried to keep main to core functions but we should probably split it a little further now that the infrastructure is in place.

Yes we should split again by creating sub-directories in Extensionsto keep it clean but for now it's great enough to try the new architecture you did ^^.
What you could do, is PR the new functionalities, then PR the new architecture (or vis versa). What do you think ?

Glad you like it. We will unfortunately have to do the file split last as that is how I've organized the branches. I'll add the next PR tonight


Feature/write binary

Problem
SimulariumIO only outputs the JSON version of .simularium format, it would be nice to also output binary.
closes #30
Solution
Split file writing functionality from TrajectoryConverter into a few writer objects, including abstract base Writer, JSONWriter, and BinaryWriter. Each of those implements Writer.save(), which replaces the old TrajectoryConverter.write_external_JSON(). The old TrajectoryConverter.write_JSON() functions are replaced with TrajectoryConverter.save() which uses the writers and defaults to writing binary (but can set binary=False to write JSON).
Note this changes the TrajectoryConverter API, renaming write_JSON() to save() and removing write_external_JSON().
Type of change

[x] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[x] This change requires a documentation update
[x] This change requires updated or new tests

Change summary:

move JSON writing to Writer and JsonWriter
add BinaryWriter to write to binary format
update TrajectoryConverter functions
add a test that checks the float values in the buffer before writing binary are correct
update tests and docs

Keyfiles:

simulariumio/constants.py - added constants for binary header, version, and end of frame symbols
simulariumio/trajectory_converter.py - moved _read_trajectory_data() to JsonWriter.format_trajectory_data() and moved part of _get_spatial_bundle_data_subpoints() to Writer._get_frame_buffer() so it can also be used by BinaryWriter. Updated to_JSON() to use JsonWriter, changed write_JSON() to save(binary=True), and removed write_external_JSON() since that's now handled by JsonWriter.save().
simulariumio/writers/binary_writer.py - format data as a numpy array of floats and then use numpy's
simulariumio/writers/json_writer.py - moved TrajectoryData reading and JSON writing from TrajectoryConverter here, except moved parts of the old TrajectoryConverter._read_trajectory_data() to Writer in order to reuse it for writing binary.
simulariumio/writers/writer.py - abstract base Writer that defines an API with format_trajectory_data() and save()

TODO implement limits for file sizes, both in frames and bytes

Codecov Report

Merging #52 (a1bba5f) into main (033e6c5) will increase coverage by 0.26%.
The diff coverage is 83.13%.

@@            Coverage Diff             @@
##             main      #52      +/-   ##
==========================================
+ Coverage   89.63%   89.89%   +0.26%
==========================================
  Files          76       81       +5
  Lines        2277     2376      +99
==========================================
+ Hits         2041     2136      +95
- Misses        236      240       +4

Impacted Files
Coverage Δ

simulariumio/trajectory_converter.py
75.00% <42.85%> (+3.65%)
:arrow_up:

simulariumio/writers/json_writer.py
58.10% <58.10%> (ø)

simulariumio/writers/binary_writer.py
91.11% <91.11%> (ø)

simulariumio/writers/writer.py
95.89% <95.89%> (ø)

simulariumio/__init__.py
90.00% <100.00%> (+1.11%)
:arrow_up:

simulariumio/constants.py
100.00% <100.00%> (ø)

...ariumio/tests/converters/test_cytosim_converter.py
100.00% <100.00%> (ø)

...ulariumio/tests/converters/test_mcell_converter.py
100.00% <100.00%> (ø)

...lariumio/tests/converters/test_medyan_converter.py
100.00% <100.00%> (ø)

...iumio/tests/converters/test_physicell_converter.py
100.00% <100.00%> (ø)

... and 21 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 033e6c5...a1bba5f. Read the comment docs.

https://docs.python.org/3/library/struct.html#struct.pack


feat: update CiliumBGPPeeringPolicy v2alpha1 CRD schema according

This PR updates the CiliumBGPPeeringPolicy v2alpha1 CRD schema according to: https://github.com/cilium/cilium/blob/5d3bf04c7e2a198e60558224c0c9ab3a747efd03/pkg/k8s/apis/cilium.io/client/crds/v2alpha1/ciliumbgppeeringpolicies.yaml

any news here?


Client secret should not be mandatory

This issue [was already reported some time ago] (https://github.com/AFNetworking/AFOAuth2Manager/pull/9), but those changes are disappeared after moving from AFOAuth2Client to AFOAuth2Manager.
I think it would be great to add it again.

I would also like to point out that the client secret and ID shouldn't be included in the body anyway. they should be encoded using HTTP basic auth. I have already reported this here and will be submitting a PR for this soon. https://github.com/AFNetworking/AFOAuth2Manager/issues/85

Thanks, @OmarPedraza. Due to the changes in 412d2566c3cdfda5f4740dd1b42415ade8747f88, a different change was required with e5c00d1. Look for this in the forthcoming release.


Run docker-compose failed with "failed to solve: executor failed running [/bin/sh -c patch -p1 < nginx.latest.patch]: exit code: 2" on Win10 & Docker Desktop

Env:

Windows 10 x64, 21H2
Docker Desktop 4.7.0 (77141)

Steps:

Run docker/docker-compose.yml

Error:
 => CACHED [ 8/35] RUN git clone https://github.com/nginx/nginx-tests                                              0.0s
 => CACHED [ 9/35] RUN git clone https://github.com/openssl/openssl                                                0.0s
 => CACHED [10/35] WORKDIR /build/openssl                                                                          0.0s
 => CACHED [11/35] RUN git checkout OpenSSL_1_1_1 -b patched                                                       0.0s
 => CACHED [12/35] COPY patches/openssl.extensions.patch /build/openssl                                            0.0s
 => CACHED [13/35] RUN patch -p1 < openssl.extensions.patch                                                        0.0s
 => CACHED [14/35] RUN ./config -d                                                                                0.0s
 => CACHED [15/35] RUN make                                                                                0.0s
 => CACHED [16/35] RUN make install                                                                                0.0s
 => CACHED [17/35] WORKDIR /build                                                                                0.0s
 => CACHED [18/35] RUN hg clone http://hg.nginx.org/nginx                                                          0.0s
 => CACHED [19/35] WORKDIR /build/nginx                                                                            0.0s
 => CACHED [20/35] COPY patches/nginx.latest.patch /build/nginx                                                    0.0s
 => ERROR [21/35] RUN patch -p1 < nginx.latest.patch                                                               0.3s
------
 > [21/35] RUN patch -p1 < nginx.latest.patch:
#0 0.260 patch: **** Only garbage was found in the patch input.
#0 0.260 patch unexpectedly ends in middle of line
------
failed to solve: executor failed running [/bin/sh -c patch -p1 < nginx.latest.patch]: exit code: 2

@fooinha

@fooinha any update?

@hsluoyz , usually I don't build this on Windows 10, but I'll give it a try as soon I get one.


Preview article (new window)

A user needs to:

be able to preview how their content looks before they decide whether to publish it.
[ ] A user can click "preview" to open a preview of their content in a new tab/window, with the content shown in the real look/feel of the final site

We now have a configurable preview system in place.


feat: port Data/Bracket

I don't really get why the "check that all files are imported" check fails...

@javra, would you mind reopening this from a branch of the main repository, rather than a fork, so that CI runs? I've just given you access to do so.


Add http4s to Scala 3 Community Build

https://github.com/lampepfl/dotty/tree/master/community-build
Http4s is too important a project not to be in there 😉 My only concern was our flaky CI, but I think it's been quite good lately? To help mitigate this, we can also only run only a subset of the test suite in the CB (I'm thinking the modules designated in https://github.com/http4s/http4s/discussions/4140#discussioncomment-261676).

Blocked by https://github.com/djspiewak/sbt-spiewak/issues/83.

Http4s is now a part of the Scala 3 Community Build! 🎉


Has someone do inference (denoise an audio) successfully?

I have been digging into the code but I haven't been able to make it work, what I want to achieve is just to denoise files.
What flow have you follow to achieve it?

I'm not sure why you put the :/ emoticon, whether it is the question that is ":/" or you couldn't make it work either. Anyway, for us it did actually work (to run inference, we haven't tried training), so we  are writing out some steps and configs to help others:
We use pycharm on Windows 10.

Cuda version: 10.1 (Update 2!, 1 won't work)
Microsoft Visual Studio 2019 Community Edition with the C++ package (can't remember which one)
Put C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\Hostx64\x64 and
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\Hostx64\x64\cl.exe in the Environment Variables in PATH (System Variables)
These are the currently installed packages in pycharm
We used Conda as well as PIP to install these (and we are aware that mixing these isn't good practice) and most of them installed themselves as part of other packages

blas | 1.0 |
ca-certificates | 2021.7.5 |
certifi | 2020.6.20 |
cffi | 1.14.6 |
cython | 0.25.2 |
h5py | 2.7.0 |
icc_rt | 2019.0.0 |
intel-openmp | 2021.2.0 |
keras | 1.2.0 |
libgpuarray | 0.6.9 |
libpython | 2.1 |
m2w64-binutils | 2.25.1 |
m2w64-bzip2 | 1.0.6 |
m2w64-crt-git | <IP_ADDRESS>36.2595836 |
m2w64-gcc | 5.3.0 |
m2w64-gcc-ada | 5.3.0 |
m2w64-gcc-fortran | 5.3.0 |
m2w64-gcc-libgfortran | 5.3.0 |
m2w64-gcc-libs | 5.3.0 |
m2w64-gcc-libs-core | 5.3.0 |
m2w64-gcc-objc | 5.3.0 |
m2w64-gmp | 6.1.0 |
m2w64-headers-git | <IP_ADDRESS>36.c0ad18a |
m2w64-isl | 0.16.1 |
m2w64-libiconv | 1.14 |
m2w64-libmangle-git | <IP_ADDRESS>09.2e5a9a2 |
m2w64-libwinpthread-git | <IP_ADDRESS>34.697f757 |
m2w64-make | 4.1.2351.a80a8b8 |
m2w64-mpc | 1.0.3 |
m2w64-mpfr | 3.1.4 |
m2w64-pkg-config | 0.29.1 |
m2w64-toolchain | 5.3.0 |
m2w64-tools-git | <IP_ADDRESS>92.90b8472 |
m2w64-windows-default-manifest | 6.4 |
m2w64-winpthreads-git | <IP_ADDRESS>34.697f757 |
m2w64-zlib | 1.2.8 |
mako | 1.1.4 |
markupsafe | 1.1.1 |
mkl | 2020.2 |
mkl-service | 2.3.0 |
msys2-conda-epoch | 20160418 |
numpy | 1.16.6 |
pip | 19.3.1 |
pycparser | 2.20 |
pygpu | 0.6.9 |
python | 2.7.18 |
pyyaml | 5.4.1 |
scipy | 0.19.0 |
setuptools | 44.0.0 |
six | 1.16.0 |
soundfile | 0.9.0 |
sqlite | 3.30.1 |
theano | 1.0.5 |
tqdm | 4.11.0 |
vc | 9 |
vs2008_runtime | 9.00.30729.1 |
wheel | 0.36.2 |
wincertstore | 0.2 |

In the keras.json config file, which is located at "C:\Users\My_User_Name.keras"  we changed the line from "backend": "tensorflow" to  "backend": "theano"
In the project itself we made sure that the loaded checkpoint pointed to the correct path
This is running slow, so perhaps the gpu isnt actually being used

not worked for me


adds MANIFEST.in, required to include the requirements file

Codecov Report

Merging #210 (06b12b7) into main (483f6c4) will increase coverage by 4.77%.
The diff coverage is 86.22%.

@@            Coverage Diff             @@
##             main     #210      +/-   ##
==========================================
+ Coverage   76.80%   81.58%   +4.77%
==========================================
  Files          34       36       +2
  Lines        3065     3350     +285
  Branches      618      647      +29
==========================================
+ Hits         2354     2733     +379
+ Misses        520      445      -75
+ Partials      191      172      -19

Impacted Files
Coverage Δ

plenoptic/metric/model_metric.py
100.00% <ø> (ø)

...mulate/canonical_computations/laplacian_pyramid.py
100.00% <ø> (ø)

plenoptic/simulate/models/frontend.py
93.45% <ø> (ø)

plenoptic/synthesize/metamer.py
84.50% <ø> (+8.26%)
:arrow_up:

plenoptic/tools/signal.py
44.26% <ø> (ø)

plenoptic/synthesize/simple_metamer.py
28.57% <14.28%> (+2.99%)
:arrow_up:

...e/canonical_computations/steerable_pyramid_freq.py
86.66% <40.00%> (+0.22%)
:arrow_up:

plenoptic/tools/display.py
68.63% <50.00%> (ø)

plenoptic/tools/data.py
67.20% <60.00%> (-0.27%)
:arrow_down:

plenoptic/simulate/models/portilla_simoncelli.py
81.30% <81.48%> (+13.22%)
:arrow_up:

... and 14 more


Multiple Redis checks are breaking the application

Since we use multiple Redis databases for our application, we want to monitor them all. I added them to the service provider like this:
        Health::checks([
            RedisCheck::new()->connectionName('default'),
            RedisCheck::new()->connectionName('cache'),
            RedisCheck::new()->connectionName('queue'),
        ]);

After that, the application is throwing a white page and is broken. If I only add one of the Redis checks, everything is working.

Could you try giving the checks a unique label and check if that works for you. If it's not, I'd accept a PR that fixes the problem.


WebPubSubClient::close emits unexpected warning

Package Name: azure-messaging-webpubsubclient
Package Version: 1.1.0, 1.0.1
Operating System: Mac
Python Version: 3.10

Describe the bug
When attempting to close the webpubsub client, a warning is emitted: "The client is stopping state. Stop recovery."
It's not clear that this warning is actionable, and I can't see any way to prevent it (other than leaving the connection open, or filtering logs).
To Reproduce
Steps to reproduce the behavior:

create webpubsub client
open connection
(exchange messages)
call client.close()

Similarly, via the context manager:
wps_url = "wss://xxxxx.webpubsub.azure.com/client/hubs/xxxxxx?access_token=xxxxxx"
credentials = WebPubSubClientCredential(client_access_url_provider=wps_url)
wps_client = WebPubSubClient(credential=credentials)
with wps_client:
    pass

Expected behavior
No warning message for happy-path usage.

Thanks for the feedback, we’ll investigate asap.

Let me take an investigation.


Fix native-platform sourcesetvariant test on gcc10

Context
While working on a different PR, I tried to run :platform-native:quickTest, which caused embeddedIntegTest failure:
> Task :platform-native:embeddedIntegTest
NativePlatformSamplesIntegrationTest > sourcesetvariant [gcc 10.2.0] FAILED

/usr/bin/ld: /home/vmerk/src/gradle/subprojects/platform-native/build/tmp/test files/NativePlatf.Test/fwc91/sourceset-variant/build/objs/main/mainExecutablePlatformLinux/3aor34f2b62iejk2eq3fn5ikr/platform-linux.o:(.data.rel.local+0x0):
multiple definition of `platform_name'; /home/vmerk/src/gradle/subprojects/platform-native/build/tmp/test files/NativePlatf.Test/fwc91/sourceset-variant/build/objs/main/mainC/dey3oyi6y0a9luwot945rff8j/main.o:(.bss+0x0): first defined here
collect2: error: ld returned 1 exit status

Looking at the sample that's being used for this test, it defines a global variable platform_name without marking it extern.
GCC defaults to to -fno-common starting with version 10, which causes the linker error above.
Please see the Porting to GCC 10 Default to -fno-common section for more details.

Contributor Checklist

[x] Review Contribution Guidelines
[x] Make sure that all commits are signed off to indicate that you agree to the terms of Developer Certificate of Origin.
[x] Make sure all contributed code can be distributed under the terms of the Apache License 2.0, e.g. the code was written by yourself or the original code is licensed under a license compatible to Apache License 2.0.
[x] Check "Allow edit from maintainers" option in pull request so that additional changes can be pushed by Gradle team
[ ] Provide integration tests (under <subproject>/src/integTest) to verify changes from a user perspective
[ ] Provide unit tests (under <subproject>/src/test) to verify logic
[ ] Update User Guide, DSL Reference, and Javadoc for public-facing changes
[x] Ensure that tests pass sanity check: ./gradlew sanityCheck
[x] Ensure that tests pass locally: ./gradlew <changed-subproject>:quickTest

Gradle Core Team Checklist

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation
[ ] Recognize contributor in release notes

@bot-gradle test and merge

Error while scheduling pre-tested commit, please contact the administrator.

@bot-gradle test this

OK, I've already triggered ReadyForMerge build for you.

@bot-gradle test this

OK, I've already triggered ReadyForMerge build for you.

@bot-gradle test this

OK, I've already triggered ReadyForMerge build for you.


Suppress gcc 7 switch statement warnings.

Not really Coverity, but that seemed the closest in spirit of the tags. The code is correct, what is needed is comments to make the compiler shut up.

There might be a bunch of these, let me collect them all, and put into an Issue, so we can resolve it all in one go.

See #2266 for a complete list of F26 warnings.

This and others were fixed in #2271


SSL certificates are not loaded in Nginx image

We were using tag owasp/modsecurity-crs:3.3.2-nginx to build our application image on top of it. It worked fine until image under that tag was updated in DockerHub and our SSL certificates suddenly stopped working. No changes were done on our side, we just rebuilt our image with the same source code and the same FROM tag. If we build from original image of the same underlying Nginx version - it works ok.
The issue is reproducible with current owasp/modsecurity-crs:3.3.2-nginx and also with latest owasp/modsecurity-crs:3.3.4-nginx-202209221209
There is no issue with image owasp/modsecurity-crs@sha256:4ab21fb9a6cab32fc04b15017b4dce68c6080ca6293726321f31077a7cf76468 which was under the owasp/modsecurity-crs:3.3.2-nginx tag at the time of our successful build 19.11.2021.
We attach SSL certificate from disk using ssl_certificate and ssl_certificate_key settings in server section of Nginx config. Actually those cert files are mapped to container as secrets in docker-compose.yml file.
Issue manifests itself as an SSL error when trying to open application page in Chrome browser, and missing SSL certificate, substituted somehow with self-signed one. See screenshots below:

Hi @andrei-piarainc, sorry for the delay.
Tried to reproduce your issue, without any success.
Created test certificates, and I'm using this docker-compose.yml:
version: "3"

services:

  crs:
    image: owasp/modsecurity-crs:3.3.4-nginx-202209221209
    ports:
      - "80:80"
      - "443:443"
    depends_on:
      - backend
    environment:
      - SERVERNAME=localhost
      - BACKEND=http://backend
      - PARANOIA=1
      - BLOCKING_PARANOIA=1
      - ANOMALY_INBOUND=5
      - ANOMALY_OUTBOUND=4
      - REPORTING_LEVEL=2
    volumes:
      - ./certs/server.crt:/etc/nginx/conf/server.crt
      - ./certs/server.key:/etc/nginx/conf/server.key
  backend:
    image: docker.io/kennethreitz/httpbin

Starting container:
❯ docker-compose up crs
[+] Running 29/29
 ⠿ crs Pulled                                                                                5.5s
   ⠿ 214ca5fb9032 Already exists                                                                                0.0s
   ⠿<PHONE_NUMBER>7f Already exists                                                                                0.0s
   ⠿ d838e0361e8e Already exists                                                                                0.0s
   ⠿ fcc7a415e354 Already exists                                                                                0.0s
   ⠿ dc73b4533047 Already exists                                                                                0.0s
   ⠿ e8750203e985 Already exists                                                                                0.0s
   ⠿ b5d0788b0531 Already exists                                                                                0.0s
   ⠿ 5557a943054d Already exists                                                                                0.0s
   ⠿ 8b69571cb347 Already exists                                                                                0.0s
   ⠿ 048fdaa394a1 Already exists                                                                                0.0s
   ⠿ 57428e837e42 Already exists                                                                                0.0s
   ⠿ 78acc6ccb17d Already exists                                                                                0.0s
   ⠿ 99fed18ab947 Already exists                                                                                0.0s
   ⠿ 168c381da758 Already exists                                                                                0.0s
   ⠿ 29fb0b5a809e Already exists                                                                                0.0s
   ⠿ a8432c3298f5 Already exists                                                                                0.0s
   ⠿ c9c47ced50e4 Already exists                                                                                0.0s
   ⠿ e1160058b5eb Already exists                                                                                0.0s
   ⠿ d44ec0028bb6 Already exists                                                                                0.0s
   ⠿ acf44cb08744 Already exists                                                                                0.0s
   ⠿ a80a12c4feab Already exists                                                                                0.0s
   ⠿ 2505253b84df Already exists                                                                                0.0s
   ⠿ d03833d83606 Pull complete                                                                                0.9s
   ⠿ a2c08c41ee70 Pull complete                                                                                1.0s
   ⠿ 0c4ba2ffe6e2 Pull complete                                                                                1.0s
   ⠿ e0b9f94f8879 Pull complete                                                                                1.6s
   ⠿ e8b83b401ea2 Pull complete                                                                                2.0s
   ⠿ d59719d493db Pull complete                                                                                2.1s
[+] Running 2/2
 ⠿ Container modsecurity-crs-docker-backend-1  Running                                                                                0.0s
 ⠿ Container modsecurity-crs-docker-crs-1      Recreated                                                                                0.2s
Attaching to modsecurity-crs-docker-crs-1
modsecurity-crs-docker-crs-1  | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
modsecurity-crs-docker-crs-1  | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
modsecurity-crs-docker-crs-1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
modsecurity-crs-docker-crs-1  | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
modsecurity-crs-docker-crs-1  | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
modsecurity-crs-docker-crs-1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
modsecurity-crs-docker-crs-1  | 20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/modsecurity.d/setup.conf.template to /etc/nginx/modsecurity.d/setup.conf
modsecurity-crs-docker-crs-1  | 20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/modsecurity.d/modsecurity-override.conf.template to /etc/nginx/modsecurity.d/modsecurity-override.conf
modsecurity-crs-docker-crs-1  | 20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/conf.d/modsecurity.conf.template to /etc/nginx/conf.d/modsecurity.conf
modsecurity-crs-docker-crs-1  | 20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/conf.d/default.conf.template to /etc/nginx/conf.d/default.conf
modsecurity-crs-docker-crs-1  | 20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/conf.d/logging.conf.template to /etc/nginx/conf.d/logging.conf
modsecurity-crs-docker-crs-1  | 20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/nginx.conf.template to /etc/nginx/nginx.conf
modsecurity-crs-docker-crs-1  | 20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/includes/proxy_backend.conf.template to /etc/nginx/includes/proxy_backend.conf
modsecurity-crs-docker-crs-1  | 20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/includes/location_common.conf.template to /etc/nginx/includes/location_common.conf
modsecurity-crs-docker-crs-1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
modsecurity-crs-docker-crs-1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/90-copy-modsecurity-config.sh
modsecurity-crs-docker-crs-1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/91-update-resolver.sh
modsecurity-crs-docker-crs-1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/92-update-real_ip.sh
modsecurity-crs-docker-crs-1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/95-activate-rules.sh
modsecurity-crs-docker-crs-1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/96-configure-plugin-includes.sh
modsecurity-crs-docker-crs-1  | /docker-entrypoint.sh: Configuration complete; ready for start up
modsecurity-crs-docker-crs-1  | 2022/10/22 15:04:03 [notice] 1#1: ModSecurity-nginx v1.0.3 (rules loaded inline/local/remote: 0/920/0)

And this is my test:
❯ curl -ksvL https://localhost/anything
*   Trying <IP_ADDRESS>:443...
* Connected to localhost (<IP_ADDRESS>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=Nginx test CRS
*  start date: Oct 22 14:35:42 2022 GMT
*  expire date: Apr 22 14:44:32 2024 GMT
*  issuer: CN=CRS
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET /anything HTTP/1.1
> Host: localhost
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.20.2
< Date: Sat, 22 Oct 2022 15:04:07 GMT
< Content-Type: application/json
< Content-Length: 321
< Connection: keep-alive
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
<
{
  "args": {},
  "data": "",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Connection": "close",
    "Host": "localhost",
    "Upgrade": "close",
    "User-Agent": "curl/7.79.1"
  },
  "json": null,
  "method": "GET",
  "origin": "<IP_ADDRESS>",
  "url": "https://localhost/anything"
}
* Connection #0 to host localhost left intact

As you can see, the certificate is being picked. Maybe there is something with secrets or some specific setup in your environment. Can you test with this docker-compose and your own certificates?

@fzipi Thanks for looking into this, I've tried your example and it worked ok with our certificates. I've just was discouraged that our setup works with previous builds of "owasp/modsecurity-crs". Anyway, looks like the problem on our side, so I close this issue. Sorry for bothering.

No worries. Maybe it is a problem with docker secrets? Happy to help if you have a way of reproducing it :)

Probably secrets do not influence, I've tried attach certs as volumes to my image, like you do, and got the same problem with. I have still working setup, based on particular digest of older owasp/modsecurity-crs image build, only blocking to upgrade ;) I'll try put here more information when I find time to look into this once again.

I was back to that issue once again and solved the problem. I just put this information here just in case someone else will search for similar.
The problem was in the way I provide default.conf for container.
For latest modsecurity-crs image I do it like this and it works:
configs:
  - source: nginx-config
    target: /etc/nginx/templates/conf.d/default.conf.template

it works for me on owasp/modsecurity-crs:3.3.4-nginx-202302060502
For plain Nginx which is the base of that image (nginx:1.22.1) I should put config in different location to make it work:
configs:
  - source: nginx-config
    target: /etc/nginx/templates/default.conf.template

To make my docker-compose.yml work for both cases I just map config in both places, and then I can switch between images without changing this part of configuration:
configs:
  - source: nginx-config
    target: /etc/nginx/templates/default.conf.template
  - source: nginx-config
    target: /etc/nginx/templates/conf.d/default.conf.template

If Modsecurity nginx gets config in /etc/nginx/templates/default.conf.template, when I open my in Chrome I get ERR_CERT_AUTHORITY_INVALID and browser blocks navigation as unsafe. If Modsecurity nginx gets config in proper place, everything works OK.
When plain Nginx gets config in /etc/nginx/templates/conf.d/default.conf.template, my application does not load at all with ERR_CONNECTION_REFUSED in Chrome.
I could imagine problem with relation path to certs in config, but this should not be the case because certs are referenced by absolute path in my config.
ssl_certificate      /etc/nginx/conf.d/certbundle.pem;
ssl_certificate_key  /etc/nginx/conf.d/certkey.pem;

Thanks for the explanation.


Updating dependencies to avoid CVEs

As per title. Dependencies updated:

spring-boot-starter-parent : from 2.2.0 to 2.3.2
cxf-rt-rs-client : from 3.3.1 to 3.3.4

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.