ENH: Finite difference approximation of Jacobian matrix

At some point I wanted to run "check_grad" on vector-valued function, but it doesn't work. Also I will need a numeric Jacobian approximation function for the future work.
I decided to implement them as new functions, because it was difficult to modify the old ones without breaking the interface, etc.
It is not well polished and subject to change, but I decided to make this pull request earlier.

Test failure is a timeout, all tests for added code pass.

The signatures of the new functions compared to approx_fprime and check_grad are almost identical right? The backwards compatibility issue of modifying the existing functions could be true - it's a bit hard to judge for me because of the very poor test coverage of approx_fprime and check_grad.

So first impression: the new functions definitely look better than the existing ones, I just wonder whether the old ones can be completely replaced by the new ones.

What will this be used for?
It's also not obvious to me why modification of existing functions would break things.
Also, if you need more advanced numerical differentiation things, the possibility of integration numdifftools into scipy was discussed/proposed in #2835, perhaps it's a good opportunity to move this forward.

@dlax numdifftools is way out of scope here, that was a full GSoC project in itself.

Numerical Jacobian estimation is used in least squares optimization. In leastsq it relies on a function implemented in MINPACK. At some point I will need such a function. And checking Jacobian implementation is an obvious generalization of checking gradient implementation, it is very useful for sure. In MATLAB it is done inside optimization routines if a user sets the corresponding flag. Maybe it's another way to go.
Original functions can be adapted (probably), but I don't see any reason to do so, except that they are already here. Is it strong reason enough? I don't know.
I think we can forget about it now. I will use this code as external for a moment. Hopefully I will have a discussion about it with my mentors soon.

Here is the description of approach taken in MATLAB http://www.mathworks.com/help/optim/ug/checking-validity-of-gradients-or-jacobians.html
Maybe I should implement this approach. What do you think? And what is the general opinion on borrowing ideas from MATLAB docs? MATLAB is the number one source for numerical methods, I guess it's very reasonable too study their approaches. And there is nothing illegal in doing so, right?

My guess, based on a quick look, is that this is similar to the approx_fprime in statsmodels, which was originally loosely based on the scipy version.
I'm using that one quite often to check analytical score/jacobian (and the same for hessian)

I looked into this when working with MINUIT, which expends most of its function evaluations computing numerical gradients. There are a number of bits of cleverness one should be aware of:

It uses a symmetric difference to allow larger steps (error is O(h**2) rather than O(h)).
It uses the central point to estimate the function's second derivative, and hence to estimate the approximation error.
It uses estimates of numerical inaccuracy (optionally provided by the user, for functions that cannot be computed to machine-epsilon accuracy) to decide when the step size h is too small.
It expends a limited amount of effort computing the derivatives, since it's better to move on rather than spend a lot of effort on really-exact derivatives.
It retains second-derivative and step-size information from one call to the next on the assumption that successive calls are at similar locations.

These criteria (and a few more sanity checks) have been hammered out by years of particle physicists, and they're appropriate for MINUIT. On the other hand numdifftools spends a lot of effort getting the derivatives exactly right, which is great if those derivatives are your key output, but can be expensive (hours, for my problem).
The point of all this is, there are conflicting criteria for numerical differentiation, so it's not going to be possible to come up with a single implementation that works for everyone. For that matter, there's a need for a MINUIT-like derivative computation in scipy to make optimizers work robustly, but it's not clear how generic it is possible to make one. (Incidentally derivative evaluation is an opportunity for parallelization.)

IMO there are several issues here which are best kept separate. First is what Nikolay needs for his GSoC work for computing/checking Jacobians. I'd say it's fine to add whatever is needed as private utility functions. @nmayorov just use leading underscores for private functions. There is no need to add these helpers to __all__ etc.
A public functionality for numerical differentiation is a separate question. Adding a simple first-order forward-difference formula is IMO not worth it, a user can do (f(x + h) - f(x)) / h themselves. OTOH, a smarter step-size selection is nice. Adding a a full-fledged differentiation framework with Richardson extrapolation and MINUIT-type sophistication (for use the cases @aarchiba describes) is clearly out of scope.
I'd say a useful middle ground could be to implement a function for a fixed-step differentiation with a selector for a first-, second- and maybe fourth-order schemes, and deprecate the semi-broken misc.derivative and  approx_fprime. (A tentative signature could be approx_derivative(f, x0, h=None, order='second') where h = None means that the step size is computed along the lines of what is done in this PR; the details are to be ironed out). This could be IMO a simple but still nice byproduct of Nikolay's GSoC.  When scipy.diff materializes, this function could be consolidated into it. Until then, we'll have something better than what we have now.

Evgeni, thanks for your input.
If this new function is supposed to be used in optimization routines, then it should probably has two additional features:

Reuse of f(x0) in forward difference, like f0=None. Btw, are any other reasons to use forward difference except computation savings?
Considerations of bounds on variables: perhaps it is more correct to evaluate function only inside the feasible region. The only reasonably simple case is bound constraints, which I will be working with.

If it will not be used in optimization routines, then I think the interface suggested by you is enough. And check_grad should be updated (perhaps as check_derivatives). But in this case I will need to reimplement Jacobian estimation again (with bounds considerations).
What to you think?

Reuse of f(x0) in forward difference, like f0=None.

Of course --- my suggestion above is only a sketch. f0 is probably useful, and so is extra args (as in f(x, *args)). It might be also worth thinking a bit about what it actually returns, esp in light of what Anne says above. Does it return an error estimate (hence, the second derivative). Does it somehow cache intermediate values.
A couple of quick ideas (not necessary good ones): variable number of returns is a mess; maybe return a Bunch with values and optional error estimates (may also evolve into a mess); have it a class with a __call__ method instead of a function so that you can store the step size and second derivative estimate from  a previous call?

Btw, are any other reasons to use forward difference except computation savings?

I suppose you mean the first-order forward difference rather than, say, second-order central difference. Don't know. Simplicity? Most likely, there's no compelling reason unless an extra function evaluation is really expensive.
I think it's not too difficult to provide several finite-difference schemes, including the simplest possible one, one workhorse (2nd order central differences) and a fancy one --- central difference is something many users can write without much thought; the coefficients for the 4th order one are not likely to be at the top of everybody's mind. Hence there's value at providing it.

Considerations of bounds on variables: perhaps it is more correct to evaluate function only inside the
feasible region. The only reasonably simple case is bound constraints, which I will be working with.

I'd take a guess: in summer 2015 you are going to be one of the heaviest and most immediate users of this functionality :-). What sort of API/functionality do you expect to need / to be useful for you?

A minor comment for your future PRs: it's better to format your commit messages as described in http://docs.scipy.org/doc/numpy/dev/gitwash/development_workflow.html

I'm thinking about the interface like this approx_derivatives(f, x0, rel_step=None, order='second', f0=None, args=(), lb=None, ub=None) Most users can completely ignore the last two parameters. Implementing high order approximations with bound consideration may be tricky. I'll start with the first and second orders. I will provide a new version tomorrow.
About bounds. They complicate this simple task, but have to be considered in numerical differentiation. It is done in MATLAB and in this CoDoSol code accompanying the paper I mentioned.
About what Anne said. All sound very good, but probably too complicated. I have a goal in mind to have a simple, but satisfactory function for derivatives approximation to use in optimization routines. In slsqp.py there is a public function approx_jacobian, approx_fprime is used here and there. So overall such function is required and it's not difficult to make it better than current implementations.

I would generalize order to method.   AFAIR, some of the methods used in statsmodels have names. Also, if they apply, then complex step derivatives are by far the most precise.

Hi! I made a new implementation in a new file. I would like to refine and finally merge, if no one has strong objections in general.
So @ev-br please review.

Build fails with numpy 1.6.2 because I try indexing of 0-d array. I will wait for the feedback, before doing anything about that.
About naming of commits: at the end I will rebase into a single commit with a good name.

Nikolay Mayorov wrote:

About naming of commits: at the end I will rebase into a single commit
with a good name.

IMHO, this is worth waiting for "the end" to cleanup the history, since
this may eventually require another review.

If someone wanted to write a reasonably reliable minimizer based on this, they would definitely want the estimate of the second derivative that you can get essentially for free with the symmetric rule. This gives an error estimate based on the function's actual second derivative rather than some a priori estimate (as used in the Press rule); they would also want an error estimate for the roundoff that allowed for a function not calculated to machine-epsilon accuracy. I'm not suggesting these features be added here (although automatic step-size determination would make the function much more reliable), but perhaps an option to return these two error estimates would allow a wrapper to adjust step sizes as needed?

If someone wanted to write a reasonably reliable minimizer based on this, they would definitely want the estimate of the second derivative that you can get essentially for free with the symmetric rule. This gives an error estimate based on the function's actual second derivative rather than some a priori estimate (as used in the Press rule);

The second derivative is required for estimating optimal step size in the first order accuracy forward scheme, for the second order scheme we need the third derivative and so on. It doesn't look very practical. Maybe another version of function can focus specifically on the first order scheme and choose this adaptive step, which sometimes, I agree, is absolutely necessary (for example differentiating log(x) near 0).
BTW, In MATLAB optimization toolbox they are fine with simple "Press rule". Maybe it's indeed a good idea to add adaptive step specifically for the first order scheme. I will discuss it with Evgeny.

they would also want an error estimate for the roundoff that allowed for a function not calculated to machine-epsilon accuracy.

Do you mean that a user can provide a custom accuracy with which a function is computed?

I am making these suggestions out of experience with MINUIT - it is a minimizer that uses numerical differentiation, and it has seen decades of use in the particle physics community. In the process it has been made quite robust against problem functions.
To compute the derivative it uses the symmetric rule but, since it requires no extra function evaluations, computes the second derivative as a check on step size. Indeed the second derivative is not actually a problem for the symmetric rule, but a step size small enough for the first-order method is also good enough for the second-order method; this is the same principle as embedded 4/5 Runge-Kutta ODE solvers: use the higher order to estimate the truncation error in the lower order, make sure it's small enough, and then use the higher order for some "free" accuracy.
MINUIT also allows you to specify that your function is not computed to machine precision, yes. In my case, function evaluations are good only to the nanosecond level, because they require some fairly laborious ODE solving. A reliable numerical differentiator must be able to cope with this, and a reliable minimizer must take this into account when deciding when to stop. For the differentiator, it amounts to supplying a different value for machine epsilon and then using this in step size selection.
The final thing MINUIT does, which I'm not sure I'm suggesting here, is when adapting its step size to the problem, give up after a relatively small number of tries. This is to avoid wasting time fiddling with step sizes when you're not anywhere near the function minimum yet anyway. Since the next iteration re-uses step size information, the process of repeatedly computing derivatives converges on the right step size before too long anyway. I'm not suggesting this, particularly; I think for a MINUIT-like optimizer we'd want a smart adaptive step-size selector to wrap around this function. The trick is it needs error estimates to come from inside this function. So I'm trying to make sure that this function is capable of returning what an adaptive step-size driver would need.

Anne, your suggestions are vey valuable because they came from practical experience. We will discuss all of them tomorrow with Evgeny.

Here's a wiki proposal covering at least some of the ideas mentioned in this discussion or in http://mail.scipy.org/pipermail/scipy-dev/2015-June/020729.html. If this nonlinear-least-squares GSoC is planning to implement sophisticated finite-differences-based numerical derivatives, then I'd suggest that its interface allow broadcasting compatible with the current usage in scipy.stats.  On the other hand, @rgommers has suggested above that the direction towards fancy numerical differentiation comparable to numdifftools would be out of scope for this GSoC.

numdifftools takes a different approach to stepsize selection: it tries a huge selection of different step sizes (ten-plus octaves?), chops off the ones that are badly affected by roundoff (based on the scattery nature of roundoff errors), and then treats the rest as a sequence converging to the right answer, which it evaluates using convergence-acceleration techniques. It works, even on my messy problem, but it's really expensive (many tens of function evaluations per input dimension; for me that's twenty-odd dimensions, times tens of function evaluations, times several seconds per evaluation). Not suitable for use inside an optimizer, but if you want derivatives to do (say) a sensitivity analysis, it's the right tool. There's no more magic inside it than is present elsewhere in scipy, but it's a relatively clean pure-python module, so it may or may not be worth pulling a version of the code into scipy. It does need to be distinct from the version for use in scipy's optimizers, and it's perhaps too expensive for some use cases. Though it would parallelize extremely well.

@argriffing The idea was to make a function on par with MATLAB numerical differentiation in optimize toolbox. It's all very simple:

forward and central schemes
help in selection of a reasonable step, suggestions of @aarchiba are more advanced, but I think worth implementing
ability to limit the region of function evaluation (bounds)

I like the idea of returning a pair (derivative, error estimate) to parallel what integrate.quad returns.
The logic of the function would then be: (i) use a simple heuristics a la Press et al to calculate the step unless provided by the user, (ii) calculate the estimate for the derivative using the finite difference scheme requested, (iii) estimate a relevant higher derivative using the same difference scheme and step size, (iv) combine it with the estimate of the roundoff error into the final error estimate.
To cover the use case where the original function cannot be calculated to the machine precision, we can add an eps kwarg parameter (a made-up name), defaulting to np.finfo(float).eps.
Would this that cover your bases @aarchiba?
Adding emcee-style map argument looks nice, but can IMO be done in a subsequent PR.

I think one would want two separate error estimates, one for round-off and one for truncation, so that driver code knows how to adjust the step size. And I don't think it's necessary to always return all this information; many users will be fine with the default step size and no error estimate. I just want to make sure it's not necessary to re-implement this function for use within an optimizer. So a full_output argument would seem to be the standard way to approach this. There's also the question of what to do for multidimensional functions (n->m); certainly you need to be able to specify a stepsize in each input dimension, so presumably we'd need to come up with some kind of summary error for all output dimensions and return n error estimates?
I agree adding parallelization through a 'map' kwarg is a minor change and can be done later. And 'eps' is a perfectly reasonable name for the function's uncertainty; in fact it's what MINUIT calls it.

I don't quite get all your suggestion. We agreed that estimating truncation error for high-order schemes requires to compute high-order derivatives (like the third derivative for central scheme). How are we going to do that? (Yes, it's possible, but clumsy).
Also in my opinion selecting step optimally inside the function is much more convenient for a usual user, than asking to adjust step judging by the errors (it will frighten people).

FWIW I'd rather avoid a variable number of returns and full_output. Simply returning a tuple such that
if res = numerical_derivative(....) than res[0] is always the value of the derivative (gradient, jacobian), and res[1] is the error estimate is IMO better. If we need to pack more, then I'd opt for an additional kwarg with output (this is what I was suggesting earlier with context dict, and we agreed with Nikolay to ditch it for now).
Regarding the multidimensional functions and component-dependent steps: I believe this is implemented already. ATM for a function mapping from R^n to R^m, the output is an m-by-n array. This way, error estimates would have the same shape.

The problem is, finding an optimal step size is a non-trivial task, and different users have different requirements. So I'm assuming that users who don't care will just ask for the Press rule and get back a plain derivative. But users who do care need more information, in particular, estimates for both round-off and truncation error (separately!), in order to be able to choose a suitable step size, that is a step size that balances the one against the other.
Estimating truncation error can be done (for example) by using a pair of methods, let's say asymmetric and symmetric, that use the same function evaluation points but are of different orders. The difference between estimates is roughly the error on the lower order. So adjust the step size so that this is small enough. Then you might as well use the higher-order method's value.

@argriffing can you elaborate on what is your suggestion re broadcasting? Is the issue that in stats, e.g. in pdf(x, a, b) a and b are assumed to be scalars, and array  values for a  and b are being broadcast over? I guess this interface would be a a bit unexpected for uses in optimization.
I see that you did some workarounds in stats in gh-2835...

full_output is annoying but standard, and the fact is that some users are going to want specialized information out of this function and others are not. For use within a minimization algorithm, returning a tuple forces users to use a trampoline.

Is the issue that in stats, e.g. in pdf(x, a, b) a and b are assumed to be scalars, and array values for a and b are being broadcast over? I guess this interface would be a a bit unexpected for uses in optimization. I see that you did some workarounds for stats in gh-2835...

@ev-br Yes I think so. I was just sharing the experience of trying to "deprecate the semi-broken misc.derivative" as mentioned in this thread.

If we are going to have variable returns, it might make sense to return something like OptimizeResult.

The advantages of a dict-like return is that it is flexible, introspection is informative, and the method is documented. If differentiation is expensive the added overhead is negligible.

I think an ordered tuple is a nice solution for a return value with extra information, but I think a plain array is also an important option if this is to be usable as a drop-in Jacobian. Hence full_output.

Note that the result of an optimization is easily available in OptimizeResult.x, we could use df for that. The result could also be an input if it is desired to provide prior results to the function.

Sorry, I meant named tuple, not ordered (they're all ordered, the point was to have a dict-like option as you describe). And yeah, dict-like returns allow extra data to be easily added later if it is needed, and existing code will ignore it.

@nmayorov It would be good if the new nonlinear least square fitting also returned a dict-like object like the other optimization routines. The old leastsq could be made a wrapper for backward compatibility.
@aarchiba Yes, that is one of the advantages. I don't think Nikolay should to spend too much time on the differentiation, it could be a gsoc project in itself, but the function should be such that it can be improved and options added without breaking a lot of stuff.

OK, this now blocks progress of gh-5019.
In its current form and shape, this PR does not add any public functionality, and simply serves as a building block of LSQ algorithms, gh-5019. Various bits and pieces can be reused for constructing a public scipy.diff functionality, but this can be done separately to avoid blocking the LSQ work.
I'm thus going to merge this soon unless further comments.
@nmayorov it'd help if you squash it down to several commits, 16 commits is too much.

I agree, this can go in.

Evgeni, I somehow missed you comment. I'll do the final review (maybe add some more tests for sparse differencing) and squash commits.

@nmayorov I suggest to remove sparse anything from this PR and move these commits to a future PR with actual sparse LSQ. This way it's easiest from the review perspective.

I removed "sparse" commits, hopefully didn't lost them in the process. Now what?

Now it's merged. Thanks @nmayorov, keep them coming!


Introduce support for value comparison to expression language

We need to have general support for value comparisons within the expression language,  for all types supported by attributes.

Done: https://github.com/istio/istio/pull/4156


Publish test results is not working properly

2020-09-19T10:24:17.3709790Z ##[warning]Failed to upload file Post_Del_APIs.postman_collection.xml to Blob Transfer exception with errorcode Unknown, exception message Microsoft.Azure.Storage.DataMovement.TransferException: The transfer failed.
We are getting above error for Publish Test Result task. PFA completed logs
logsado.txt
Task definition :
- task: PublishTestResults@2
displayName: 'Publish Test Results **/test-results.xml'
inputs:
testResultsFiles: '$(System.DefaultWorkingDirectory)/CICD_RM_Services/$(SERVICENAME)/test-results.xml'
searchFolder: '$(System.DefaultWorkingDirectory)'
failTaskOnFailedTests: true

@ashu16815 Are you still able to reproduce this issue ?
This seems to be connection issue. Might be an intermittent failure.
Which type of agent are you using in your pipeline?
If possible can you please share fiddler traces for more analysis ?

@ashu16815 Can you please share the logs so that we can diagnose the issue?

I'm getting the same error, but right after implementing an outbound firewall four our self-hosted agents. Our agents are able to talk to Azure Devops just fine for things like publish artifact steps, but publish build results appears to depend on other blob storage accounts and it's not documented what needs to be opened up.

I'm getting the same error, but right after implementing an outbound firewall four our self-hosted agents. Our agents are able to talk to Azure Devops just fine for things like publish artifact steps, but publish build results appears to depend on other blob storage accounts and it's not documented what needs to be opened up.


Selectors for Protocol empty state

Protocol Empty state needs the selectors for below elements:

[x] Import a protocol to get started
[x] Choose file button
[x] Drag and drop section
[x] Protocol Library link
[x] Protocol Designer link
[x] Python API documentation

all the selectors are available


导入china.js,实用中国地图，“南沙诸岛”四个字乱码。详情见如下，十万火急

head

in the body,there are  only one div

show the place “南沙诸岛” with wrong words
who can tell me what's wrong with it? wait online ........
i guess that it maybe something wrong with china.js。 open china.js there are no “南沙诸岛”.where differe from others.

设置页面为 UTF-8 编码

哥哥，改了也不行啊。能试过的都试过了


Tiny regression bug

Hi, it's me again! :)
In the new version the "db" property was changed from this:
@property
def db(self):
    return self.mongo

to this:
@property
def db(self):
    return self.mongo.db

I don't think that this was intended since this pre-determines the collection to be "db" and also conflicts with the actual query call:
    _results = list(self.db[self.collection].aggregate(_agg))

Oh, I just saw that I have Merge commit in there :/
Feel free to do the fix yourself to keep the history clean!

Hi @reinoldus
the pymongo syntax is like pymongo_object.db.collection, so it should work this way.  self.db has been "self.mongo.db" since the first commit.  Did you have a problem that made you suspect it changed?

Hi,
I am slightly confused where my version is coming from, but wouldn't the way it is done right now, fix the database name to "DB"?
https://api.mongodb.com/python/current/tutorial.html#getting-a-database

Wouldn't it be more generic if the "pymongo_object" would be the connection to the database already?

hey there, if you're using Pymongo with Flask (flask_pymongo), then the database name is already specified in the connection string (as the MONGO_URI variable in the config, like app.config["MONGO_URI"] = "mongodb://localhost:27017/myDatabaseName").  So mongo as passed to mongo-datatables is the pymongo object and mongo.db refers to the database specified in the connection string (myDatabaseName).
Similarly, when i use it with Django then mongo is the connection (https://github.com/pauljolsen/django-and-mongo-datatables/blob/master/mysite/polls/views.py#L59) and the database name is determined with the initialization of the connection (https://github.com/pauljolsen/django-and-mongo-datatables/blob/master/mysite/polls/models.py#L4).
so bottom line is the database has already been determined with the connection, self.mongo refers to the pymongo object and self.mongo.db is the database (not named "db", but whatever specified in the connection).  hope this helps!

Hi! Thank you for clearing that up, I wasn't ware of that!


DocsArchive-<IP_ADDRESS>Auto archive content

Please don't merge this PR until target archive repo PR https://github.com/OPS-E2E-PPE/docs-archive-test-target/pull/767 is merged into live branch.
Auto archive content to https://github.com/opstest2/docs-archive-test-target.git

Docs Build status updates of commit cf7e8b7:
Pending

Docs Build status updates of commit cf7e8b7:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error-GitBranchDeletedOrForcePushed] Cannot sync git repo to specified commit because branch Release_Archive_master_2021-02-02-08-24-59 has been deleted or has been force pushed

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel


replace appsody build with buildpacks

The build step presently uses the appsody binary for its implementation, and makes use of custom "stacks" referenced through the repository boson-project/stacks.   Appsody offers many features which are not applicable for our "faas" context, and can be omitted for speed improvements and clarity.  Additionally, breaking the dependency on a locally installed binary is of course preferable.  The tentative choice of technology that has emerged is buildpacks.
We would ideally replace the appsody.Builder implementation in ./appsody/builder.go (which implements the faas.Builder interface) with a buildpacks.Builder implementation in ./buildpacks/builder.go.  This could utilize the libcnd library https://github.com/buildpacks/libcnb or similar.
Other references that might help are https://buildpacks.io/docs/install-pack/ and https://github.com/lance/buildpacks

some progress: https://github.com/matejvasek/buildpacks/commit/6ebf74e719f67a80d8c84787762c33b74ce1e5e5
https://github.com/matejvasek/faas/commit/825e1608690bcf8f1144e2277d49b9e92bb8413c


EL-569 Workaround for Puma/k8s dropped request problem

See: https://github.com/puma/puma/blob/master/docs/kubernetes.md#running-puma-in-kubernetes
Link to the Jira ticket
Describe what you did and why.
Added the sleep in preStop, to delay the point when Puma ignores incoming requests, so that it is definitely after the point when nginx stops sending it requests.
Checklist
Before you ask people to review this PR:
Tests and rubocop should be passing
Github should not be reporting conflicts; you should have recently run git rebase main.
There should be no unnecessary whitespace changes. These make diffs harder to read and conflicts more likely.
The PR description should say what you changed and why, with a link to the JIRA story.
You should have looked at the diff against main and ensured that nothing unexpected is included in your changes.
You should have checked that the commit messages say why the change was made.

On the UAT env I tried a rolling update:
kubectl -n $NS_UAT rollout restart deployment el-569-puma-k8s-wo-laa-estimate-eligibility
During which I hammered the home page: https://el-569-puma-k8s-wo-check-client-qualifies-legal-aid-uat.cloud-platform.service.justice.gov.uk/
And it worked all the time. So no worse and before, and potentially bettter


Beta release

Codecov Report
Base: 31.80% // Head: 31.79% // Decreases project coverage by -0.00% :warning:

Coverage data is based on head (254ec91) compared to base (86d04cd).
Patch coverage: 45.90% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##             beta     #604      +/-   ##
==========================================
- Coverage   31.80%   31.79%   -0.01%
==========================================
  Files          96       96
  Lines       11273    11276       +3
==========================================
  Hits         3585     3585
- Misses       7366     7369       +3
  Partials      322      322

Impacted Files
Coverage Δ

server/metadata/tenant_tracker.go
0.00% <0.00%> (ø)

server/services/v1/management.go
0.00% <0.00%> (ø)

server/metadata/tenant.go
35.64% <72.72%> (ø)

server/metadata/dictionary.go
51.28% <85.71%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


IntelliJ Sync issues opening the plugin

I'm trying to contribute towards the plugin, specifically for CLion. However, everytime I import the workspace and import the clwb project view, I get a lot of red lines and missing imports when looking at the CLion plugin source, making the IDE useless. I've tried both Ubuntu and macOS, on Bazel 0.29.1. Any ideas?

@jin also tried it at BazelCon and got similar issues.

Oh, I forgot that you have to set up the correct IntelliJ/CLion SDK in the project structure to make this work.
Specifically: https://www.jetbrains.org/intellij/sdk/docs/basics/getting_started/setting_up_environment.html

Yes, setting up the correct IntelliJ/CLion SDK is important. As there hasn't been any comments since Dec 2019, I assume that this issue isn't current anymore and can be closed.


Pattern replace to add a prefix

Thanks for this open source project.
I am facing an issue whereby some json like messages are having numeric filed names that break the final avrò schemas
I would like to use the pattern replace transform to add a prefix/suffix to all fields that are numeric.
I don't know how many fields will satisfy this property so i cannot use a regular ReplaceField to rename those fields in advance
Can this be achieved with something like
pattern: "/d"
Replace with: "prefix_$1_suffix"
?

here an example of the java matcher class: https://www.tutorialspoint.com/javaregex/javaregex_matcher_replaceall.htm
seems like regex groups could work: https://github.com/jcustenborder/kafka-connect-transform-common/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/transform/common/PatternRename.java#L76
this way also this should work:
{
  "transforms" : "patternRename",
  "transforms.patternRename.type" : "com.github.jcustenborder.kafka.connect.transform.common.PatternRename$Value",
  "transforms.patternRename.field.pattern" : "^[^\d]*(\d+)",
  "transforms.patternRename.field.replacement" : "prefix$1"
}


Remove kremlin submodule

fixes #362

Looks like the ci script.sh does this. Then this probably shouldn't get checked in in the first place?

This never should've been checked in -- I don't know why @Frosne had a submodule there, and why it was part of her PR. This must have been added after I reviewed the code, or maybe I missed it. Removing this it the right fix, which I understand is what your patch is proposing. I also see that the CI, which is designed to deal with these issues, has removed it: see https://github.com/project-everest/hacl-star/commit/d78d9c4d871a1a371b770fc053f32e17b0e2f6a6 and search for "submodule" (I couldn't figure out a way to send a direct link to the relevant part of the diff).
Can you confirm that everything is back to normal?

Can you confirm that everything is back to normal?

My local copy is a little messed up now after toying with this. But the git pull on CI works again. Locally it's fine after manually patching up git too. So I think this should solve it.

Closing since the CI job removed that git submodule before I could merge this PR


Fix issue with TabPaneFooter Width/MinWidth not being respected

Description

The layout of the TabView didn't take into account changes of the Width/MinWidth property of the TabPaneFooter. This PR changes that and listens to changes made to those two properties.
Motivation and Context

Closes #1615 , not sure if would close issue #3280
How Has This Been Tested?

Add new API test, also checked in the tear out sample of the XCG for future testing of that scenario (e.g. new titlebar api for TabView)
Screenshots (if appropriate):

/azp run


LEGO: Pull request from lego/hb_04604851-bac4-4681-9f74-73de611d6e48_20220303163807319 to main

LEGO: Pull request from lego/hb_04604851-bac4-4681-9f74-73de611d6e48_20220303163807319 to main with localized lcls

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 48.278%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
14819

Relevant Lines:
29138

💛  - Coveralls


stars can get slow sometimes

some of the stars are out of view, and so it returns an error, and i have it set up to just move on if said error happens, but, it then goes and has this giant line of invalid stars that it keeps trying to set every loop, and then itll add new ones, until 90% of the stars it's trying to set are out of view
essentially i need to make it so that if trying to set a star returns an error, it removes that star from the save file

oh this was fixed lmao


Treat port env vars as expandable only if they self reference

Addresses #348
Provide additional port test case.
Also, update newest OS X versions build flags.

+1. This also resolves #182.

thanks, i'll look into it

@jflatow do you have a Makefile example with the issue you described? i'm having a bit of trouble replicating it

Sure, for example create a simple app called bar, with rebar.config:
{erl_opts, [debug_info, warnings_as_errors, warn_export_all]}.
{deps,
 [{'discodb', ".*",
   {git, "https://github.com/jflatow/discodb.git", "master"}, [raw]}]}.
{pre_hooks,
 [{compile, "make -C $REBAR_DEPS_DIR/discodb erlang"}]}.

And for completeness, src/bar.app.src:
{application, bar,
 [{vsn, []},
  {mod, {agency, []}},
  {applications, [kernel, stdlib]}]}.

I tried out your example but still can't reproduce it, can you paste the error you're getting and the output of rebar --version you're using? thanks!

On anything from b816c69e343c8fd757c59fe8703eeda597f4da0a...9281b848f563e4928b565fab57b610c7368c0be6 I get an error:
==> erlang (compile)
Compiling c_src/ddb.c
c_src/ddb.c:9:10: fatal error: 'discodb.h' file not found
#include <discodb.h>
         ^
1 error generated.

On 022afd4b08fe024520849e50fd2bd03294fe210a or earlier, it works.

got it, i see it now, i was using the 2.6.1 rebar that's in the path, that one doesn't have the regression

c43f2e55 revert fixes the regression


Update commit  convention to follow Google convention

Description
As an open source project commit messages are important for open source contributors to understand
Suggested solution
We could follow Google's commit convention in Angular.
Alternative
No response
Additional context
No response
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

[ Triaged ] Good to start developing


⚠ ClamAV has degraded performance

In 116f496, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 50 ms

Resolved: ClamAV performance has improved in 6cbe0bb.


Improve installer

Removed cocoapods dependency (Lets end users manage this if they need it, and not all users need it)
Install kubectl from binary (No longer requires Xcode to compile)

These changes only impact mac installer, not linux or windows.


Issue on Scaffold Genration

I am using laravel 5.5 and php7. I installed Scaffold Interface package.
Its working fine.  Dashboard roles and users everything is fine.
But i lick on Scaffold Interface means i am going to create a CRUD .

click on any button nothing happens. and only one default value is there i filled i got this error

Please help me.
Advanced Thank you.

@kjareena Thanks for submitting this issue, it's graph relation table issue, but don't worry, the entity that you have been creating is completely working!!
I've already mention this issue as a bug in #114 .


Snabb config get-state command

This PR introduces a snabb config get-state which will be used to retrive the state data (counters) of the lwaftr. The command allows for specification of a path to get  subset or individual counter.
There are a small number of things left to do at the time or writing this PR:

[ ] Fully test functionality
[ ] Narrow down using path commands
[ ] Produce good documentation (small stub documentation from a previous version exists)
[ ] Create selftests, specifically for lib.yang.state

As part of this PR, I've also introduced the counters to the lwaftr. The way it works is any leaf in the yang schema with yang config false has its leaf's name compared against the available counters (which have been found by searching under the PID/apps directory). This somewhat limits and hinders how our state schema looks.
I think in the future we may wish to change this so that the counter name is a path to a schema leaf and the get-state code looks based on the full path. This would allow for a nested, more yang-ish state schema.
@wingo

This has been closed as it as made into another PR (PR #593), that PR has been successfully merged.


Add Erase Function

set color is transparent if you want erase line for iOS

Thanks for your contribution!


[RESTEASY-1998] Prepare test for resteasy.preferJacksonOverJsonB property for client on deployment

Prepare test for resteasy.preferJacksonOverJsonB property for client on deployment
This PR replaces https://github.com/resteasy/Resteasy/pull/1667
Jira: https://issues.jboss.org/browse/RESTEASY-1998
Master PR: https://github.com/resteasy/Resteasy/pull/1732

@tterem thanks for this PR and the master one. Please see my review on the other one.


Merge DSL2 into dev branch

This PR merges the current up-to-date DSL2 version into the branch, so we can prepare for releasing v2.0.0
Since we are note planning a new release before going for DSL2, I don't think we need to be too careful here. The functionality of both versions should be the same. In the end we just need to test the DSL2 version a lot before releasing it.

Thanks for the review @mashehu!
Will address these things later.


Gitab registry: always using port 5000, ignoring the service config port

Describe the bug
By default the gitlab registry port is set to 4567 in services.gitlab.registry.port, but in my tests the registry always uses port 5000, whatever is configured.
Steps To Reproduce
Enable the registry with any port, except 5000:
{
  services.gitlab = {
    enable = true;
    registry = {
      enable = true;
      port = 4567;
      certFile = config.sops.secrets.gitlab_registry_cert.path;
      keyFile = config.sops.secrets.gitlab_registry_key.path;
      externalAddress = "registry.example.com";
      externalPort = 443;
    };
  };
}

Verify:
$ curl -I http://localhost:4567
curl: (7) Failed to connect to localhost port 4567 after 0 ms: Couldn't connect to server

$ curl -I http://localhost:5000
HTTP/1.1 200 OK
Cache-Control: no-cache
Date: Tue, 22 Oct 2024 08:28:28 GMT

Expected behavior
Should use the port configured :)
Notify maintainers
@globin @krav @talyz @yayayayaka
Thanks again <3
Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.6.57, NixOS, 24.05 (Uakari), 24.05.20241017.4eb33fe`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.8`
 - nixpkgs: `/nix/store/z7j461v8da67nwrad4xz50dx2l5wk0wz-source`

Add a :+1: reaction to issues you find important.

have you opened the port in networking.firewall.allowedTCPPorts ?

No because I use nginx as a proxy, but in the "steps to reproduce" I run the queries directly from the host:
$ curl -I http://localhost:4567
curl: (7) Failed to connect to localhost port 4567 after 0 ms: Couldn't connect to server

$ curl -I http://localhost:5000
HTTP/1.1 200 OK

This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/gitlab-registry-and-certfile/54947/4


[BEAM-6106] Revert "[BEAM-5320] Finish Python 3 porting for testing module (#7029)"

This reverts commit 772fa67d62703f9462f1ee97ae7a3b94e8655659.
Breaks python post commit tests.

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

Run Python PostCommit

LGTM

It would be helpful in the future to link to a failing test so someone can easily review.

(if postcommit turns green here that is eve more important, of course)

Here you go: https://builds.apache.org/job/beam_PostCommit_Python_Verify/6626/console
For reference:  the PR description has a link to failing post commit test in the table and the linked JIRA has a link too.

Run Python PostCommit


Select Area just doesnt seem to work at all

I setup my brush:
Size 1
Shape Cube
Type Fill (or overlay)
Mode Selection
Switch to selection item
select position 1 then position 2 (IN A STRAIT LINE) a few blocks later
the brush window comes up again
hit submit
Nothing Happens
Then I click on the ground with the brush tool
and a 3x3 stone layer pops up that is not aligned with my selected region.
I guess I just dont get it.  how do I make a simple line of blocks?

Sounds like a bug. Does creating a bigger cuboid area work, or is that broken too?


Corrected grammar

"8 or less bit columns" is incorrect. It should read "8 or fewer bit columns" since column is a count noun, not a mass noun.

@Adam-Brown : Thanks for your contribution! The author, @MikeRayMSFT, has been notified to review your proposed change.

#sign-off


Fix for integrated_bioprocess.py

Thanks for your fix. A few other files seem to be needed to be changed as well (https://github.com/BioSTEAMDevelopmentGroup/Bioindustrial-Park/tree/master/biorefineries/cellulosic/systems/fermentation), for example in integrated_bioprocess.py, DAP load of seed train seems to be 0.33 g /L as well (while it should be 0.67 g /L).

@zasddsgg, got it! All fixed across all files. Feel free to reopen the issue if I missed anything.
Thanks,


Add HTML formatted reports from templates for dashboard

The reports should generate HTML and machine readable (json) formatted reports in dashboard format to be exposed through any web server containing both summaries and all enriched data. It should provide drill downs to detailed pages, expose messages, events, failure reasons and capture the overall features and capabilities of all operators. It should serve as the foundation for all level data.

This is redundant is will be addressed by the much wider issue https://github.com/opdev/opcap/issues/77


Update dependencies and workflows

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.00%. Comparing base (ca1b546) to head (d7d0cb4).

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #675   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           74        74
  Lines         2065      2065
  Branches       254       254
=========================================
  Hits          2065      2065

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Let the TEST_JAVA_HOME env variable set the JVM used to run tests in gradle/gradle build

Context
As part of dogfooding the Gradle Kotlin DSL, we need to run the gradle/gradle build with Java >=8. This change is a first step toward this and let the TEST_JAVA_HOME environment variable set the JVM used for test execution.

CI is green

@big-guy we want to:

always run the gradle/gradle build with Java 8
keep compiling using Java 7
choose what JVM (version, vendor) to use to run tests depending on the ci job

This PR introduces a way to do #3.
Before that change, the tests are run with the same JVM as the one running the outer build.
Does that answer your question?

As a style/consistency issue, unless there's a reason that env vars work better, we should follow the way we do java7.home and use a system property instead.

@wolfs this is ready for another round of review

Thanks!

As soon as the CI configuration is adjusted so it passes the testJavaHome I guess we can retire the java9Test tasks, right?

With some other adjustments, yes! I'll do that as a follow up.


Floating point exception after upgrading from 2.13.1 to 2.16.1

Hey-hey!
We've just tried to upgrade our Heroku installation to the latest version and faced the following issue:

Unfortunately, do not have any other information for now.
/cc @pcreux

Looking at the logs, it seems that the following image was "Started" four times but never completed: https://gist.githubusercontent.com/pcreux/d067cfc0ddb5d3071cebc48d62b4e43e/raw/922da3142f6d71124f53c358dc17f205d37ce98d/ecctajm403r91uztkycx047ihjgf.webp
Let me know if you can reproduce that issue.

@pcreux Yup, you found it! I'll try to dig it.

The source of the issue was fixed in libvips by this PR: https://github.com/libvips/libvips/pull/1935. I updated dependencies in the Docker image, so v2.16.2 works as expected.

And thanks for the sample!

Thank you @DarthSim !
Would it make sense to version the docker images? I find it a bit counter-intuitive that re-deploying the same image on March 4th would fix this issue. Also, if the updated dependency introduced another bug (say, for animated heic with squirrels) we have no way to easily deploy the March 3rd image.
Do you have any recommendations to prevent such errors from crashing imgproxy on Heroku? In our case, Heroku gave up after imgproxy crashed twice in a row. Could imgproxy handle and recover from such errors? Or could it (somehow) manage a cluster of processes and restart them on crash? Or is there a docker image around with extra tooling (nginx, upstart?!) to keep imgproxy alive?

Would it make sense to version the docker image? I find it a bit counter-intuitive that re-deploying the same image on March 4th would fix this issue. Also, if the updated dependency introduced another bug (say, for animated webp with squirrels 😅) we have no way to easily deploy the March 3rd image.

What do you mean? Docker images are versioned: https://hub.docker.com/repository/docker/darthsim/imgproxy/tags?page=1&ordering=last_updated

Do you have any recommendations to prevent such errors from crashing imgproxy on Heroku? In our case, Heroku gave up after imgproxy crashed twice in a row. Could imgproxy handle and recover from such errors?

imgproxy catches and handles every error it can catch. Unfortunately, this kind of error can't be caught. The good news is that this kind of error is very rare.

Or could it (somehow) manage a cluster of processes and restart them on crash? Or is there a docker image around with extra tooling (nginx, upstart?!) to keep imgproxy alive?

Well, this is the responsibility of the system where you launch your app, in your case - Heroku, but Heroku lets you down here. Upstart and friends won't help you either. However, you can use a wrapper-script that will restart a process every time it exits with non-zero status:
until imgproxy; do
    echo "imgproxy crashed with exit code $?.  Respawning.." >&2
done

You need to build a Docker image on top of imgproxy's one, add this script to it and use it asENTRYPOINT. This should be quite enough to handle crashes without stoping a dino.

Hey @DarthSim, thank you for your reply!

Docker images are versioned

Well, they are... but the Docker image tagged v2.16.2 was replaced on March 4th with updated dependencies. I would expect a tagged image to be immutable to be confident that I'm deploying the exact same image across environments.

Heroku lets you down here

Not the first nor the last time. :D
until imgproxy; do is much simpler than upstart. I'll give that a try.
Thank you!

Well, they are... but the Docker image tagged v2.16.2 was replaced on March 4th with updated dependencies.

No, it wasn't. Updated deps is the only diff between 2.16.1 and 2.16.2, you can check it yourself in commits.

Oh, @DarthSim, I'm so sorry, my mistake. I thought that I was already on 2.6.2.


Set precise date for v3.6.0 release

@hugovk, I'm sorry, I meant to update the previous history PR before merging it to master, and I messed up, hence this additional PR.


Exercise does not update solution

What is wrong?
After getting a solution  reviewed and checked out, the corresponding exercise continues to be an old one.
query investigateSolutionBug {
  solution: uuid(id:265954)  {
    __typename
    ... on Solution {
      currentRevision {
        id
      }
      exercise {
        id
        currentRevision {
          id
        }
      }
      revisions {
        nodes {
          id
        }
      }
    }
  }
  currentSolutionRevision: uuid(id:265955)  {
    ... on SolutionRevision {
      content
      repository {
        id
      }
    }
  }
  exercise: uuid(id:90208) {
    ... on Exercise {
      currentRevision {
        id
      }
      solution {
        id
        currentRevision {
          id
          content
        }
      }
    }
  }
  currentExerciseRevision: uuid(id:265953) {
    __typename
  }
}

{
  "data": {
    "solution": {
      "__typename": "Solution",
      "currentRevision": {
        "id": 265955
      },
      "exercise": {
        "id": 90208,
        "currentRevision": {
          "id": 265953
        }
      },
      "revisions": {
        "nodes": [
          {
            "id": 265955
          }
        ]
      }
    },
    "currentSolutionRevision": {
      "content": "{\"plugin\":\"solution\",\"state\":{\"prerequisite\":{\"id\":\"1337\",\"title\":\"Vierecke\"},\"strategy\":{\"plugin\":\"text\",\"state\":[{\"type\":\"p\",\"children\":[{\"text\":\"\"}]}]},\"steps\":{\"plugin\":\"rows\",\"state\":[{\"plugin\":\"text\",\"state\":[{\"type\":\"p\",\"children\":[{\"text\":\"Raute, Quadrat\"}]}]}]}}}",
      "repository": {
        "id": 265954
      }
    },
    "exercise": {
      "currentRevision": {
        "id": 265953
      },
      "solution": {
        "id": 154856,
        "currentRevision": {
          "id": 158963,
          "content": "{\"plugin\":\"solution\",\"state\":{\"prerequisite\":{\"id\":\"1705\",\"title\":\"Vierecke\"},\"strategy\":{\"plugin\":\"text\",\"state\":[{\"type\":\"p\",\"children\":[{}]}]},\"steps\":{\"plugin\":\"rows\",\"state\":[{\"plugin\":\"text\",\"state\":[{\"type\":\"p\",\"children\":[{\"text\":\"Füge hier gerne eine Lösung hinzu!\"}]}]}]}}}"
        }
      }
    },
    "currentExerciseRevision": {
      "__typename": "ExerciseRevision"
    }
  }
}

Notice that the exercise.solution.currentRevision.id is not the same as solution.currentRevision
Steps to reproduce
Unfortunately it is not reproducible so far. But anyway the problem remains: how to make this exercise to get its correct solution
Worths investigating
At the same time the exercise was edited, maybe it is hanging on the old exercise revision.
Reported by @AndreasHuber

For documentation:
The fix made in this ticket so far makes sure that from now on no new solution is created if there is already a solution.
When the user creates a new solution, there is a small chance that they get a "solution already exists" error if someone already made a solution but it is waiting for review and therefore not yet displayed under the exercise. The frontend may present this error case to the user in an understandable way or create a new revision instead of a new solution.
To fix the existing cases of several solutions per exercise in the database, there is https://github.com/serlo/db-migrations/issues/7.
In general, saving solutions and exercises separately has a number of drawbacks without a known advantage compared to saving the solution as part of the exercise. It is a plan already made before to change this at some point, though I cannot find an issue for it at the moment.


PF - Handle all fencetypes

Handles all fence types, and not just oak.
Note that the fence handling code will eventually be better written and moved out of IsSolid.

lgtm.

Thanks.
You should have waited for Appveyor, I think.

Sorry, won't happen again.


SubFlow Execute Twice

I added a subFlow，and scheduled both flow and subFlow with different trigger time，for example，flow schedule is 0 0 2 ? * *，subFlow schedule is 0 0 3 ? * *.
But jobs in subFlow executed at 2 o'clock，and re-executed at 3 o'clock，flow didn't wait for subFlow finish running.

I find it too. I just want to run my flow once at an accurate time，which like 0 0 10 19 12 ? 2019.
But  it will just run again and again(not only twice). I think its a big problem.

I added a subFlow，and scheduled both flow and subFlow with different trigger time，for example，flow schedule is 0 0 2 ? * *，subFlow schedule is 0 0 3 ? * *.
But jobs in subFlow executed at 2 o'clock，and re-executed at 3 o'clock，flow didn't wait for subFlow finish running.

Please confirm whether the pipeline configuration in the concurrent settings is reasonable


Improve usability of snapshots

@brian-brazil

:+1:
I'll be interested to see how you word this in the docs, as we'll still want some freedom to change internals like this in the future.


Please invite me to the GitHub Community Organization

Name
Donald
Discord Username (if applicable)
.JK
Additional Context
I'd love to join this organization and collaborate with you

It's great having you contribute to this project
If you would like to continue contributing to open source and would like to do it with an awesome inclusive community, you should join our Discord Server and our GitHub Organisation - we help and encourage each other to contribute to open source little and often 🤓 . Any questions let us know.

Invitation sent for the GitHub Organisation. Welcome to the community 🎉#
Don't forget after accepting to make it public so it appears on your GitHub profile for everyone else to see, you can do this by finding your name in the GitHub organisation list and change the dropdown to public https://github.com/orgs/jaidevstudio/people
Like this

Tips for practising:
Remember contributing to open source is not just about code, its about collaboration, communication and adding value
I hope that helps

Please check your email or organisation page 🙏


fix(appSec): main account S3 Key separation

Issue #, if available:
Description of changes:
Checklist:

[x] Have you successfully deployed to an AWS account with your changes?
[x] Have you written new tests for your core changes, as applicable?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Looking at the SWBStack, I see cfn_nag suppressions marked as TODO's. Any idea when are these going to be addressed ? It might be worthwhile to revisit them, I see some of them are no longer relevant for e.g.: https://github.com/aws-solutions/solution-spark-on-aws/blob/janeyu/key-rotation/solutions/swb-reference/src/SWBStack.ts#L785


ubpm: init at 1.7.3

Description of changes
Universal Blood Pressure Manager
https://codeberg.org/LazyT/ubpm
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Do you know why the Darwin builds are broken ?

Have no idea - I don't have mac to check it. Probably I should exclude it?

Darwin should be supported based on what it said on their website.

Thanks for the review. Well, I still have this derivation in my personal configs and sometimes use it, so I don't mind fixing it and moving to nixpkgs - maybe someone else will benefit from it. It's just I'm not sure what's wrong with macs and how to fix it.

No problem, let's fix the other issues and we'll merge it as it is. If someone needs it for apple, another PR will have to be provided.


Update minishift to 1.0.0

[x] brew cask audit --download {{cask_file}} is error-free.
[x] brew cask style --fix {{cask_file}} reports no offenses.
[x] The commit message includes the cask’s name and version.

Thank you for the contribution. It was merged directly as 6c4e76b7f5e4e5e0a458b7e9b8170c06518a4201. Your contribution is still credited to you.


Copy Kokoro continuous build configs to be by branch name

This introduces new build configs that will be identified by their branch name. Because we're moving to having multiple persistent branches on which we want CI, we need to distinguish between these with multiple Kokoro jobs (with multiple statuses).
Note that there is no actual configuration in these configs, since the common config has everything they need.
Also changed comments in all of these configs to refer to the Google source repository as "internal" instead of "upstream".
Part of https://github.com/google/iree/issues/2279

Looks like continuous.cfg files will be removed once everything is switched over?

Correct.


Lifecycle Hooks

@barneycarroll i noticed your mithril.exitable.js repo for animation hooks and also remember you asking a while back how I would handle animating on removal. df2e45cb75caf30b07912624d01e84a3aa2c06ae is the result.
lifecycle hooks:

nodes: {_hooks:} - will/didInsert, will/didReuse, will/didMove, will/didRemove
views: vm.hook() - will/didRedraw, will/didMount, will/didUnmount

will* node-level hooks allow a Promise return and can delay the event until the promise is resolved, allowing you to do what you need to CSS animate, etc.
will* view-level hooks are not yet promise handling, so cannot be used for delayed anims yet, but you can just rely on the view's root node's hooks to accomplish the same thing, just not externally like you can hook vms.
there is no redraw-locking during pending promise resolution right now, nor are multiple promises coalesed by .all() or similar. so there's a possibility of getting the vtree into a confused state if there are complex staggered/unaligned animations of sibling nodes. this has not been tested. how do you handle this in mithril.exitable.js besides global redraw lock (& queue?) ?
here's a demo [1] of all the hooks firing (color coded by type). didUnmount() is excluded since it still has some issues WRT the fact that it does recursive bottom-up child removal.
[1] https://leeoniya.github.io/domvm/demos/lifecycle-hooks.html

the demo is a bit rough as it requires directly interacting with DOM eg node.el.classList.remove("removing");. this may be okay, but ideally you should be able to use a non hook-invoking version of vm.patch(node, {style: ...}). I'm thinking of a good way to get this done since patch takes oldNode/newNode, so would require a somewhat clumsy vm.patch(node, [node.tag, {style: ...}, node.body]) that would also patch the children and invoke hooks, etc.
i like the terseness of the declarative style of snabbdom [1] for animations that could eliminate a lot of the rAF boilerplate in the demo if I implemented them as sugar over the hooks. they're more limited in what they can do vs raw hooks but likely cover 85% of what you'd need them for.
i was considering adding syntax like this to the style object: _wr-opacity: [initialVal, rafDelayedVal] (willRemove opacity values). or _wr-class: [initialClass, rafDelayedClass] to the props object, but it's a lot of syntax pre-processing overhead.
kinda interested in seeing how nicely i can reproduce the reorder-anim demo [2]
gonna sleep on it..
[1] https://github.com/paldepind/snabbdom#delayed-properties
[2] http://paldepind.github.io/snabbdom/examples/reorder-animation/

Hiya @leeoniya, in answer to your question, Exitable simply freezes global draw. This means that any changes to the view model over the course of an exit animation (including other exitable components disappearing and triggering their own exits) is batched and takes course during the first redraw after the last animation.
Your question prompted me to reflect on whether this really was the best strategy. It's not difficult to imagine somebody wanting their application lifecycle to carry on undisturbed while exit animations occur. Surely something must be possible with { subtree : 'retain' }? Well at this point we quickly realise that Mithril is not the ideal library for that level of control. The problem returns to the fact that view focused libraries that are agnostic as to model structure can detect a change in the view model but they can't infer the semantics of its cause. From a user perspective, a modal slowly fading away while they carry on working on editing their todo list sounds like an obvious, trivial thing. But we're not in a position to be able to determine that that's obvious: as far as we know, the thing animating out could be one of the todos. But then, how do you resolve new list states. including order changes, with the fact that an old item must remain while it exits? Besides, Mithril is intrinsically global in its treatment of the view: when you go against that, you raise all sorts of conflicts with no procedural answers. So basically, saying "no" to redraws while animating out saves you from a can of worms :)
Snabbdom is great. I hadn't seen the delayed property, but that's ingenious. Expressing one state and the next in a single stateless snapshot sounds like heresy, but it's so useful. Previously I had been keeping drawn flags in my components, writing imperative code in my views to toggle them, and writing ternary style properties. That's really neat though. Having said that, it's actually confusing to read a virtual DOM expression and understand that the immediate style properties could only be a starting point — if anything, you'd expect the persistent styles to be the ones immediately under style and the disposals initial, disposable ones to occupy a non-standard hook. After all, the point of this special API hook is to define starting points for animations — if something fades in you don't want it to be visible on first draw — but that starting point is the exception to the rule; if I find I need to tweak appearance, I shouldn't be thinking "right, I should probably look under style.delayed".
TBH Snabbdom doesn't really need that convoluted hook. Distinct starting styles are an implementation detail that can and should be safely tucked away somewhere more appropriate — the obvious place for this is willInsert: since any virtual node is entirely disposable, modifying if once on that initial hook is exactly what's needed.
Your distinction of 'move' as a lifecycle hook is interesting. One thing I think everybody can agree on is that detecting an insertion in Mithril is a terrible API.  So in a way, it's admirable that I wouldn't have to write my own logic to determine that something has moved in DOMVM. It does raise some questions, for instance "where was it before?", which one would imagine to be pertinent. The problem right now (as far as I can see) is that in order to preform a movement animation, one would need to record a reference to the old live DOM in 'willMove' and then determine the right animation conditions with reference to those in 'didMove'. Is that a problem? Should conditional modifications occur in the otherwise generic lifecycle method signature? Not sure. I am currently working on a bunch of example animated virtual DOMs for Exitable, which might help us with questions like that. Arguing about API ergonomics without use case references is difficult ;)
As far as calling your demo dirty, I don't think direct live DOM modification in application code is a sign of failure in the virtual DOM API. It's only a problem if live changes linger on in a way the library isn't aware of that cause problems later on. The notion that different states of CSS can express all desirable animations is naive — as a designer, CSS transitions are often not enough, and when they aren't it's horrible to try and shoehorn intended behaviour into them. Unless the virtual DOM library is saying it can account for every implementation detail of the whole animation — which to be frank is a terrible idea — then I think it's essential and intuitively obvious to expose the live DOM so that an external library can do the work.

thanks for your feedback!

It does raise some questions, for instance "where was it before?"

yes, this is on the todo in fact and should be trivial to feed to the did/willMove hooks as args. right now all node hooks get the node itself as the first arg. i didnt wamt to commit to wildly dissimilar hook sigs beyond that just yet. i also had the initial thought to feed it oldIndex as arg #2 into the will/didmove. will/didReuse already gets oldNode, newNode so you can do your own diff to determine if it's an actual "update" or not. i have a commint in the demo that says DIY diff :) a d does just this (diffs content) since domvm doesnt do eager diffing before patch but does them deeper and lazily.
re snabbdom, the delayed css declarative style looks very tempting to me and very easy to add as sugar but already i see how limited it is. in my very basic demo i turn the div red pre-remove then in the nexttick must kill that same class to fade out. only the second action corresponds to the "delayed" and the first on is not the initial class, so you hav to use a hook anyhow. instead of just delayed you need an immediateDelayed and a delayedDelayed or something else splot it into prev/post hooks or other hacky things which always rear their head in overly-declarative libs/DSLs.

Hooks slated for renaming for clarity WRT actual DOM ops performed and vdom terminology:
Reuse -> Recycle
	willRecycle(node, newNode)
	didRecycle(node, oldNode)
Move -> Reinsert
	willReinsert(node, newIdx, newParent)
	didReinsert(node, oldIdx, oldParent)

Right now, reinsert can only happen within same parent so new/oldParent will always match, but could differ in future for transplanting components, etc. Dunno yet.
I think I'll init a "hook ctx" that persists between matched will/did hooks. This'll let you remember any prior state from will* hooks for use in did* hooks without having to create an external retention mechanism and allows you to keep/reuse a hooks object without recreating the hooks for every single node.
Question is, how to pass this ctx through to hooks?

As arg? which? 1? 2? Hurts ergonomics, where would you stick in above examples?
As the this not to pollute args? Too magical?

willRecycle(ctx, node, newNode)
willRecycle(node, ctx, newNode)

ctx is optional, and i dont wanna burden arg 1 with an optional thing for the sake of restArgs uniformity/beauty. Leaning towards this, despite disliking it.

Isn't this ctx redundant too? I would keep all my view data on the higher scope view model.

For view-level hooks it is. For node-level hooks it isnt if they're static and shared. Consider:
function MyView(vm, data) {
	// shared node hooks
	var nodeHooks = {
		didInsert: node => {...},
		willRemove: node => {...},
	};

	return () =>
		["ul", data.map(item =>
			["li", {_hooks: nodeHooks}, item]
		)];
}

In each hook invocation you may want a fresh context for the current node. You could technically piggyback on the node itself by assigning your own custom prop to it like node.__myCtx = {}; and this will of course still be accessible in both will and did. But there's still extra management with cleanup, etc. and poor interaction/collisions with other hooks. On the other hand, this would be a good hack if you did in fact need cross-hook communication outside of matched will/did pairs.

I don't understand. Why is the vm not suitable?
The point is the user has several opportunities to get hold of unique identifying objects. These can be used directly as storage — vm.state = 'foo', or used as keys in maps if the user is afraid of pollution or collision — states.set( vm, 'foo' ).

I don't understand. Why is the vm not suitable?

Yes, one can always do the bookkeeping manually in the closure. I would not set anything on the vm that you dont explicitly want to expose to the outside world. There's actually a pre-initialized empty export object vm.exp.* to expose internal view APIs or state (for suggested uniformity and collision avoidance).
function MyView(vm, data) {
	// some per-node stuff to remember between hooks
	var wmNodes = new WeakMap();

	// shared node hooks
	var nodeHooks = {
		willInsert: node => { wmNodes.set(node, {someProp: "mooo"}); },
		willRemove: node => { wmNodes.get(node).someProp; },
	};

	return () =>
		["ul", data.map(item =>
			["li", {_hooks: nodeHooks}, item]
		)];
}

I'm not sure how much a ctx would improve ergonomics if it's only limited to matched will/did hooks. It seems more useful to have cross-hook retention as above. So ctx would be most useful if it was already a flat vm-level weakmap of all nodes. seems expensive and unnecessary tho.

86dc536512c180d2ead3955ec6932988397ecb55 renames the Reuse and Move hooks.

What prompted 'reinsert' instead of 'move'?

reInsert communicates the fact that it gets detached from the dom more clearly, IMO. and also matches the internal implementation of using insertBefore for both initial insert/hydrate during mounting and re-insert/reorder on redraw. for node-level stuff I want to match the underlying DOM terminology as closely as possible. obviously, "recycle" is vdom specific.


Check for Secure connection should be configurable

Is your feature request related to a problem? Please describe.
When I ran Flask app + Authlib behind Nginx SSL with proxy_pass directive, I am still getting InsecureTransportError exception.
This is happening because App runs inside server on the "insecure" http, while web facing Nginx uses secure HTTPS and then proxies requests.
Describe the solution you'd like
I know, that we can use environment variable AUTHLIB_INSECURE_TRANSPORT=1. But in this case transport is still secure and there should be possibility to configure library in other way to work in such stacks.
For example it can be done with some native configuration to the library.
Additional context
Example nginx config:
server {
  server_name exmple.com;
  listen 443 ssl;

  location / {
    client_max_body_size 0;
    gzip off;

    proxy_http_version 1.1;

    proxy_set_header    Host                $http_host;
    proxy_set_header    X-Real-IP           $remote_addr;
    proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
    proxy_set_header    X-Forwarded-Proto   $scheme;

    proxy_pass http://localhost:5000;
  }

If you are using gunicorn, here is the configure:
https://docs.gunicorn.org/en/stable/settings.html#secure-scheme-headers

It is your wsgi server's responsibility to handle wsgi.scheme.


Replaced eval() with ast.literal_eval() for secure conversions

What does this PR do?
The PR fixes a security vulnerability which allows for executing arbitrary code on the target system in the context of the user running the program. The attacker can execute commands on the target OS running the operating system by setting the PL_TRAINER_GPUS when using the Trainer module.
Before:

In pytorch_lightning/utilities/argeparse.py at Line 124

        if not (val is None or val == ""):
            # todo: specify the possible exception
            with suppress(Exception):
                # converting to native types like int/float/bool
                val = eval(val)

After:

        if not (val is None or val == ""):
            # todo: specify the possible exception
            with suppress(Exception):
                # converting to native types like int/float/bool
                val = literal_eval(val)
            env_args[arg_name] = val

Before submitting

[x ] Was this discussed/approved via a GitHub issue? (not for typos and docs)
[ x] Did you read the contributor guideline, Pull Request section?
[ x] Did you make sure your PR does only one thing, instead of bundling different changes together?
[ x] Did you make sure to update the documentation with your changes? (if necessary)
[ ] Did you write any new necessary tests? (not for typos and docs)
[ x] Did you verify new and existing tests pass locally with your changes?
[ ] Did you list all the breaking changes introduced by this pull request?
[ ] Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review
Anyone in the community is welcome to review the PR.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

[x] Is this pull request ready for review? (if not, please submit in draft mode)
[x] Check that all items from Before submitting are resolved
[x] Make sure the title is self-explanatory and the description concisely explains the PR
[x] Add labels and milestones (and optionally projects) to the PR so it can be classified

PS: This vulnerability was first disclosed on huntr.dev

Hi @carmocca, as per our discussion on huntr.dev, can you please review the fix and mark the fix on huntr.dev accordingly? Thank You 😃

Will do! Just needs to be merged first.


Feat: Add codeium_ls_binary option

This patch adds an option which lets the user specify the path to the codeium language server binary.
This can be very useful for users who have already installed codeium on their system and would like to use it from the codeium.vim plugin.
Note: There might be a better option name for this !

Another attempt at this feature used the name g:codeium_bin, but I defer to you to decide which is better.

Another attempt at this feature used the name g:codeium_bin, but I defer to you to decide which is better.

codeium_bin is fine to me. I changed to that


create config files

Addresses #1

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@4a87dbd). Click here to learn what that means.
The diff coverage is n/a.

@@          Coverage Diff           @@
##             master    #2   +/-   ##
======================================
  Coverage          ?    0%
======================================
  Files             ?     5
  Lines             ?   255
  Branches          ?     0
======================================
  Hits              ?     0
  Misses            ?   255
  Partials          ?     0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4a87dbd...78a4c83. Read the comment docs.


Is the repo still alive?

The last release is from one year ago, is this loader still stable and\or ready to use? Should I use dnscrypt manually without loaders?

Hi, Absolutely. The project is still alive, but the dnscrypt-proxy code has undergone great changes and I need time to adjust the project. You can use dnscrypt-loader with old versions of dnscrypt-proxy without problems (try version 1.6) See the manual and more details on the Closed Issues tab.

Thank you very much for the insight! :relaxed:

The installation and use went fine =)
Is there an ETA for the updates needed to use the latest version of dnscrypt? Just to have an general idea :wink:


Inheritance isn't working

I tried using inheritance but I cannot make it work. I even made the file the smallest possible, here's what I have in ~/.ssh/assh.yml
hosts:

  common:
    IdentityFile: ~/.ssh/production.pem
    HostName: <IP_ADDRESS>

  aloha:
    User: ubuntu
    Inherits:
    - common

aash build:
# ssh config generated by advanced-ssh-config

# host-based configuration
Host aloha
  User ubuntu

Host common
  HostName <IP_ADDRESS>
  IdentityFile ~/.ssh/production.pem

# global configuration
Host *
  ProxyCommand assh proxy %h %p

and ssh aloha results in ssh: Could not resolve hostname aloha: nodename nor servname provided, or not known
I checked for syntactical error in the yaml file but I cannot see anything, is this a bug or an error of mine? :(

Actually, the inheritance isn't exported to ~/.ssh/config, it's only used when proxying to an host
I will try to export inheritances to ~/.ssh/config, I just need to benchmark it before :)

It's ok that it's not exported, but the proxy doesn't seem to be using it either because I don't reach the connection point. Or is it because of the lack of a HostName that I never reaches the proxy? Is there a current way you would recommend to make the sample config work?

Using the latest commit, you can see the computed host:
$ assh -D proxy test-bbb
DEBU[0000] Loading config file '/Users/moul/.ssh/assh.yml'
DEBU[0000] Loading config file '/Users/moul/.ssh/assh.d/hosts.yml'
DEBU[0000] Loading config file '/Users/moul/.ssh/assh.d/test.yml'
INFO[0000] Host test-bbb
INFO[0000]   ControlMaster auto
INFO[0000]   ControlPath ~/tmp/.ssh/cm/%h-%p-%r.sock
INFO[0000]   ControlPersist yes
INFO[0000]   HostName <IP_ADDRESS>
INFO[0000]   IdentityFile ~/.ssh/id_rsa
INFO[0000]   Port 22
INFO[0000]   PreferredAuthentications hostbased,publickey,keyboard-interactive,password
INFO[0000]   PubkeyAuthentication yes
INFO[0000]   ServerAliveCountMax 5
INFO[0000]   TCPKeepAlive yes
INFO[0000]   User ubuntu
INFO[0000]   VisualHostKey yes
DEBU[0000] Writing SSH config file to "/Users/moul/.ssh/config"
DEBU[0000] Trying gateways: [direct]
DEBU[0000] Connecting to <IP_ADDRESS>:22

hosts:
  test-aaa:
    HostName: <IP_ADDRESS>
    IdentityFile: ~/.ssh/id_rsa

  test-bbb:
    User: ubuntu
    Inherits:
    - test-aaa

defaults:
  ControlMaster: auto
  ControlPath: ~/tmp/.ssh/cm/%h-%p-%r.sock
  ControlPersist: yes
  EscapeChar: ~
  GSSAPIKeyExchange: no
  Gateways:
    - direct
  IdentifyFile: ~/.ssh/id_rsa
  Port: 22
  PreferredAuthentications: hostbased,publickey,keyboard-interactive,password
  PubkeyAuthentication: yes
  ServerAliveCountMax: 5
  ServerAliveInterval: 30
  TCPKeepAlive: yes
  User: root
  VisualHostKey: yes

Haha, I just discovered some new ignored keys because of case sensitivity :)

Hum, I found the issue, the inheritance correctly work in assh, however, some settings are needed in the .ssh/config file, i.e: User, IdentityFile, basically everything not related to HostName, Port, Gateway, etc
So I will finally need to compute each host before saving the ~/.ssh/config file :)


feat(headers): added header with tabs

Closes #260
Description
This adds a Header component example with tabs.
Is this a breaking change (Yes/No):
No

@taroj1205
In macOS X, when I open the menu, the display looks strange as shown in the image. Is this just my environment?

@koralle
This will be fixed in https://github.com/yamada-ui/yamada-components/pull/263.


WorldEnvironment is affecting editor's windows

Godot version
godot 4 beta 16
System information
Windows 11, Vulcan, GTX 1050 TI (driver geforce game ready 528.24)
Issue description
WorldEnvironment is affecting editor's windows:
for example, if glow is enabled and blend mode set to replace, all windows of editor became black

glow is enabled, blend mode set to additive, and strength set to 2

also if you move to other scene all became normal

so the scene with world evn node should be open to see this bug
(back to node_2d from node_2d_2 and there is bug again)

Steps to reproduce

create project (Forward+)
create Node2D
add WorldEnvironment
create New Environment resource
set background mode to canvas

enable glow
for example, change your blend mode to replace
for example, open project settings

Minimal reproduction project
Bug.zip

Duplicate of https://github.com/godotengine/godot/issues/69122
Fixed by https://github.com/godotengine/godot/pull/72343
The fix will be in Beta 17


Clear end of line

Fixes issue #43

hi, @alfredodeza , have you tested this patch?

@Liuchang0812 yep. Did you had any issues with it?

:+1: wonderful.  I'm going to test it.


what is the max-input-length of codet5? 512 or 1024? I found 2048 can also been input.

it is may be that the position max embedding is 512.

Is the max token length equal to 512? Is that only for input? Or the output as well? How can the token length be increased without shifting to codeT5 large model.