ImportError when using CommonMark-py with Google App Engine

When I try to use CommonMark-py -which uses python-future- with App Engine I get this error, if I put import CommonMark at the top:
ERROR    2016-08-26 14:04:59,126 wsgi.py:263]
Traceback (most recent call last):
  File "/home/super/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/home/super/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
    handler, path, err = LoadObject(self._handler)
  File "/home/super/google_appengine/google/appengine/runtime/wsgi.py", line 85, in LoadObject
    obj = __import__(path[0])
  File "/home/super/WebProjects/myapp/blog.py", line 3, in <module>
    from models import Post
  File "/home/super/WebProjects/myapp/models.py", line 1, in <module>
    import CommonMark
  File "/home/super/WebProjects/myapp/libs/CommonMark/__init__.py", line 4, in <module>
    from CommonMark.main import commonmark
  File "/home/super/WebProjects/myapp/libs/CommonMark/main.py", line 15, in <module>
    from CommonMark.dump import dumpAST, dumpJSON
  File "/home/super/WebProjects/myapp/libs/CommonMark/dump.py", line 3, in <module>
    from builtins import str
  File "/home/super/WebProjects/myapp/libs/builtins/__init__.py", line 8, in <module>
    from future.builtins import *
  File "/home/super/WebProjects/myapp/libs/future/builtins/__init__.py", line 10, in <module>
    from future.builtins.iterators import (filter, map, zip)
  File "/home/super/WebProjects/myapp/libs/future/builtins/iterators.py", line 43, in <module>
    from future.types import newrange as range
  File "/home/super/WebProjects/myapp/libs/future/types/__init__.py", line 243, in <module>
    from .newrange import newrange
  File "/home/super/WebProjects/myapp/libs/future/types/newrange.py", line 25, in <module>
    from future.backports.misc import count   # with step parameter on Py2.6
  File "/home/super/WebProjects/myapp/libs/future/backports/__init__.py", line 17, in <module>
    from .misc import (ceil,
  File "/home/super/WebProjects/myapp/libs/future/backports/misc.py", line 900, in <module>
    from subprocess import check_output
ImportError: cannot import name check_output
INFO     2016-08-26 14:04:59,140 module.py:788] default: "GET /blog/5822463824887808 HTTP/1.1" 500 -
INFO     2016-08-26 14:04:59,196 module.py:788] default: "GET /favicon.ico HTTP/1.1" 304 -

But, when I import CommonMark inside a method I get:
ERROR    2016-08-26 13:38:08,116 webapp2.py:1552] cannot import name check_output
Traceback (most recent call last):
  File "/home/super/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1535, in __call__
    rv = self.handle_exception(request, response, e)
  File "/home/super/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1529, in __call__
    rv = self.router.dispatch(request, response)
  File "/home/super/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/home/super/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1102, in __call__
    return handler.dispatch()
  File "/home/super/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 572, in dispatch
    return self.handle_exception(e, self.app.debug)
  File "/home/super/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 570, in dispatch
    return method(*args, **kwargs)
  File "/home/super/WebProjects/myapp/blog.py", line 29, in get
    self.render('permalink.html', post=post, url=self.url)
  File "/home/super/WebProjects/myapp/handlers.py", line 33, in render
    self.write(self.render_str(template, **kw))
  File "/home/super/WebProjects/myapp/handlers.py", line 30, in render_str
    return render_str(template, **kw)
  File "/home/super/WebProjects/myapp/handlers.py", line 20, in render_str
    return t.render(kw)
  File "/home/super/google_appengine/lib/jinja2-2.6/jinja2/environment.py", line 894, in render
    return self.environment.handle_exception(exc_info, True)
  File "/home/super/WebProjects/myapp/templates/permalink.html", line 1, in top-level template code
    {% extends "base.html" %}
ImportError: cannot import name check_output
INFO     2016-08-26 13:38:08,186 module.py:788] default: "GET /blog/5822463824887808 HTTP/1.1" 500 228
INFO     2016-08-26 13:38:08,281 module.py:788] default: "GET /favicon.ico HTTP/1.1" 304 -

I am used to add other libraries to my App Engine projects as described in App Engine's installing a library document but this library causes these errors even though it is working with webapp2 outside App Engine!
I reported this issue to CommonMark but Mr. @nikolas suggested reporting it here.
Note: I am using App Engine Standard Environment and Python 2.7.12. Also, It seems like App Engine is using version 2.7.2.

GAE has a patched subprocess that disables running anything; I guess future will have to ignore the import error unless really needed.

Thank you @QuLogic for your explanation. It seems I have to look for other Markdown library until this issue is fixed!

@badersur CommonMark only started requiring the future module in version 0.6.2 - see the changelog: https://github.com/rtfd/CommonMark-py/blob/master/CHANGELOG.md#062-2016-01-08
So, a temporary solution could be to use CommonMark version 0.6.1, until the issue is resolved in future.

Thank you @nikolas for telling me about this release.
I tested it and it worked nicely except for producing non HTML5 tags (hr, br and img)! On the other hand, it didn't wrap iframe with p tag like an other library!

I'm not sure what you mean - these tags are in the HTML5 spec:

https://www.w3.org/TR/html-markup/hr.html
https://www.w3.org/TR/html-markup/br.html
https://www.w3.org/TR/html-markup/img.html

But in any case, if you have an issue with this html representation, that's not CommonMark-py's decision, but the CommonMark spec itself: http://spec.commonmark.org/ So any suggestions there you can bring up in the commonmark forums.

Sorry. I meant the library produces
,
 and  instead of
... without the trailing /.
BTW, I included your comments on an answer to my question on Quora: How can I use CommonMark-py on Google App Engine.

Ah okay, well this is as defined in the commonmark spec: http://spec.commonmark.org/0.26/#example-599
If you think this should be changed you can open an issue here: https://github.com/jgm/CommonMark/issues or open a discussion here: https://talk.commonmark.org/c/spec

My bad! I thought because it is a specification then it will be generating proper HTML5 tags! Maybe I should read the lengthy document :sweat:

This should be fixed now with future v0.16.0. Please try this again on GAE and re-open this issue if you encounter any more problems.

Thank you @edschofield for fixing this issue. It's working now :+1: :)


Update conda channel order priorities

Update conda channel order priorities (close #641 )
PR checklist

[X] PR is made against dev branch
[X] This comment contains a description of changes (with reason)
[X] Ensure the test suite passes (./scripts/test.sh -p docker -t ALL).
[X] CHANGELOG.md is updated

No idea how or why, but this is fixing my issues when building a new container from scratch


add tool to extract equilibrium pose from simulations

It would be super useful for debugging and explaining perses  ddG predictions if we would be able to write ligand poses that represent equilibrium during the simulations. This could e.g. be in the form of a PDB snapshot every n steps, where n is a user-set flag?

@JenkeScheen Thanks for opening the issue!
Just trying to clearly understand this. We are currently working on a streamlined way of extracting the trajectories from the simulation results (.nc files). I guess we could make it in a way that we extract every n steps and also initializing it after we have detected equilibration or something like that. Would this satisfy your requirements?

Thanks @ijpulidos, it would. The use-case here is that when I present FE predictions to chemists that are trying to explore a sub-pocket with a ligand series they want to be able to do some sort of qualitative assessment that the R groups they came up with are hanging out in the intended areas (and that the ligand isn't just jumping out of the pocket, of course). A collection of snapshots (preferably at ~ equilibrium) would allow that assessment.
Ultimately we would want to be able to automate generation of a GIF showing a simulation of the ligand in the pocket but snapshots are a great starting point!

I'll plan to implement both a CLI and Python API to extract equilibrium snapshots of both initial and final molecules.
For a CLI, how does this look as a first attempt?
perses extract-snapshots --nsnapshots 10 --initial initial.pdb --final final.pdb

Ultimately we would want to be able to automate generation of a GIF showing a simulation of the ligand in the pocket but snapshots are a great starting point!

For the record, this is a nice thing to have, but it will never be as useful as being able to interactively inspect the structure (e.g. in fragalysis). But I can see utility if we can generate both the GIF of 3D interactions and a 2D interaction diagram and push this to the real-time dashboard.

For a CLI, how does this look as a first attempt?

Input:

Are .pdb files enough as input or do we need to inject e.g. .nc as well?

Output:

Perhaps an --out flag to allow custom out names?
Would the output be a multi-pdb of nshapshots snapshots of the PL complex?
It could be valuable to generate a large number (10-100) of ligand poses, perhaps as a multi-sdf?

For the record, this is a nice thing to have, but it will never be as useful as being able to interactively inspect [..]

Agreed, interactive is king. Although, beside being a decent placeholder until we can upload this type of visualisation to e.g. fragalysis, GIFs are nice because they can be easily embedded in slidedecks without weighing them down too much.


Add functionality and tests for weighting samples in metrics

As described in #4313, it would be very useful in models where sample_weight is utilized to monitor accuracy and other metrics in a way that respects the weighing of the samples, either removing 0-weighted samples, removing some dimensions in temporal samples, or just scaling the vectors themselves.
This pull request adds this functionality by allowing metrics to be passed in as dicts with a"metric" key that holds the string (or function) value that would usually be the passed in metric, and a "weighted" key that should only hold the "True" value. This metric is wrapped in weighted_metric, allowing it to now take sample_weight values as well, similar to how weighted_objective works.
This pull request also includes tests for this new functionality, inspired by the weighted loss tests. Tests include verifying:

new functionality does not disturb traditional metric calculation
new functionality does not change metric values on trivial weights
new functionality does change metric values when considering removing samples through 0 weighting
new functionality works in traditional weighting_mode, as well as temporal, in both regular and "twist" format

I realized that nonzero is not doing what I expected to. Will require more work.


KP-6690 snyk

There was more going on with this PR than I thought.  I think we need to scrap the PR and create a new branch that doesn't try to convert the project to gradle.


pub should install from local cache, when network fails

I've previously installed packages and they are in my local cache. I do not have network connectivity now (I am in a place with a firewall and I can't use git over SSH port 22).
Running pub install fails. I would expect a warning, but for it to continue because I know I have these packages in my local cache.

This issue has been moved to dart-lang/pub#184.


Clear wasm module global

Glb-parser currently fails to work with draco because it looks at window global for wasm module.
This PR changes WasmModule to clear the global module object on window.

Could this cause issues on multi engine instances apps if it clears a global?

Could this cause issues on multi engine instances apps if it clears a global?

Nope. I'd like to add support for multiple applications or wasm instances, but that won't be affected by this change.


Mean absolute error regression output layer

Hi,
I am interested in looking at the detail of how this MAERegressionOutput layer is implemented and specifically how the sub gradient is computed. However, I am not able to find the source code snippet for this part, would you please point me to that? (I am using MXNet python package) Thanks!

It's in the backend at src/operators/regression_output-inl.h.
@tqchen How about we put some comment in symbol.py to explain where/how things are registered.

This issue is closed due to lack of activity in the last 90 days. Feel free to reopen if this is still an active issue. Thanks!


Changes to filter_molecules and smirnoff99Frosst

I am addressing a few issues in the PR:

Issue #37 - We've created a new repo to store versions of MiniDrugBank so I've moved the jupyter notebook over there. I also removed intermediate molecule sets that we don't use and were poorly documented.
I didn't write an issue for this yet, but the fact that DrugBank_atyped.oeb didn't have 3D coordinates has made generating MiniDrugBank or just using that file a pain. I generated a parm@frosst and tripos typed file for DrugBank with 3D coordinates using oeomega. I'm going to add the script to the filter_molecules directory and the two new molecule files to molecules directory.
I'm working on replacing AlkEtOH with AlkEthOH per issue #36
updated SMIRKS pattern discussed in issue #17
Replaced 'R' with 'x' in smirnoff99Frosst
Added read_typelist as discussed in issue #20

@davidlmobley I believe this is ready for review when you have time

Noted.


Border Patrol service matcher should look at both subdomain and path

Currently, BP filters by subdomain and then by path. Thus, may choose an incorrect ServiceIdentifier.
Instead, it should look at combination of subdomain & path to determine the serviceIdentifier, so that following config can work:
subdomain: foo, path: /fun
subdomain: foo, path: /sun

Introducing CustomerIdentifier (that identifies the subdomain & login manager (i.e. auth method). ServiceIdentifier (only identifies the service)


something's off with demutation

if in the flux_dp example we permute the last three statements in the inner loop to this:
         G = sum_gradients(gs,G)

         # clip the gradient
         clip!(L2,gs)
         norm_convert!(gs)

we suddenly infer a privacy of (0, s_36) for the input data where s_36 is a variable in (0,1] that does not appear anywhere else. that is very wrong, typechecking should fail!

actually, mutation elaboration just deletes the inner loop altogether:
                          Mutation elaborated term is:
                          BBLet unbounded_gradient = ([DMModel,Vector{Any},Any] -> ?
                          BBLet init_model = ([] -> ?
                          BBLet loss = ([Any,Any,Any] -> ?
                          FLet train_dp = LamStar ([data :- (Any, interesting), labels :- (Any, interesting), eps :- (Any, uninteresting), del :- (Any, uninteresting), eta :- (Any, uninteresting), k :- (Integer, uninteresting), b :- (Integer, uninteresting)])
                          (
                            SLet model :- Any = BBApply (Just init_model)[[unbounded_gradient, init_model, loss, train_dp]]([])
                            TLet [dim :- Any, _ :- Any] = Size (Just data)
                            TLet [model :- Any] = Loop (ceil [/ [- [Just k, - [1.0, 1.0]], 1.0]], [model], (Just i,gen_loop_capture_1))
                            (
                              TLet [model :- Any] = Just gen_loop_capture_1
                              SmpLet [D :- Any, L :- Any] <- Sample (Just b, Just data, Just labels)
                              SLet G :- Any = ZeroGrad Just model
                              TLet [model :- Any] = TLet [G :- Any] = ScaleGrad (/ [1.0, Just b], Just G)
                              TLet [G :- Any] = Gauss (/ [2.0, Just b], Just eps, Just del, Just G)
                              TLet [G :- Any] = ScaleGrad (Just eta, Just G)
                              SubGrad (Just model, Just G)
                              Tup [Just model]
                            )
                            Just model
                          )
                          LastTerm Just train_dp

This is because we one part of a mutlet was recognized to have no effect (e.g. being pure while the other isn't). Though it might be of course in error about that.
Fixing this after #158.

After the rewrite of #189 the flux_dp code (with switched order of statements as described in op) now gives the following deproceduralized term:
                  BBLet unbounded_gradient = ([DMModel,Any,Any] -> ?)
                  BBLet init_model = ([] -> ?)
                  BBLet loss = ([Any,Any,Any] -> ?)
                  FLet train_dp =
                    LamStar ([data :- (Any, interesting), labels :- (Any, interesting), eps :- (Any, uninteresting), del :- (Any, uninteresting), eta :- (Any, uninteresting), k :- (Integer, uninteresting), b :- (Integer, uninteresting)])
                    (
                      SLet n_params :- Any = 31810.0
                      SLet model :- Any = BBApply (init_model)[[train_dp, loss, init_model, unbounded_gradient]]([]) -> BBVecLike DMModel (n_params)
                      TLet [dim :- Any, hole_0 :- Any] = Size (data)
                      TLet [gen_model#_13_6 :- Any] =
                        Loop (ceil [/ [- [* [k, / [dim, b]], - [1.0, 1.0]], 1.0]], [model], (i,gen_loop_capture_7))
                        (
                          TLet [model :- Any] = gen_loop_capture_7
                          SmpLet [D :- Any, L :- Any] <- Sample (b, data, labels)
                          SLet G :- Any = ZeroGrad model
                          TLet [G :- Any] =
                            Loop (ceil [/ [- [b, - [1.0, 1.0]], 1.0]], [G], (j,gen_loop_capture_2))
                            (
                              TLet [G :- Any] = gen_loop_capture_2
                              SLet d :- Any = Row (D, j)
                              SLet l :- Any = Row (L, j)
                              SLet gs :- Any = BBApply (unbounded_gradient)[[train_dp, loss, init_model, unbounded_gradient]]([model, d, l]) -> BBVecLike DMGrads (n_params)
                              ### NOTE these lines ########################
                              # Even though the next two statements after this one
                              # are not relevant in the result we still include them
                              SLet G :- Any = SumGrads (gs, G)
                              SLet gen_gs#_25_0 :- Any = MutClipM (Clip(L2), gs)
                              SLet gen_gs#_25_1 :- Any = ConvertM (gen_gs#_25_0)
                              ######################################
                              Tup [G]
                            )

                          SLet gen_G#_26_3 :- Any = ScaleGrad (/ [1.0, b], G)
                          SLet gen_G#_26_4 :- Any = Gauss (/ [2.0, b], eps, del, gen_G#_26_3)
                          SLet gen_G#_26_5 :- Any = ScaleGrad (eta, gen_G#_26_4)
                          SLet gen_model#_13_6 :- Any = SubGrad (model, gen_G#_26_5)
                          Tup [gen_model#_13_6]
                        )

                      gen_model#_13_6
                    )

                  train_dp

This is correct (and the code which is the culprit in creating the


Fix back navigation for created threads.

Clicking back from a locally created thread navigates the user back to the new thread activity used to create it. This should just lead back to the main activity instead.

Fixed as of db910c2


Info not displayed on index view

Hello,
Info not displayed on index view.
I understand that this feature is to use on detail view, but can you add a feature to force display on index view?
Tooltip  doesn't looks well in my case.  I use your component to show subscription status with a short label

Regards

It was trivial to implement, I've just sent a PR for this. Hope that helps


Restore support for Ruby 2.3 and 2.4.

We recently released 5.0.0 and 5.1.0. These would have been 4.2.0 and 4.3.0 had it not been for the fact that we dropped support for Ruby 2.3 and 2.4.
Today I discovered https://stats.rubygems.org/ which suggests 55% of Ruby users are on one of the versions we dropped support for. I hadn't expected that many people to be using unsupported rubies.
I think we should release 5.2.0 where we restore Ruby 2.3 support.

I also think we should make 4.2.0 and 4.3.0 releases, back-porting what's in 5.0.0 and 5.1.0.
But for how long do we keep doing these parallel releases?

Good question, you've opened a bit of a can of worms.
I knew 2.3 / 2.4 were still in use, but I never would have guessed 50%. I think it's a dilemna that perhaps is best served by the rails analogy. So you serve a couple of majors back, but only with critical things.
So we could make 4.2 / 4.3 releases that re-enable lower rubies, but then the new stuff only goes into cucumber5, and any bug fixes can go into 4/5

Any further thoughts on this. Currently cucumber rails uses 2.4+ and the metrics on that graph over the last month don't really show much change.
The only thing I "would" say is (And I don't know the answer), are those numbers where they are because of people wanting to use those versions, or because CI systems are using those systems and so the numbers are high (Chicken / Egg e.t.c.)

I hadn't thought of that - high 2.3 and 2.4 numbers possibly due to CI servers using old versions.

Even if CI servers were using old versions, I doubt it would have much of an impact on the stats. According to the stats the last chart shows that close to 100% of builds are not CI, so high numbers of old rubies are almost certainly not due to CIs using them.

High 2.3 and 2.4 numbers are probably from people still using  2.3 and 2.4 in their apps. 50% sounds about right for that, especially when you count JRuby 9.1 or earlier users who are stuck on older Ruby version compatibility. Since people could be depending on their cucumber tests to help them verify Ruby updates, not supporting the versions that they're updating from seems a bit counterproductive.

bump

Do we have any thoughts on how to tackle this @aslakhellesoy ? Did my rails analogy make sense.
So the 4.x revisions only contain bug fixes, all new functionality goes into cucumber 5.x?

Restoration of support of ruby 2.3 and 2.4 has not been done after a year now
We did not had any complain
I guess people still relying on those unsupported ruby version are not concerned with upgrading their cucumber either
Do we still want to discuss and/or try re-enabling support for those old rubies?

Given that we haven't had any complaints, I don't see this as high priority. I'm closing this. We can reopen if someone complains.


containerd 1.4.0 opened thousands of pipes

Description
Hello,
thanks for your great software.
We faced to a very annoying issue today on one of our kubernetes production node.
Linux system has reached max open files (65535). We couldn't connect to node due to that issue.
After evicting pods with the kube api we saw that containerd is responsible of this issue.
Steps to reproduce the issue:
We didn't succeed to reproduce this
Describe the results you received:
If we look at file-nr after evicting the pods we have this result:
root@vp-3356:~# cat /proc/sys/fs/file-nr
64320	0	65536

Here is a fragment of the crazy descriptor open file list
container 11141                        root 3176r     FIFO               0,13       0t0   15870263 pipe
container 11141                        root 3177r     FIFO               0,13       0t0   15866211 pipe
container 11141                        root 3178r     FIFO               0,13       0t0   15868912 pipe
container 11141                        root 3179r     FIFO               0,13       0t0   15870264 pipe
container 11141                        root 3180r     FIFO               0,13       0t0   15891524 pipe
container 11141                        root 3181r     FIFO               0,13       0t0   15868913 pipe
container 11141                        root 3182r     FIFO               0,13       0t0   15910083 pipe
container 11141                        root 3183r     FIFO               0,13       0t0   15891525 pipe
container 11141                        root 3184r     FIFO               0,13       0t0   15911032 pipe
container 11141                        root 3185r     FIFO               0,13       0t0   15910084 pipe
container 11141                        root 3186r     FIFO               0,13       0t0   15918701 pipe
container 11141                        root 3187r     FIFO               0,13       0t0   15911033 pipe
container 11141                        root 3188r     FIFO               0,13       0t0   15924488 pipe
container 11141                        root 3189r     FIFO               0,13       0t0   15918702 pipe
container 11141                        root 3190r     FIFO               0,13       0t0   15944117 pipe
container 11141                        root 3191r     FIFO               0,13       0t0   15924489 pipe
container 11141                        root 3192r     FIFO               0,13       0t0   15938473 pipe
container 11141                        root 3193r     FIFO               0,13       0t0   15944118 pipe
container 11141                        root 3194r     FIFO               0,13       0t0   15971630 pipe
container 11141                        root 3195r     FIFO               0,13       0t0   15938474 pipe
container 11141                        root 3196r     FIFO               0,13       0t0   15964615 pipe
container 11141                        root 3197r     FIFO               0,13       0t0   15971631 pipe
container 11141                        root 3198r     FIFO               0,13       0t0   15980112 pipe
container 11141                        root 3199r     FIFO               0,13       0t0   15964616 pipe
container 11141                        root 3200r     FIFO               0,13       0t0   15985896 pipe
container 11141                        root 3201r     FIFO               0,13       0t0   15980113 pipe
container 11141                        root 3202r     FIFO               0,13       0t0   15993728 pipe
container 11141                        root 3203r     FIFO               0,13       0t0   15985897 pipe
container 11141                        root 3204r     FIFO               0,13       0t0   15993740 pipe
container 11141                        root 3205r     FIFO               0,13       0t0   15993729 pipe
container 11141                        root 3206r     FIFO               0,13       0t0   16028767 pipe
container 11141                        root 3207r     FIFO               0,13       0t0   15993741 pipe
container 11141                        root 3208r     FIFO               0,13       0t0   16014045 pipe
container 11141                        root 3209r     FIFO               0,13       0t0   16028768 pipe
container 11141                        root 3210r     FIFO               0,13       0t0   16039682 pipe
container 11141                        root 3211r     FIFO               0,13       0t0   16014046 pipe
container 11141                        root 3212r     FIFO               0,13       0t0   16042428 pipe
container 11141                        root 3213r     FIFO               0,13       0t0   16039683 pipe

If we count them it seems crazy (it may not be so accurate)
root@vp-3356:~# lsof  -n |grep -c "container.+pipe"
691254

Describe the results you expected:
Output of containerd --version:
containerd github.com/containerd/containerd v1.4.0 09814d48d50816305a8e6c1a4ae3e2bcc4ba725a

Any other relevant information:
root@vp-3356:~# uname -a
Linux vp-3356.vm.vptech.eu 5.4.0-0.bpo.4-amd64 #1 SMP Debian 5.4.19-1~bpo10+1 (2020-03-09) x86_64 GNU/Linux

We won't reboot the node for node, we will add a new one, if you need to have more runtime information

Is there a way for you to get the paths of these fifos so we can see that the source is?

@nerzhul also.  is your FD list at runtime or after these pods/containers have been drained?  Is this an issue of a leak after the container has exited or a runtime issue with too many open fds?

they are here after all pod except DS are drained
as we see here it's the runtime not the containers whcih seems to open it. I will try tomorrow to find if i can see what is that pipe, but it sounds quite hard to find :)

Also one point, why does your system have such a low limit?
my fedora box has:
cat /proc/sys/fs/file-nr
1472    0<PHONE_NUMBER>854775807

65536 does seem low.. @nerzhul could you also share some information about your node setup, version of k8s,..  I glanced over and saw you have another open issue regarding Prometheus monitoring.
20lts
michaelbrown@homeserver:~$ cat /proc/sys/fs/file-nr
2352	0<PHONE_NUMBER>854775807
michaelbrown@homeserver:~$ uname -a
Linux homeserver 5.4.0-48-generic #52-Ubuntu SMP Thu Sep 10 10:58:49 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

16lts
root@mike-VirtualBox:/run/containerd# cat /proc/sys/fs/file-nr
7872	0	810953
root@mike-VirtualBox:/run/containerd# uname -a
Linux mike-VirtualBox 4.4.0-190-generic #220-Ubuntu SMP Fri Aug 28 23:02:15 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

@mikebrow thanks for your feedback
I can agree that our setuped max-file is a little bit low, we can increase it, but there was only 10 pods on this node on the failure.
I will add a reasonable factor based on minimum node size we can have in our setup (i think we can multiply by 5 or 10 safely)
But this doesn't explain the thousands of pipes opened by containerd, it's not files opened by containers here.
For the node setup here it is for this one:

Debian 10.5
Kernel 5.4.0-0.bpo.4-amd64
Kubelet 1.18.8
Containerd 1.4.0

Containers running on this node at failure time

CoreDNS 1.6.9
Kube-Proxy 1.18.8
OpenNTPD (alpine 3.12 version)
node-exporter 1.0.1
kube-router 1.0.1
3 company owned apps with no specific load

I'm trying to find information about all those pipes, if possible

@crosbymichael fedora, and my arch as desktop has unlimited file descriptors, but it's not a good solution, too many filedescriptors can mean kernel resource exhaustion at a point :)

Thanks, yes I agree.  We need to see if we are leaking somewhere or what is going on, even if you have a really low max-file :)
Do you have health-checks enabled for these containers?  Just trying to drill down and see what area to look at first.  Most of the pipes should show up like:
container 900 924 container    root    8u     FIFO               0,25       0t0      33801 /run/containerd/io.containerd.runtime.v2.task/default/test/log
container 900 924 container    root    9w     FIFO               0,25       0t0      33801 /run/containerd/io.containerd.runtime.v2.task/default/test/log
container 900 924 container    root   13u     FIFO               0,25       0t0      28963 /run/containerd/fifo/857536107/test-stdin
container 900 924 container    root   14u     FIFO               0,25       0t0      28963 /run/containerd/fifo/857536107/test-stdin
container 900 924 container    root   15w     FIFO               0,25       0t0      28963 /run/containerd/fifo/857536107/test-stdin
container 900 924 container    root   17u     FIFO               0,25       0t0      28965 /run/containerd/fifo/857536107/test-stdout
container 900 924 container    root   18w     FIFO               0,25       0t0      28965 /run/containerd/fifo/857536107/test-stdout
container 900 924 container    root   19u     FIFO               0,25       0t0      28965 /run/containerd/fifo/857536107/test-stdout
container 900 924 container    root   20r     FIFO               0,25       0t0      28965 /run/containerd/fifo/857536107/test-stdout
container 900 924 container    root   21r     FIFO               0,25       0t0      28963 /run/containerd/fifo/857536107/test-stdin

I just tested 1.3.7 vs 1.4.0 and I don't see any different in the number of pipes that are open, it's actually less in 1.4.0.  I still want to look into this more to see if there are some duplicates that are not handled correctly, but I'm not sure this is a functionality change in 1.4.0.  Maybe your machines had a bad max-file configuration and that is why you see it on this server.

okay we had the same issue on another node today with dozen of thousands pipes, i miss some time to investigate more, will try to find what is that

i found some intesting things about the pipes, it seems pipes are coming from a running pod which cannot be killed due to open file limit exhaustion, containerd is unable to kill it in time whereas other pods are drained.
I run a counter on the process uid running under containerd process and i found this:
root@vp-node:~# for FD in $(ls /proc/31867/fd/); do stat -L /proc/31867/fd/$FD; done|awk '/Uid/{print $0}'|sort | uniq -c
      6 Access: (0600/prw-------)  Uid: (    0/    root)   Gid: (    0/    root)
  22050 Access: (0600/prw-------)  Uid: (  999/systemd-coredump)   Gid: (    0/    root)
      2 Access: (0600/?rw-------)  Uid: (    0/    root)   Gid: (    0/    root)
      3 Access: (0666/crw-rw-rw-)  Uid: (    0/    root)   Gid: (    0/    root)
      4 Access: (0700/prwx------)  Uid: (    0/    root)   Gid: (    0/    root)
      2 Access: (0777/srwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)

another interesting thing i note after this, is that containerd is unable to kill the containerd whereas it was terminated correctly.
See the following output:
root@vp-node:~# crictl stop 400a7e7d0ab30
^C^C^C^C^C^C^C
^Z
[1]+  Stopped                 crictl stop 400a7e7d0ab30
root@vp-335e:~# ps ax|grep redis
27594 pts/1    S+     0:00 grep redis
root@vp-node:~# crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID
400a7e7d0ab30       893bb508e38fe       8 days ago          Running             redis               0                   eb8c78dab84fa
e2b914a04adda       200a90f4ed375       2 weeks ago         Running             openntpd            0                   e9911b317fa53
8d7ca9ec1b8a3       1588e7f4a520c       4 weeks ago         Running             redash-server       0                   6de5139c7161d
bd1a1a8fdf327       59db05dddfdd5       4 weeks ago         Running             fluentbit           0                   0051cf0235b0a
46336700007e7       685a73a3c754b       4 weeks ago         Running             kube-rbac-proxy     0                   cca206ae69d73
50b53641ecf85       0e0218889c33b       4 weeks ago         Running             node-exporter       0                   cca206ae69d73
4c130081189a1       16acc1ede9f93       4 weeks ago         Running             kube-proxy          1                   2294ff106d7d2
4fde426f70c53       ab0f9cd91cf93       4 weeks ago         Running             ipmasq-agent        0                   ba1dfc1d5fd7e
root@vp-node:~# ps ax|grep redis
27705 pts/1    S+     0:00 grep redis
root@vp-node:~#

As you can see the pod is marked as running, but no process is present on the machine
If we look at containerd logs we see that:
Oct 28 08:51:41 vp-335e.vm.vptech.eu containerd[1187]: time="2020-10-28T08:51:41.409385776Z" level=error msg="ExecSync for \"400a7e7d0ab30a4b2439cab692dad1a4841c3e57df81580b7de59d5af0b43913\" failed" error="failed to exec in container: failed to start exec \"dd60a3fd9af9d333f463069c67446e8cc754f05d2f0bfc0335e452e179da0bf8\": OCI runtime exec failed: container not running: unknown"
Oct 28 08:51:41 vp-335e.vm.vptech.eu containerd[1187]: time="2020-10-28T08:51:41.409834244Z" level=info msg="ExecSync for \"400a7e7d0ab30a4b2439cab692dad1a4841c3e57df81580b7de59d5af0b43913\" with command [sh -c redis-cli ping] and timeout 1 (s)"
Oct 28 08:51:41 vp-335e.vm.vptech.eu containerd[1187]: time="2020-10-28T08:51:41.436408191Z" level=error msg="ExecSync for \"400a7e7d0ab30a4b2439cab692dad1a4841c3e57df81580b7de59d5af0b43913\" failed" error="failed to exec in container: failed to start exec \"721a182e357364ce58bafcca73315fc0ebe16b8555971fdc1ec2250fe1f2e2ad\": OCI runtime exec failed: exec failed: cannot exec a container that has stopped: unknown"
Oct 28 08:51:41 vp-335e.vm.vptech.eu containerd[1187]: time="2020-10-28T08:51:41.436906323Z" level=info msg="ExecSync for \"400a7e7d0ab30a4b2439cab692dad1a4841c3e57df81580b7de59d5af0b43913\" with command [sh -c redis-cli ping] and timeout 1 (s)"
Oct 28 08:51:43 vp-335e.vm.vptech.eu containerd[1187]: time="2020-10-28T08:51:43.410343937Z" level=error msg="ExecSync for \"400a7e7d0ab30a4b2439cab692dad1a4841c3e57df81580b7de59d5af0b43913\" failed" error="failed to exec in container: failed to start exec \"4ac501d73fa74cdd04315ff7ef628df554f46a76e6617041fabab2d59866e43c\": OCI runtime exec failed: exec failed: cannot exec a container that has stopped: unknown"
Oct 28 08:51:43 vp-335e.vm.vptech.eu containerd[1187]: time="2020-10-28T08:51:43.410969391Z" level=info msg="ExecSync for \"400a7e7d0ab30a4b2439cab692dad1a4841c3e57df81580b7de59d5af0b43913\" with command [sh -c redis-cli ping] and timeout 1 (s)"
Oct 28 08:51:43 vp-335e.vm.vptech.eu containerd[1187]: time="2020-10-28T08:51:43.439926687Z" level=error msg="ExecSync for \"400a7e7d0ab30a4b2439cab692dad1a4841c3e57df81580b7de59d5af0b43913\" failed" error="failed to exec in container: failed to start exec \"3b43963518492cc3adbc9106fe25555a896bb3912471f19c5afbcc35f0da1882\": OCI runtime exec failed: exec failed: cannot exec a container that has stopped: unknown"
Oct 28 08:51:43 vp-335e.vm.vptech.eu containerd[1187]: time="2020-10-28T08:51:43.440378167Z" level=info msg="ExecSync for \"400a7e7d0ab30a4b2439cab692dad1a4841c3e57df81580b7de59d5af0b43913\" with command [sh -c redis-cli ping] and timeout 1 (s)"
Oct 28 08:52:10 vp-335e.vm.vptech.eu containerd[1187]: time="2020-10-28T08:52:10.142224240Z" level=error msg="ExecSync for \"400a7e7d0ab30a4b2439cab692dad1a4841c3e57df81580b7de59d5af0b43913\" failed" error="failed to exec in container: failed to start exec \"590b93b23941611f5df9b48c0a0cd05d7cbe4f127a924e2c43cad2ba116f728b\": OCI runtime exec failed: exec failed: cannot exec a container that has stopped: unknown"
Oct 28 08:52:10 vp-335e.vm.vptech.eu containerd[1187]: time="2020-10-28T08:52:10.142691174Z" level=info msg="ExecSync for \"400a7e7d0ab30a4b2439cab692dad1a4841c3e57df81580b7de59d5af0b43913\" with command [sh -c redis-cli ping] and timeout 1 (s)"
Oct 28 08:52:10 vp-335e.vm.vptech.eu containerd[1187]: time="2020-10-28T08:52:10.168419748Z" level=error msg="ExecSync for \"400a7e7d0ab30a4b2439cab692dad1a4841c3e57df81580b7de59d5af0b43913\" failed" error="failed to exec in container: failed to start exec \"d57a8a1baea7666259f46580fa0ba1a17ca2702872b67f93dc92aac16c19c3ac\": OCI runtime exec failed: exec failed: cannot exec a container that has stopped: unknown"
Oct 28 08:52:10 vp-335e.vm.vptech.eu containerd[1187]: time="2020-10-28T08:52:10.168904733Z" level=info msg="ExecSync for \"400a7e7d0ab30a4b2439cab692dad1a4841c3e57df81580b7de59d5af0b43913\" with command [sh -c redis-cli ping] and timeout 1 (s)"

it seems containerd is lost after this outage and cannot stop the container, but continues to run the ping command

and @crosbymichael yes healthcheck are run on this pod, as you see in previous logs. I suspect it's the leak source because after the pod was not running, the FS continues to increase regarding the healthcheck part, i suspect the leak is something there

I looked more deeply in the containerd issues and it sounds like #4509

Same errors here but with docker.
Upgrading containerd from 1.3.2 to 1.4.0 fills syslog with errors about healthcheck and OCI runtime exec failed.
Errors reports container ids but if I try to inspect or something containers not exists.
Downgrading back to 1.3.2 solve for now but still digging for a future update.
Thanks @nerzhul for the detailed info, learnt a lot ;)

I'm closing this as it's fixed now


TypeError: Cannot read property 'makeMutable' of undefined

Describe the bug
When using any of the transition or duration classes I'm getting the TypeError: Cannot read property 'makeMutable' of undefined.
To Reproduce
Steps to reproduce the behavior:

Create any React Native component with e.g. duration-300 transition-transform classNames.
Build the app for Ios simulator (that's where I'm testing my app)
Run the Expo app
See error

Expected behavior
I would've expected nativewind to be able to resolve the tailwind classes and create styles for it.
Expo Snack
Is this still possible in v4? Can't find anything in the v4 docs.
Screenshots

Additional context
My package.json for versions:
"dependencies": {
    "@expo/metro-runtime": "~3.1.3",
    "classnames": "^2.5.1",
    "expo": "~50.0.6",
    "expo-status-bar": "~1.11.1",
    "expo-updates": "~0.24.10",
    "nativewind": "^4.0.29",
    "react": "18.2.0",
    "react-dom": "18.2.0",
    "react-native": "0.73.4",
    "react-native-reanimated": "^3.7.0",
    "react-native-render-html": "^6.3.4",
    "react-native-web": "~0.19.6"
  },
  "devDependencies": {
    "@babel/core": "^7.20.0",
    "@quickdevelopment/wp-js": "^4.0.2",
    "@types/react": "~18.2.45",
    "tailwindcss": "^3.4.1",
    "typescript": "^5.1.3"
  }

Reproduction
Here's a link to my react-native app demo.
Steps to reproduce:

git clone https://github.com/keygun-development/react-app-demo
npm i
npm run ios

Please include a minimal reproduction repo. makeMutable is an export from react-native-reanimated, so your project is failing to import that dependancy. There are many reasons why module resolution can fail, none of which I can debug from the information provided.

In the meantime I'm using a different approach for this transition.
I'm using the Animated namespace along with the Pressable component exported by react-native.
When pressing the pressable component I call a function that changes the scale using Animated.spring and when letting go I call a function that sets it back to the old value.
I have a preview for anyone wondering how to achieve this:
const ScaleableContent: React.FC<IScaleableContent> = (props) => {
  const [scale] = useState(new Animated.Value(1));

  const handlePressIn = () => {
      Animated.spring(scale, {
          toValue: 1.05, // Change this value for the scale
          useNativeDriver: true,
      }).start();
  };

  const handlePressOut = () => {
      Animated.spring(scale, {
          toValue: 1,
          useNativeDriver: true,
      }).start();
  };
  return (
    <Pressable onPressIn={handlePressIn} onPressOut={handlePressOut} className="aspect-[14/9]">
        <View>
            <Animated.View style={{transform: [{scale: scale}]}}>
                {/*Here goes your content*/}
            </Animated.View>
        </View>
    </Pressable>
  );
}
export default ScaleableContent;

This is a problem in react-native-reanimated: 3.7.0 or newer: https://github.com/software-mansion/react-native-reanimated/issues/1132
Downgrading to 3.6.3 works great.

@luixo Thanks! After downgrading to 3.6.3 it does in fact work.


Feature Request: autodestroy, autoredeploy and autobuild features

Hey guys,
Any thought on getting something similar features to what Tutum is doing?
https://support.tutum.co/support/solutions/articles/5000012175-autodestroy
https://support.tutum.co/support/solutions/articles/5000569896-autoredeploy
https://support.tutum.co/support/solutions/articles/5000638474-automated-builds
This would be awesome to get and makes a lots of sens IMHO :)

This is something very powerful and useful - I'd love to have it in Rancher. The only way to do it at the moment is hacking around with container watching docker events - far from ideal.

Closing due to inactivity.  Those links are also obsolete.


Cyborgs take structural damage

Description
Cyborgs will take structural damge when hit with weapons that can apply that damage type.
Not sure if this is intended, but it feels really bad to take extra damage when cyborgs have no way of getting any type of damage resistance that i am aware of.
Reproduction
Hit a cyborg with a weapon that does structual damage and use a health analyser to check the damage.
Screenshots
being hit with an emergency crowbar:

It kind of makes sense though.
Maybe make borgs take more damage before they break and keep things as is? Or make borgs have damage resistances to other types of damage but not structural?
But I think that if they are immune to poison and maybe radiation damage, being vulnerable to structural damage balances things out.

It kind of makes sense though. Maybe make borgs take more damage before they break and keep things as is? Or make borgs have damage resistances to other types of damage but not structural? But I think that if they are immune to poison and maybe radiation damage, being vulnerable to structural damage balances things out.

It kind of makes sense though. Maybe make borgs take more damage before they break and keep things as is? Or make borgs have damage resistances to other types of damage but not structural? But I think that if they are immune to poison and maybe radiation damage, being vulnerable to structural damage balances things out.

 Maybe make borgs take more damage before they break and keep things as is? Or make borgs have damage resistances to other types of damage but not structural? But I think that if they are immune to poison and maybe radiation damage, being vulnerable to structural damage balances things out.

 From both a balancing perspective and a gameplay perspective, having borgs their current HP is bad.
From a game balance sense due to how absolutely expensive borg parts are, and from a gameplay perspective because of how annoying it can be to play as a borg.

It kind of makes sense though. Maybe make borgs take more damage before they break and keep things as is? Or make borgs have damage resistances to other types of damage but not structural? But I think that if they are immune to poison and maybe radiation damage, being vulnerable to structural damage balances things out.

 From both a balancing perspective and a gameplay perspective, having borgs their current HP is bad. From a game balance sense due to how absolutely expensive borg parts are, and from a gameplay perspective because of how annoying it can be to play as a borg.

I'm hopeful some of this will be partly remedied in [#21307]

This will be good because cyborgs literally die in 2 hits from a wielded fireaxe. Im not a structural engineer or game designer but something that is made out of steel shouldent be more vulnerable to an axe than flesh and bone.


Content of registry key ?

What is saved in the backed up registry key?

The browser settings and extensions for each profile.


use current directory as default workspace-folder

#332

@microsoft-github-policy-service agree

#29

Looks like this merged PR already updated the commands ? --> https://github.com/devcontainers/cli/pull/378/files/5793e9eda8a852a7a94a7c96215bc1ab7c200586#diff-eb1f4feeeea826490afd8c5331c930fa33c019a97ffc6ae6b42b6dd95acf7803L513
Seems like it's released in v0.29.0 ?

I hope they merge this in at some point

This one definitely slipped through the gaps ; apologies for the delay, and thanks for your patience.
Looping in @chrmarti for an extra set of eyes.


[DDMD] [cleanup] Stop using local_assert, and stop defining file in every file

Please stop calling changes [cleanup] as if they were dirty. They're not. In this particular case, there is a reason they exist in the C++ source - it makes for smaller executables. This reason may not exist for D, but it doesn't make the code 'dirty'.
[Refactor] seems more appropriate for this sort of change.

IMO it is dirty, because it leaks the internals of tassert into every file that uses it.  It's not an attack on you, and I do understand the reason this is in the code.

This adds significant bloat to the executable without benefit.

it leaks the internals of tassert into every file that uses it.

It's C++, it's the only way to make what tassert does work, tassert is trivial, and I've been using tassert.h this way for 30 years with no problems whatsoever.

This adds significant bloat to the executable without benefit.

Wrong.  It adds a 0.36% size increase when compiled with dmc.
With gcc it adds 0.057%.  Calling that significant bloat is a lie.
The benefit is that the glue layer files that import tassert can now be converted to D.

Calling that significant bloat is a lie.

Daniel, this is inappropriate and I won't tolerate it. Please keep it professional. Thanks.

Apologies, I'm getting very frustrated.  In my opinion, 0.36% is not significant bloat, and this change is not without benefit.  I would appreciate if you could explain why you view it as significant and why you don't consider allowing automated conversion a benefit.
I can understand not wanting to accept changes that are just churn with no benefit, but if there was no benefit then I wouldn't be doing this.  I'm working on automatic conversion for DDMD, and as I understand it this is something we both want.

Thank you.
0.36% is significant to me. Bloat, weight, bankruptcy, performance problems, all happen by allowing creeping smaller bits through. Keeping things lean and fast is a constant vigilance sort of thing, or you wake up one day and discover you're bankrupt. dmd is meant to be lean and fast, and 0.36% is definitely on the radar.
Anyhow, helping ddmd translation out is a good reason. Redo for the glue layer code only which is dealt with by the ddmd translator, and I'll pull it.
(Note that dmd does this optimization in its D code generation!)

0.36% is significant to me. Bloat, weight, bankruptcy, performance problems, all happen by allowing creeping smaller bits through. Keeping things lean and fast is a constant vigilance sort of thing, or you wake up one day and discover you're bankrupt. dmd is meant to be lean and fast, and 0.36% is definitely on the radar.

IMO this is a sort of slipperly slope argument, and while it is not completely without merit, it is not sufficient justification.  DMD's allocation patterns make this amount negligible, and the strings will only need to be loaded in the event of an assertion failure, which is as far as you can get from the fast path.  I think your stance puts too much importance on executable size at a cost to ease of understanding the code.

Anyhow, helping ddmd translation out is a good reason. Redo for the glue layer code only which is dealt with by the ddmd translator, and I'll pull it.

Ok, but I'm not sure how to do this cleanly without dropping use of tassert from the glue layer completely.  Is that acceptable?

(Note that dmd does this optimization in its D code generation!)

I was a little surprised that dmc doesn't appear to do string pooling.

Ok, but I'm not sure how to do this cleanly without dropping use of  tassert  from the glue layer completely. Is that acceptable?

Yes. Just use <assert.h>.

Updated.

Yes. Just use <assert.h>.

assert.h isn't actually used when building with dmc.  Just removing the tassert include makes it pick up whichever is used in mars.h.

All green.

thanks!

Thanks.


Changing the flight-track in Main window does not change name in open TopView

Currently, loading a new flight track does not update the flight track name within the TopView window (maybe affecting other views as well).
Desired state: Changing the flight track should update the name in the view as well.
Not checked for MSCOlab

Already fixed in current version.

related
https://github.com/Open-MSS/MSS/pull/1210


Ability to add wallet name when connecting to remote node

Currently, when connecting to a remote mode, you cannot specify the wallet name, having ability to do it in onboarding process would be helpful

Motivation and Context:
The feature request was requested in #1262
How Has This Been Tested?

Screenshots (if appropriate):
Types of changes:

Checklist:

[x ] My code follows the code style of this project.
[x] I have reviewed and updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[x] I have added tests to cover my changes where needed.
[x] All new and existing tests passed.
[x ] My commits have been squashed into a concise set of changes.

@mrfelton have adjusted the tests to add wallet name.  (The field is placed as the last element in all forms, so have looked for it accordingly)


Pass a style object to any react-vis component to apply styles directly

I'd like to be able to pass a style property anywhere to style my charts programatically, without having to work around the react-vis stylesheet.

This sounds like a reasonable proposal. There are of course a wealth of different systems for adding styling to react-components via javascript. Does anyone have strong opinions about them?

Now that @jckr's style PR is merged i think this can closed? Marking as resolved for now, feel free to re-open

@mcnuttandrew link to that PR or documentation of the feature? We'll need that for an upcoming feature...thanks!

@ericsoco I'm not sure where the PR has gotten off to, but here's the commit https://github.com/uber/react-vis/commit/6752d8ad8e3267564e4056dc0ab9f93bef4530a7


USB support on System76 Lemur Pro

Hey :) I've been really loving manjaro-sway on the frame.work and raspberry pi 400 (different project?)
I ended up popping it onto my Lemur Pro a few days ago and just tried my USB ports for the first time. The flash drive I usually recover my data from wasn't recognized at all and my PS4 controller glows on and glows off, so I know something happens when a device is connected.
Are there any commands I can run to try and troubleshoot this issue?
(thanks again for the support a few months back getting the battery data wired up :)

frame.work

that is an x86 device, so you'll probably have used this edition too.

flash drive I usually recover my data from wasn't recognized at all

so.. it didn't show up in pcmanfm? or what did you expect to happen?

PS4 controller glows on and glows off, so I know something happens

I have no idea what would be supposed to happen when you connect it.

it didn't show up in pcmanfm?
Correct, it doesn't show up in pcmanfm.

My PS4 controller does connect now and show up in lsusb though. I'm thinking maybe just my flash drive is incompatible?

So your usb drive does not show up in lsusb? Could you check dmesg? Either way: since we use the same kernel as regular manjaro, this should imo not be a sway-specific issue.

It looks like my flash drive has stopped working on all machines. It was just unfortunate timing that the first failure happened to be when I went to restore data on a fresh install. Sorry for the churn.


Proposal Discussion

Hi there, your work is quite impressive. Let's connect to discuss a potential opportunity.
Telegram: @verecke1
Discord: vereckel
WhatsApp: +44 7341 822373

Hi there.
Thanks.
Let me know for what I'm good at.


case insensitivity for value::String

It would be great to provide the ability to mark a field as case insensitive when used in conjunction with the derive macro. The serde-aux crate only seems to work for nested structs.

Thanks! I would prefer for this to be a feature request against the serde-aux crate.


Building and dev server results in TypeError: Cannot read property 'compilation' of undefined

Do you want to request a feature or report a bug?
I'm reporting a bug
What is the current behaviour?
When running npm start I get a TypeError: Cannot read property 'compilation' of undefined. The same error occurs when trying to run npm build.
If the current behaviour is a bug, please provide the steps to reproduce.
I'm using the preact default template with version 2.1.0 and this custom preact.config.js
What is the expected behaviour?
It should just start the dev server or build the production version.
Please mention other relevant information.

Node version: 8.11.1
npm version: 5.6.0
Operating system: Windows 10
CLI version: 2.1.0

Have you tried running the build without the custom config file?

Can you provide a link to the repo? So I can go through that and see where the issue is?

@hassanbazzi Good idea. But it still won't work.
@reznord I'm afraid that's not possible since it's a private repository for my company.

@NicolaiSchmid It may also be an issue with your code. Can you search for compilation in your codebase and see what comes up?

I did that already, and there's no such string in my codebase..

This looks like an error caused by webpack. I stumbled upon a similar message last week which was caused by mixing webpack 3 and webpack 4 loaders and plugins

Has the plugin and/or loader api changed in webpack 4?
So, do I need to use specific versions for plugins and loaders, which are taylored specificly for wb 3?

Yes, they changed the api quite a bit. Three current release of preact-cli is based on webpack 3 and the next version will include webpack 4.

But shouldn't the cli work, after I removed my custom preact.config.js? Because then there wouldn't be any misversioned loaders and plugins.

yes, it should. Seems like the cause is something else. Do you have a small repo where the issue can be reproduced?

I'm gonna whip something up tomorrow.
BTW: The issue is also reproducible on Ubuntu 18.4

I'm sorry it took so long, but I just tried reproducing the bug with a fresh project. And tried step by step, adding configuration from my project and I can't seem to reproduce it.
If I stumble upon this error again, I will reopen the issue, but so far the v3 pre-release has served me well.


Magnet link body needed on API

Currently the message api route does not include a magnet link body as part of the post. It should.

since we are not using bittorrent going forward this will not be required.


Not recognizing Citizens

server paper 1.14.4
Citizens-2.0.25-b1746
Quests-3.8.0

Thanks for the report. Which build of Quests are you using? There should be a build number after "3.8.0"

#b27

Thank you. As far as I can tell, it should work properly with that build. Would you mind providing your startup log?
Sent with GitHawk

https://pastebin.com/8mEueRZ6

Another plugin is forcing Citizens to load before Quests. I’ll try to find a solution for this.


Buttons No Longer Work Atfer Importing An Image In

Describe your bug here.
https://github.com/user-attachments/assets/ea5196a2-60b6-4991-ae12-d132a04252a3
ITS NOT WORKING 😭😭😭
so uhh its in the title and the video so uhh
Command Prompt/Terminal logs (if existing)
No response
Are you modding a build from source or with Lua?
Lua
What is your build target?
Windows
Did you edit anything in this build? If so, mention or summarize your changes.
no

Fixed at https://github.com/ShadowMario/FNF-PsychEngine/commit/f100b63d95508295d36e06baf39fefe69f1e3fd5


Fixed number of active cases

Description of PR
There was problem with the number of active cases. The formula to calculate the number of active cases is as follows:

Active = Confimed - Recovered - Deceased

The problem with this is that if for any day the number of recovered + number of deceased is more than number of confimed then the number of active cases will be negative for any day, but this is logically wrong (how can a number of active cases be negative for a day?). The formula is updated to:

Active = Max(0, Confimed - Recovered - Deceased)

This will also fix the problem with the time series 'Daily' graphs. This formula is also added for the total active cases. There might be a situation when this can also be negative.
Type of PR

[x] Bugfix

Relevant Issues
Fixes Number of daily active cases.
Checklist

[x] Compiles and passes lint tests
[x] Properly formatted
[x] Tested on desktop
[x] Tested on phone

Screenshots
Before:

After:

Please refer to the discussion in #1313 and reopen this if you think it's still valid.

Aayush, I got the point that these are active cases are actually, "the change in the total Active ", but the other 2 charts are not "change in x type of cases".
So, should we rename the chart to "change in the total Active ", as for daily basis, we can't account for total number of cases? Also as the other 2 charts are not depicting "change in cases, but rather" the actual cases" and using this chart as a a chart depicting "change" is actually asymmetric.
One option is to rename the chart to "the change in active cases" (which is logically correct, but asymmetric from the given 2 charts) and other is to limit minimum value to 0.
What do you think?

@vibvibgyor The other three charts are also the "change in x type of cases". They just happen to be the same as the actual cased declared that day. It all boils down to the fact that the other three types of cases are always increasing. Thus, any change (delta) will be the same as the actual new cases. For active cases, there's a difference since they can decrease as well, and hence, the delta loses it's earlier meaning.

@shuklaayush I got your point. Thanks for explaning.


Issue with ./gradlew nativeImage and data jdbc for native  images

There seems to be an issue using the  ./gradlew nativeImage  to create a native image with 2.1.4, jdk11 and graalvm 20.3
for a jdbc app.
The native  command works
➜  jdbc_mirco git:(main) ✗ native-image --no-fallback --class-path build/libs/jdbc_mirco-*-all.jar
[data-jdbc-h2:31924]    classlist:   1,531.65 ms,  0.94 GB
[data-jdbc-h2:31924]        (cap):   3,966.99 ms,  0.94 GB
[data-jdbc-h2:31924]        setup:   4,999.03 ms,  0.94 GB
[data-jdbc-h2:31924]     (clinit):   1,108.58 ms,  4.33 GB
[data-jdbc-h2:31924]   (typeflow):  19,085.36 ms,  4.33 GB
[data-jdbc-h2:31924]    (objects):  33,740.15 ms,  4.33 GB
[data-jdbc-h2:31924]   (features):   2,938.34 ms,  4.33 GB
[data-jdbc-h2:31924]     analysis:  60,342.66 ms,  4.33 GB
[data-jdbc-h2:31924]     universe:   1,883.11 ms,  4.33 GB
[data-jdbc-h2:31924]      (parse):   3,164.06 ms,  4.46 GB
[data-jdbc-h2:31924]     (inline):   5,847.98 ms,  6.48 GB
[data-jdbc-h2:31924]    (compile):  23,975.84 ms,  7.67 GB
[data-jdbc-h2:31924]      compile:  36,239.00 ms,  7.67 GB
[data-jdbc-h2:31924]        image:   5,955.65 ms,  7.69 GB
[data-jdbc-h2:31924]        write:   1,328.11 ms,  7.69 GB
[data-jdbc-h2:31924]      [total]: 112,450.91 ms,  7.69 GB

the task fails, I tried    "io.micronaut.application" version 1.0.5 and  1.2.0
➜  jdbc_mirco git:(main) ✗ ./gradlew nativeImage

> Task :nativeImage
[micronaut-graalvm-app:32024]    classlist:   1,508.32 ms,  0.96 GB
[micronaut-graalvm-app:32024]        (cap):   3,857.28 ms,  0.96 GB
[micronaut-graalvm-app:32024]        setup:   4,873.12 ms,  0.96 GB
[micronaut-graalvm-app:32024]     (clinit):   1,155.05 ms,  5.63 GB
[micronaut-graalvm-app:32024]   (typeflow):  15,022.93 ms,  5.63 GB
[micronaut-graalvm-app:32024]    (objects):  19,724.85 ms,  5.63 GB
[micronaut-graalvm-app:32024]   (features):   4,414.42 ms,  5.63 GB
[micronaut-graalvm-app:32024]     analysis:  44,340.34 ms,  5.63 GB
Error: Unsupported features in 9 methods
Detailed message:
Error: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: VarHandle object must be a compile time constant
Trace:
        at parsing org.h2.util.Bits.readDouble(Bits.java:184)
Call path from entry point to org.h2.util.Bits.readDouble(byte[], int):
        at org.h2.util.Bits.readDouble(Bits.java:184)
        at org.h2.util.geometry.EWKBUtils$EWKBSource.readCoordinate(EWKBUtils.java:217)
        at org.h2.util.geometry.EWKBUtils.addCoordinate(EWKBUtils.java:475)
        at org.h2.util.geometry.EWKBUtils.parseEWKB(EWKBUtils.java:376)
        at org.h2.util.geometry.EWKBUtils.parseEWKB(EWKBUtils.java:292)
        at org.h2.value.ValueGeometry.getEnvelopeNoCopy(ValueGeometry.java:238)
        at org.h2.mvstore.db.MVSpatialIndex$FindBoundsCursor.check(MVSpatialIndex.java:474)
        at org.h2.mvstore.rtree.MVRTreeMap$RTreeCursor.fetchNext(MVRTreeMap.java:527)
        at org.h2.mvstore.rtree.MVRTreeMap$RTreeCursor.hasNext(MVRTreeMap.java:484)
        at javax.security.auth.Subject.hashCode(Subject.java:928)
        at java.util.HashMap.hash(HashMap.java:339)
        at java.util.HashMap.get(HashMap.java:552)
        at com.oracle.svm.jni.access.JNIReflectionDictionary.getLinkage(JNIReflectionDictionary.java:146)
        at com.oracle.svm.jni.functions.JNIFunctions.RegisterNatives(JNIFunctions.java:349)
        at com.oracle.svm.core.code.IsolateEnterStub.JNIFunctions_RegisterNatives_7728ce15b57af339792ad95c60f247990e0df65e(generated:0)
Error: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: VarHandle object must be a compile time constant

Steps to reproduce
Create a new project via mn create-app knoma.web.jdbc_micro --features data-jdbc,http-client,graalvm --jdk=11
Bumped up io.micronaut.application to 1.2.0 and create controller via mn create-controller Person
Next I created the native-image.properties with Args = --verbose --report-unsupported-elements-at-runtime
I also tried
nativeImage {
    args('--report-unsupported-elements-at-runtime')
    imageName('micronaut-graalvm-app')
}

./gradlew nativeImage still fails
but native-image --no-fallback --class-path build/libs/... works after adding the other properties like name and class.

@knoma which GraalVM version are you using, Java 8 or Java 11?

I'm using graalvm64-11.0.9 (20.3)

I also tried to add id "io.micronaut.application" version '1.2.0' to https://github.com/micronaut-graal-tests/micronaut-data-jdbc-graal/tree/2.2.x_h2 and to run the  ./gradlew nativeImage and I got the same failure.

Can you please give it a try with ./gradlew nativeImage and Java 8? I'm asking because I've seen issues like the one you posted with Java 11 so I want to make sure if this is one of those.

I tried the Gradle native image task with jdk8 and graalvm 20.2 see Gitter thread and it worked. I had the hope that jdk11 will work with graalvm

Just tried micronaut-data-jdbc-graal and "io.micronaut.application" version '1.2.0' with ./gradlew nativeImage by using graalvm-ce-java8-20.3.0 which worked. I was able to compile a native image and run it.

I tried this today with graalvm-ce-java11   21.1.0 and micronaut 2.5 and it worked with a new jdk11 project
➜  bigdata-user-signals-kafkaten git:(main) cd ..
➜  Project cd micronaut/jdbc-micro
➜  jdbc-micro ./gradlew nativeImage
Starting a Gradle Daemon (subsequent builds will be faster)

BUILD SUCCESSFUL in 6s
3 actionable tasks: 3 up-to-date

I will close this


ignore killall process not found error

Ok, it wasn't a necessary line.. the init script kills it.


Update tutorial-monitoring.md

Small grammatical fix.

@boykoc : Thanks for your contribution to the Azure documentation! The author, @davidmu1, has been notified to review your proposed change.


Error executing action extract on resource 'libarchive_file[consul.zip]'

I'm still looking into what changed since I had this working last week. I'm getting this error now from the consul::install_binary recipe:
uninitialized constant Archive::EXTRACT_OWNER
Initial googling shows that it's related to the ruby gem that unzips files (libarchive I suppose). I don't believe the version of that has changed, but I can't be certain just yet. I thought it merited an issue in this cookbook repo since it might happen to others. I'll add more info if I get it. If anyone has any suggestions, that would be great.

consul-cookbook depends on libarchive cookbook, which is broken as of chef-dk v0.4
see https://github.com/reset/libarchive-cookbook/issues/9

Is this still an issue with ChefDK 0.6?

#169


Pass through modifiers propType

This is related to #1198.
An alternative approach (which I'm happy to change to) would put the modifiers Set shape in a separate tiny file, and import it where needed.

Coverage remained the same at 84.794% when pulling d1446709ad2ed80a892ec58c4c647cdca0b6a291 on modifiers_proptypes into 6a19133b6138cddadb9a145bf738e3e5542807e7 on master.

@majapw yes, exactly right. Separate file it is!

k updated, PTAL - i also made the propType stricter.


How get a list of all audio albums?

Maybe I'm missing something, but I just can't figure out how to get a list all audio albums with this API.  The equivalent REST API path would be:
/library/sections/<section_id>/albums
I was trying to call "searchAlbums()" with no params, but that was raising an error:
  File "./api-test.py", line 84, in <module>
    main()
  File "./api-test.py", line 79, in main
    generate_playlist("cask.local", name, list_size)
  File "./api-test.py", line 41, in generate_playlist
    all_albums = music_section.searchAlbums()
  File "/usr/local/lib/python2.7/dist-packages/PlexAPI-2.0.0a_.................................._.version.of.this.api-py2.7.egg/plexapi/library.py", line 270, in searchAlbums
    return self.search(libtype='album', **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/PlexAPI-2.0.0a_.................................._.version.of.this.api-py2.7.egg/plexapi/library.py", line 193, in search
    if libtype is not None: args['type'] = utils.searchType(libtype)
  File "/usr/local/lib/python2.7/dist-packages/PlexAPI-2.0.0a_.................................._.version.of.this.api-py2.7.egg/plexapi/utils.py", line 261, in searchType
    if libtype in SEARCHTYPES + SEARCHTYPESSTRS:
TypeError: unsupported operand type(s) for +: 'dict' and 'list'

I think it's an oversight. Audio support is very new and your probably the first person to really dig in. I'll make it work tomorrow.

Cool, thanks for doing all this.  No need to rush on my account, I'm just playing around. :-)
I see now that the code is mostly video oriented.  I was also hoping to get a client object to "playMedia()" on an audio playlist, but looks like that won't work as implemented.  Something to keep in mind!

commit 1a679411dc1d8e3dab76b0344409f9bb50284e6d
Author: Michael Shepanski<EMAIL_ADDRESS>Date:   Tue Apr 12 22:47:46 2016 -0400
Ability to list all albums; Fix setting object values when not found (NA) and we know its the full object

commit c75db1df103f664e7417f06ebc6e54c2323e2651
Author: Michael Shepanski<EMAIL_ADDRESS>Date:   Tue Apr 12 22:55:45 2016 -0400
Fix broken utils.searchType()

Some things that still need to be done here:

Create tests for audio playback.
Ability to play playlists (or convert a playlist to a playqueue).

Changes look great!
 You're doing all the hard research, I love it!

commit 5701af7ed40bac952992820ee57706f037bd7e74
Author: Michael Shepanski<EMAIL_ADDRESS>Date:   Wed Apr 13 22:36:12 2016 -0400
Ability to play playlists


Customize Row Totals

Hi,
I am trying to create pivot table which is depicted in following picture. I need to customize row totals as ClientValue - StaffValue. Could you please provide a way to do that.

Thanks,
Kiran

You'll have to write a custom aggregator. You can look at the existing aggregators as a starting point.

can i get some example aggregater functions

There are some docs here: https://github.com/nicolaskruchten/pivottable/wiki/Aggregators


As a user, I want keyword searches to prioritize matches in the author, title and public notes fields so that I can easily find works using keywords.

notes for testing
Search has been reconfigured so that matches in title, author, and editorial notes result in a higher relevance ranking (current boost levels: author^20, title^10, notes^10). This change in search configuration impacts all keyword searching, so re-testing existing search functionality with this change will be important.
A score field for logged in users has also been added to facilitate this. A keyword match will markedly boost a score, a phrase match (advanced elocution vs elocution) will do so even more.

to test the new relevance ranking, try the following searches in the "keyword or phrase" input, and make sure you have set sorting to relevance:

search on terms that will match author names; to check relevance against text matches, try search terms like blood, wood, cone, french. Items where the author matches should be listed before items with text only matches
similar for terms in title: matches in titles are considered more relevant than matches in full text; possible search terms to try prosody, discourse; title is not boosted as much as author, so author matches should show first; maybe test by search terms like english which appear in both author and title fields
similar for terms in editorial notes: matches there should be more relevant than elsewhere

to test that the updated search configuration hasn't broken anything:

test that searching on all fields included in keyword search still works (book metadata and page content; specific fields are source id, title, author, notes, publication date, enumcron, publication place, publisher)
test that you can still use exact phrase and boolean searches, and they work properly (note that the change in configuration means it's harder to get a syntax error, e.g. with an incomplete exact phrase, because the query parser we're using is more error tolerant)
test that pages with matching text still display with the book metadata and terms are highlighted
test that searching on a term that appears in the full text of a book but nowhere in the metadata displays the book record with matching pages
test that phrase matching boost scores as well (i.e., advanced elocution vs just elocution)

Dev notes
From 2018-11-14 meeting notes :

Title (but boost less)
Author
Notes

Mostly everything with the relevance ranking looks good (I tested everything you suggested, and more--thank you for the detailed notes!) My only question is whether exact matches in the title should be prioritized over stemming searches even without quotes? For example, searching "lives" yields results with "living" and "live" before some results with "lives" in the title. Don't know if you have control over this or not--might just be Solr.
Boolean and exact phrase generally appear fine. Metadata and full text generally appear searchable. Some search issues that might be broken:

Searching a source ID does not work in the backend the same as the frontend--on the backend, it yields the unique result only, searching mdp.39015016748504 as a unit. On the front end, it yields results with the letter combination ("mdp") and even when you put quotations around the source ID seems to be searching the numbers as discrete units (a search of "mdp.39015016748504" yields 614 page hits).
-Enumcron is also a little tricky—searching July-Dec 1890 (no quotes) yields only 1 result in the admin, but 3,000+ in the frontend (it’s searching the months separately as keywords) and searching with quotes yields no results, unless you put in the entire (rather than partial) enumcron for the result I was trying to get (“v. 38 (July-Dec 1890)” yields njp.32101065266304, but "July-Dec 1890" yields nothing). The prod site yields the enumcron result first even though it is also searching the months separately.
-When I search advanced elocution (no quotes) the first result is "Exercises on etymology" (njp.32101074757137) because "elocution" is in in the author metadata field (Graham, William, teacher of elocution). The number 2 result therefore gets booted from number 1 ("Advanced elocution" (uc1.$b293521)) despite being more relevant to the phrasal search. Might just be a quirk of the new boosting--you get uc1.$b293521 when you use quotes around "Advanced elocution."
-When I searched “iamb meter pentameter”~50 the first result only highlighted the word meter on the keyword detail page, even though pentameter and iamb also appeared in the results. Is this syntax not correct?

@bwhicks I'll let you decide whether to close or not--the source ID at the very least seems like a bug that needs fixing, and maybe the proximity search.

Responses to what you've found based on my research so far:

You're right, it does seem to be tokenizing the id. I'm not sure why this is - we have it configured to use the standard tokenizer, which the documentation says should not tokenize on a period that isn't followed by a whitespace. I can investigate more on that one. I was able to confirm what you said in your notes.  I also discovered that the search supports field searching within the keyword - e.g., try srcid:mdp.39015016748504. Do you want this capability? What field name would make sense to you for this? Would you prefer srcid or source_id ?
enumcron is currently configured as a string field and not text, which means it doesn't get tokenized the way other fields do. (I was somehow thinking of it as something for display, not for search!) That should obviously be changed. Also available in the keyword input, e.g. enumcron:"v. 38"
Currently author is boosted more than title, based on comments from MM when we first discussed, but that's easy to revisit. Do you want to try having them boosted the same? Should title be boosted more than author?
I think proximity search is handled automatically now - I'll see what I can share with you in terms of documentation for how phrases are handled and what we can configure. Basically, it's always doing a secondary proximity search and increasing relevance when items match, and we can configure how big the range on that proximity is.

Regarding exact match vs. stemmed term - I looked and found something I can try that might have an impact on that (search on both the tokenized and untokenized version of the field).
Regarding the fields that can be used in the keyword search, I think these are the ones that make sense:

srcid (preference on how to label it? I believe we can easily alias these)
title
subtitle
enumcron
author
pub_date
pub_place
publisher

I can restrict it to make sure only these are exposed, so we don't include things we don't mean people to search on. Let me know if you don't want any of these.

More updates.

According to the solr docs, the source id shouldn't be tokenized on ., but clearly it is getting tokenized. Is it sufficient if we can have the record with that id show up first?
I'm going to change how enumcron is indexed, local tests indicate that addresses some of the problems you're seeing. (will require a full reindex, unfortunately)
Based on your comments, I switched title and author to be boosted at the same level - I think it's behaving better, but there may be searches I'm not thinking of where this is a problem.
Here is documentation on extended dismax query parser - the top has a summary of how it works. It says that full lucene syntax is supported, so the proximity search ought to work. I tried some simpler cases, which I think shows it working ok: "questions purpose"~10 shows the highlighting you would expect. I suspect the highlighting doesn't behave the same way as the querying, and 50 is probably too big a range to display in the one snippet.

Thanks for the updates! My list numbers below don't match yours (I'm responding to both your comments at once since they overlap)

As we discussed in the meeting I think that fielded searching for power users is a great capability--I would want to be able to search all of those fields you listed, and I don't see any problem with letting users do so. As a lay person "source_id" makes more sense to me, especially since pub_date and pub_place already use underscores.
RE: title v author relative boosting. I think the case I tested is a fringe case. My issue was that "elocution" was included in the HT metadata under the author field (Graham, William, teacher of elocution), but obviously isn't an author name. A brief look in OpenRefine suggests that this is pretty uncommon. I need to think more about whether making title and author even will be a problem.
The highlighting for the proximity search case I tested is working now.
I think if you can get the source ID record to show up first (and your time would be better spent elsewhere than fighting with solr on this issue) that would be sufficient. Does it still get tokenized if you field search source_id?

Updated the test site with my latest Solr config and index changes.
Dispensing with numbers because, as you said, they aren't matching up very well anymore!

Try the search with the current boosting.  I've added subtitle (which I think we'd missed), although in the version that's on the test site title and subtitle are weighted evenly, and I'm guessing title should have priority. Regarding boosting on title vs author, I think the question is: if a search term matches both author and title, which record would you want to see first?
Title and subtitle should now match on unstemmed versions of the words first (i.e. lives or living before live); it has occurred to me we should add similar logic to the title and author fields, but I propose we create a separate issue for that and prioritize against other work.
I have changed how enumcron is indexed - searches on this field should work better
Searching by hathi id should result in that record showing first (when you sort by relevance).  If you search with the srcid: notation, you should get only one record (term does not get tokenized).
I agree with source_id but have not yet had the chance to implement.  Perhaps we could track that separately from this issue?

I do think title should be boosted more than subtitle. I hope that's easy enough to change?

Re: title v. author boosting, thank you, the way you asked that question is really helpful. I searched "Johnson", and I am getting works about Johnson (works with Johnson in the title) first, and I think I would expect to get works by Samuel Johnson first (and all clustered together, followed by works about SJ with his name in the title). Sorry for going back and forth on this. Again, I think the "elocution" example I tested was a fringe case. So maybe boost author more over title?

Re: the aggregate question, it looks like things with metadata hits are sorting before things with full-text only hits, which is what we want. The full-text only hits are where it might get dicey? Some works with higher max relevance scores are appearing below some works with lower max relevance scores (on my search of hood, max 5.1 appears before max 7.5). But as I said on Slack, I think it is working well enough that it is not a priority right now.

Tested the stemming issue with lives and lives is appearing first, but then I get Living, Live, Live, living, living living. Is it case sorting? I tested with classical too and it's giving me "classical" first, but then I get some results with classic early in the title instead of one more results with classical later in the title, which is appearing lower on the list.

I thought about whether to integrate this logic for author too, but couldn't think when it would actually be a problem? I don't think there are authors with close enough names? And was there another field you think we need to add the logic too? You mentioned title again but it's already working for title and subtitle? I don't think it's a priority unless you have an argument for it.

Enumcron is kind of working better. It doesn't seem to recognize whether there is a space between the period or not, whether you use quotes or not ("v.1" with quotes doesn't return results with "v.1" in the enumcron; it returns stuff with v 1 in the full text). But it works great if you do a fielded enumcron search. I think it's fine as long as we document in the About section.

Hathi ID search without quotes gives that record first. With quotes OR with srcid: notation you get only one record. (It still tokenizes--it returns 510 matching hits inside the book for instance. But that's fine, I think)

I will make source_id a new issue now.

@mnaydan thank you for your careful testing and detailed notes!
My understanding of what is left to be done, please confirm:

boost title more than subtitle
boost author more than title

I'm not sure what's going on with the stemming (and not sure I'm duplicating it); do you need anything more on it or do you think it's good enough?

Yes! Thank you for summarizing. I think the stemming is good enough if you do. So just those two things, unless you think we need to add the stemming logic to author or another field.

Title is now boosted more than subtitle! (tested with "rules")
Author is now boosted more than title! (tested with "Johnson" and "elocution")


Added sample code for KeyPress events

Summary
Added sample code for KeyPress, KeyUp, KeyDown events
Fixes dotnet/docs#7852

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: hunbernd sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

Thanks, @hunbernd, for adding the Visual Basic code. I'll merge your PR now.


Fixes encryption service tests.

removes unneeded temporary file creation
fixes key size problem
adds new test to validate wrong key size handling
fixes simple SonarQube issues

This pull request makes the build fail.

This pull request makes the build fail.

SonarQube analysis reported 1 issue
Note: The following issues were found on lines that were not modified in the pull request. Because these issues can't be reported as line comments, they are summarized here:

 EncryptionService.java#L94: Private method 'initEncryption' is never used.

All is well.


isoleucine and leucine: make sure that it is considered the same

We need to check that I = L when we count the precursor ions or peptidoforms. If not, it will favour software suites that do not, and duplicate sequences in the output by artificially increasing the counts.

I do not understand, the moment workflows return the peptides they need to be present on proteins. If the workflow randomly choses to either assign I or L for peptides that have both forms in terms of being present on a protein (not sure how much this actually happens for the fastas we have), the specificity should suffer.
Do you have any indication that this is currently happening?

Easy way to make sure this is the case is to just replace all I with L. However I actually do not necessarily like this; it changes the output of the workflow and there might reasons it assigned I over L, e.g., there is a difference in retention time. Although in MS2 you of course cannot resolve, it usually gets resolved with a fasta DB.
Maybe we can first see if this is an issue with the fasta DB? I suspect it is not.

I think the number is substantially low that this is not an issue. Assuming every possibility is identified we are at 692, identification rates are likely much much lower, if present at all. Furthermore, I think the quant accuracy will suffer if the number of identifications is boosted this way. We can still implement this if it is seen as an issue of course.
`from Bio import SeqIO
import re
from itertools import product
from tqdm import tqdm
def generate_tryptic_peptides(sequence, max_missed_cleavages=3):
# Split the sequence on K or R (except if followed by P)
cleavage_sites = [m.start() + 1 for m in re.finditer(r"(?<=[KR])(?!P)", sequence)]
cleavage_sites = [0] + cleavage_sites + [len(sequence)]
peptides = []
for i in range(len(cleavage_sites) - 1):
    for j in range(i + 1, min(i + 2 + max_missed_cleavages, len(cleavage_sites))):
        peptide = sequence[cleavage_sites[i]:cleavage_sites[j]]
        peptides.append(peptide)

return peptides

def generate_il_variants(peptide):
# Generate all possible I/L substitution variants of a peptide
positions = [i for i, aa in enumerate(peptide) if aa in "IL"]
variants = []
for substitutions in product("IL", repeat=len(positions)):
    variant = list(peptide)
    for pos, sub in zip(positions, substitutions):
        variant[pos] = sub
    variants.append("".join(variant))

return variants

def count_il_ambiguities(fasta_file, max_missed_cleavages=3):
peptide_set = set()  # Set of all unique tryptic peptides
# Read the FASTA file and generate tryptic peptides
for record in SeqIO.parse(fasta_file, "fasta"):
    sequence = str(record.seq)
    peptides = generate_tryptic_peptides(sequence, max_missed_cleavages)
    peptide_set.update(peptides)

ambiguity_count = 0

# Check each peptide for I/L ambiguity overlaps
for peptide in tqdm(peptide_set):
    if len(peptide) < 7:
        continue
    if len(peptide) > 45:
        continue
    if peptide[-1] != "K" and peptide[-1] != "R":
        continue

    il_variants = generate_il_variants(peptide)
    overlaps = [variant for variant in il_variants if variant in peptide_set]
    if len(overlaps) > 1:
        print(peptide,overlaps,ambiguity_count)
    ambiguity_count += len(overlaps) - 1  # Subtract 1 to exclude the peptide itself

return ambiguity_count

Example usage
fasta_file = "ProteoBenchFASTA_DDAQuantification.fasta"  # Replace with the path to your FASTA file
il_ambiguity_count = count_il_ambiguities(fasta_file, max_missed_cleavages=1)
print(f"Number of ambiguities caused by I/L indistinguishability: {il_ambiguity_count}")`
Number of ambiguities caused by I/L indistinguishability: 692


stdlib benchmarks regression 2024-07-29

There are some big regressions in *...Map_Id_All_Errors benchmarks. In some cases, even 500%.
Screenshots:

https://enso-org.github.io/engine-benchmark-results/stdlib-benchs.html#org_enso_benchmarks_generated_Map_Error_Benchmark_Vector_ignore_Map_Id_All_Errors

Good commit: https://github.com/enso-org/enso/commit/f31c084f43c33e0d988d1bf2f02d5c93bd1f5db9
Bad commit: https://github.com/enso-org/enso/commit/f0e9616d39d4fc24c8074bd4681fed182d0ede00
Range: https://github.com/enso-org/enso/compare/f31c084f43c33e0d988d1bf2f02d5c93bd1f5db9...f0e9616d39d4fc24c8074bd4681fed182d0ede00

Blocked by #10706. Running Enso-only benchmarks gives nonsense values. For example:
$ enso --run test/Benchmarks Map_Id_All_Errors
Benchmarking 'Map_Error_Benchmark_Vector_ignore.Map_Id_All_Errors' with configuration: [warmup={3 iterations, 3 seconds each}, measurement={3 iterations, 3 seconds each}]
Warmup duration:    18805 ms
Warmup invocations: 1
Warmup avg time:    18725.603 ms (+-NaN)
Measurement duration:    17741 ms
Measurement invocations: 1
Measurement avg time:    17734.925 ms (+-NaN)
Benchmark 'Map_Error_Benchmark_Vector_ignore.Map_Id_All_Errors' finished in 36563.609 ms
Benchmarking 'Map_Error_Benchmark_Vector_report_warning.Map_Id_All_Errors' with configuration: [warmup={3 iterations, 3 seconds each}, measurement={3 iterations, 3 seconds each}]
Warmup duration:    15681 ms
Warmup invocations: 1
Warmup avg time:    15674.661 ms (+-NaN)
Measurement duration:    15361 ms
Measurement invocations: 1
Measurement avg time:    15354.849 ms (+-NaN)
Benchmark 'Map_Error_Benchmark_Vector_report_warning.Map_Id_All_Errors' finished in 31045.643 ms
Benchmarking 'Map_Error_Benchmark_Array_ignore.Map_Id_All_Errors' with configuration: [warmup={3 iterations, 3 seconds each}, measurement={3 iterations, 3 seconds each}]
Warmup duration:    16347 ms
Warmup invocations: 1
Warmup avg time:    16342.969 ms (+-NaN)
Measurement duration:    15964 ms
Measurement invocations: 1
Measurement avg time:    15960.464 ms (+-NaN)
Benchmark 'Map_Error_Benchmark_Array_ignore.Map_Id_All_Errors' finished in 32313.34 ms
Benchmarking 'Map_Error_Benchmark_Array_report_warning.Map_Id_All_Errors' with configuration: [warmup={3 iterations, 3 seconds each}, measurement={3 iterations, 3 seconds each}]
Warmup duration:    16355 ms
Warmup invocations: 1
Warmup avg time:    16352.555 ms (+-NaN)
Measurement duration:    16418 ms
Measurement invocations: 1
Measurement avg time:    16414.449 ms (+-NaN)
...

We first need to fix the JMH stdlib benchmark invocation.

However, the PR that most likely caused the regression is #9625. I cannot verify it at the moment, though.

Let's make sure @GregoryTravis is aware of that.

To me, it seems like #9625 changed the behavior of dataflow errors so that the stack traces are now attached all the time again. They used to be attached only iff JVM assertions were enabled.

@Akirathan it actually isn't the stack trace that is the problem. As @JaroslavTulach suggested, it is the overhead of calling hasContextEnabled each time in DataflowError.withDefaultTrace. In the affected benchmarks ("Map_Id_All_Errors"), it is called 50M times, because it is testing a 100k array/vector with an error on each element. The other benchmarks are not affected in the same way. So the regression was occurring in an extreme test of this code.
I reproduced the regression locally, and it was not adding stack traces at all.
(Of course Stack traces definitely do add overhead, but these will only be added if the user explicitly requests them using Context.Dataflow_Stack_Trace.with_enabled, which is an advanced feature.)
A possible temporary fix for this would be to again require the "-ea" flag to even check the Execution_Environment settings; we could then create a custom flag for this to use instead of "-ea".
However, we might consider leaving it as-is, since this only affects error paths, and seems to only be noticeable in the case of a large number of errors. (In regular usage, we restrict the errors to 100, I believe.)

it is the overhead of calling hasContextEnabled each time in DataflowError.withDefaultTrace

Speed it up!

Pavel, please take a look and speed the hasContextEnabled check up.

Investigating the regression in PR #11153 on latest develop (03369b927321722e35d5bdff2edda3a0a2c1d901) reveals that the problem is not in ExecutionEnvironment.hasContextEnabled, but in EnsoContext.getExecutionEnvironment. To prove this theory, apply the following patch:
diff --git a/engine/runtime/src/main/java/org/enso/interpreter/runtime/EnsoContext.java b/engine/runtime/src/main/java/org/enso/interpreter/runtime/EnsoContext.java
index 1f44b2329..bbb8d8b1a 100644
--- a/engine/runtime/src/main/java/org/enso/interpreter/runtime/EnsoContext.java
+++ b/engine/runtime/src/main/java/org/enso/interpreter/runtime/EnsoContext.java
@@ -874,8 +874,7 @@ public final class EnsoContext {
   }

   public ExecutionEnvironment getExecutionEnvironment() {
-    ExecutionEnvironment env = language.getExecutionEnvironment();
-    return env == null ? getGlobalExecutionEnvironment() : env;
+    return ExecutionEnvironment.LIVE;
   }

   /** Set the runtime execution environment of this context. */

and then, the score of Map_Error_Benchmark_Vector_ignore.Map_Id_All_Errors benchmarks goes from 1.2 to 0.132.
TL;DR; The most important part to optimize is EnsoContext.getExecutionEnvironment and not ExecutionContext.hasContextEnabled.

TL;DR; The most important part to optimize is EnsoContext.getExecutionEnvironment and not ExecutionContext.hasContextEnabled.

I am not completely sure.

   public ExecutionEnvironment getExecutionEnvironment() {
-    ExecutionEnvironment env = language.getExecutionEnvironment();
-    return env == null ? getGlobalExecutionEnvironment() : env;
+    return ExecutionEnvironment.LIVE;
   }

By returning a compilation constant from getExecutionEnvironment you can eliminate cost of other computations based on this constant (including hasContextEnabled), but the problem is: getExecutionEnvironment cannot be a compilation constant  all the time- that was the whole point of #11173
We can speculate with Assumption on the getExecutionEnvironment value to never be changed during execution. It would:

speed the benchmark up
in majority of cases (when Context.with_enabled & co. isn't used) it would speed the CLI execution up
however the IDE will be using #11173 and there the speculation wouldn't help at all

We can go with the Assumption route, but we probably also need to think about optimizing hasContextEnabled.


build(deps): updates Apache Rat from 0.12 to 0.13

Motivation and Context
Updates Rat to 0.13 - the old 0.12 URL results in a 404 (https://dist.apache.org/repos/dist/release/creadur/apache-rat-0.12/apache-rat-0.12-bin.tar.gz).

Thanks to @erisu's hint, I'm closing this due to #247

Here is also the revert PR where we went back to rat 0.12: https://github.com/apache/cordova-coho/pull/252