MongoDB aggregation should perform $match before $unwind

SlamData Community 4.0.10
MongoDB 3.0.1
Windows OS

The following SQL² query takes 15 minutes to execute in user's environment.  Collection size is ~ 200K documents and 5GB total data size.
SELECT measureEnrollments[*].measureKey
FROM memberMeasures
WHERE year=2017 and memberNumber="123456"

It results in the following MongoDB aggregation request:
db.memberMeasures.aggregate(
  [
    {
      "$project": {
        "__tmp4": {
          "$cond": [
            {
              "$and": [
                { "$lte": [{ "$literal": [] }, "$measureEnrollments"] },
                {
                  "$lt": ["$measureEnrollments", { "$literal": BinData(0, "") }]
                }]
            },
            "$measureEnrollments",
            { "$literal": [undefined] }]
        },
        "__tmp5": "$$ROOT"
      }
    },
    { "$unwind": "$__tmp4" },
    {
      "$match": {
        "$and": [
          { "__tmp5.year": NumberInt("2017") },
          { "__tmp5.memberNumber": "123456" }]
      }
    },
    { "$project": { "value": "$__tmp4.measureKey", "_id": false } }],
  { "allowDiskUse": true });

I believe the $match phase should be the first phase in the pipeline rather than second to last.  This would allow MongoDB to utilize it's indexes, which would rapidly and dramatically reduce the resulting data set that the $unwind phase would have to process.

As a point of reference: when the MongoDB query was changed manually in the Mongo shell, and the order operations were changed as suggested, the query returned immediately with correct results.

Another example of this if it helps:
Quasar release 14.5.7, Linux (Ubuntu 12.04. Also tried 16.04), OpenJRE
MongoDB backend: 3.2.10
Collection size: ~1600 documents
Query:
SELECT objects.uri, objects.info.id FROM objects WHERE objects.annotations[*].project_id = "some_project_id"
Document structure:
{   uri: "xxx",    "info": {"id": "yyy"},   "annotations": [     {       "project_id": "zzz", ..     }, ...   ]}
This query takes about 38 seconds to complete. Upon inspection of the aggregation generated, it looks like this:
db.objects.aggregate(   [     {       "$project": {         "__tmp4": {           "$cond": [             {               "$and": [                 { "$lte": [{ "$literal": [] }, "$annotations"] },                 { "$lt": ["$annotations", { "$literal": BinData(0, "") }] }]             },             "$annotations",             { "$literal": [undefined] }]         },         "__tmp5": "$$ROOT"       }     },     { "$unwind": "$__tmp4" },     { "$match": { "__tmp4.project_id": "some_project_id" } },     {       "$project": {         "uri": "$__tmp5.uri",         "id": {           "$cond": [             {               "$and": [                 { "$lte": [{ "$literal": {  } }, "$__tmp5.info"] },                 { "$lt": ["$__tmp5.info", { "$literal": [] }] }]             },             "$__tmp5.info.id",             { "$literal": undefined }]         },         "_id": false       }     }],   { "allowDiskUse": true });
Like mentioned by @damonLL this could also use a $match at the beginning stage of the pipeline. With the $match manually added and tried in mongo shell, the query only takes about 1.8 seconds

@zeeiyerWork Let's make sure this doesn't slip past June. It's a customer request causing some pain.

Thanks @jdegoes and @zeeiyerWork

This is fixed in new Mongo.

Actually, to clarify, this is fixed in QScript, so all QScript connectors will filter before unwinding if the filter doesn’t reference the flattened values.

@sellout In the case above, the filter is using the flattened values though. Did you mean to say that the fix will not add a $match at the first stage for the above example?

@sellout The behavior we need is as much of the filter as can be pushed to the beginning is pushed to the beginning, while the remainder can occur after the unwind.

Yes, thanks for the clarification @emptyslot and @jdegoes – I retract my assertion.
It’s only @damonLL’s original query that is fixed. It’s true that we should pull as much of a filter as possible ahead of an unwind.
@emptyslot I’m not sure how your query can have the $match first. As of MongoDB 3.2, we could take advantage of the $filter operator, and do something like:
[{ "$project": {
     "uri": "$objects.uri",
     "id": "$objects.info.id",
     "__tmp0": { "$filter": ["$objects.annotations", "tmp", { "$eq": ["$$tmp.project_id", "some_project_id"] }] } } },
 { "$unwind": "$__tmp0" },
 { "$project": { "uri": true, "id": true, "_id": false } }]

Is that what you mean, or am I missing some other approach?

@sellout The idea is to use a match stage in the beginning with an $elemMatch on the array that we're unwinding such that the match stage eliminates all documents whose arrays do not contain our predicate. I don't have that manually written aggregate handy, but I can provide that tomorrow if you want.

@emptyslot I think that’s a different query … or, at least in that case, we’d have to first $match with $elemMatch, which eliminates some records, then $unwind, then { "$match": { "__tmp4.project_id": "some_project_id" } } to filter on the remaining records.
Only using $elemMatch, we would get one result for each input field, but the query as stated would give us n duplicate results in the case where one document had n elements with a matching project_id in the annotations array.

@sellout In cases where we can filter some things out, but not everything (e.g. due to MongoDB's "any" semantics with arrays), we should definitely try to do that, knowing that after unwind (etc.) we can perform additional filtering to eliminate all the non-matching elements.

Cool, so I’ll handle this as a few things:

[ ] push parts of a filter that don’t refer to the flattened field before the $unwind (this is a QScript change)
[ ] duplicate other filter components before the $unwind to pre-filter arrays that don’t match at all (this is a Mongo change).
[ ] on 3.2+ use $filter before the unwind (this will be a QScript change to allow “dimensional operations” directly on arrays in MapFunc, and a Mongo change to handle the FilterArray MapFunc that’ll be added by the QScript change).

Sounds awesome. 👍

@sellout @jdegoes Thanks guys! Just to make sure we're on the same page:
@sellout You were right in your comment that we can't use only $elemMatch. My idea was to use $elemMatch to pre-filter (at the first stage of pipeline using match) documents in which the arrays won't match the predicate after unwinding anyway. This is the only stage where MongoDB can use indexes to make aggregation faster. Then, unwind and after the unwind stage, essentially, apply the same filter to eliminate unwound items from the array that don't match the (same) predicate. Note that the set of items that will be filtered will be much less in this case because of the first stage match doing $elemMatch.
I hope I've not confused things further :)

@sellout @jdegoes It would seem that pre-filtering arrays could also be handled as a QScript change via another new MapFunc for arrays, ArrayExists. All mongo versions would support ArrayExists via $elemMatch and then post-filter the unwound values and 3.2+ would additionally support FilterArray via $filter which would eliminate the need for the post-filter.

@wemrysi
This can already be accomplished without any QScript changes.
ArrayExists can be emulated with EXISTS(array[_].foo > 2), where EXISTS is the normal SQL reduction that returns true if at least one element of the set is true. So if it would be helpful, perhaps QScript can generate these in order to pull up additional pieces of a filter earlier in the pipeline.

@jdegoes Yes, that's true but every connector would still then have to recognize that pattern and translate it into an array-level operation. To clarify things a bit, ArrayExists and FilterArray are optional operations that connectors with support for array-level operations can utilize in their implementation, they aren't something that would be exposed via SQL^2 (see #2449 for some detail). I was just suggesting that we handle what was described as a mongo-specific task (pre-filtering arrays for existence) in a more general way such that any connector with capability for array-level operations could benefit.

@wemrysi
What you propose makes it more difficult, not easier, to support new connectors, since every connector must support EXISTS(array[_].foo > 2). So if they all have to support that, then making them also support ArrayExists will be more work, not less.
Additionally, if a connector has support for array-level operations, then if a user specifies a query like EXISTS(array[_].foo > 2), then of course we would want to accelerate that by using the connector's native support for this operation, rather than utilizing a slow, suboptimal way of implementing the query.
If we want to detect and accelerate that in backends that support array-level operations, then it makes more sense, in my opinion, to not natively support these as MapFuncs, since that doesn't really solve the general problem but is a hack around the lack of a general solution.
That's my 2 cents anyway.

That said I don't have major preference since you can introduce a combinator for translating ArrayExists to ordinary SQL2, but it still means if a connector relies on this approach, it will fail to optimize user-land queries.

@jdegoes
No connector will ever have to support ArrayExists, array-level capabilities are just that, a capability which some may have and others not.
As you say, all connectors must support EXISTS(array[_].foo > 2), which would be translated to a QScript representation involving a reduction, left-shift and a filter, if I'm not mistaken. However, if a connector indicates to the QScript compiler that it has the capability for array-level operations, then, instead of the operations I mentioned, the connector would receive an optimized QScript program equivalent to (pseudocode) ArrayExists(array, _.foo > 2) obviating the need for that connector to detect these sorts of patterns in its own implementation.
Thus only those connectors that indicate support/desire for such array-level operations will ever see them used in a QScript program. This is the same concept as with FilterArray mentioned above.
Does this clarify things or have I missed your point?

@wemrysi I think I understand now, and if it's totally opt-in on a per-connector basis, and also optimizes user-land queries that effectively do the same, it makes total sense. 👍

@beckyconning This is an important issue we can start tackling now. Changes won't be visible until QScript filter bubbing is done, but we can make sure the Mongo connector plans the expected QScript correctly.


Add ASCII word boundaries to the lazy DFA.

In other words, \b in a bytes::Regex can now be used in the DFA.
This leads to a big performance boost:
sherlock::word_ending_n                  115,465,261 (5 MB/s)  3,038,621 (195 MB/s)    -112,426,640  -97.37%

Unfortunately, Unicode word boundaries continue to elude the DFA. This
state of affairs is lamentable, but after a lot of thought, I've
concluded there are only two ways to speed up Unicode word boundaries:

Come up with a hairbrained scheme to add multi-byte look-behind/ahead
to the lazy DFA. (The theory says it's possible. Figuring out how to
do this without combinatorial state explosion is not within my grasp
at the moment.)
Build a second lazy DFA with transitions on Unicode codepoints
instead of bytes. (The looming inevitability of this makes me queasy
for a number of reasons.)

To ameliorate this state of affairs, it is now possible to disable
Unicode support in Regex::new with (?-u). In other words, one can
now use an ASCII word boundary with (?-u:\b).
Disabling Unicode support does not violate any invariants around UTF-8.
In particular, if the regular expression could lead to a match of
invalid UTF-8, then the parser will return an error. (This only happens
for Regex::new. bytes::Regex::new still of course allows matching
arbitrary bytes.)
Finally, a new PERFORMANCE.md guide was written.

This
state of affairs is lamentable, but after a lot of thought, I've
concluded there are only two ways to speed up Unicode word boundaries:

Here's another idea: what if the DFA ran even when there were Unicode word boundaries? Instead of actually trying to match them, we could just ignore them. If the DFA reports a match, then it could be filtered through a (slower) engine that does support Unicode word boundaries. The key insight here is that word boundaries are assertions, and as such, never extend or shorten a match. The problem with this approach is that it could result in quadratic behavior, since if the filter rejects the DFA's match, the search would have to start again immediately after the start of the previous search. sigh


text:visible and outline:visible style parameters

text:visible and outline:visible does not appear to work on tangram JS.
Based on the docs, outline parameters can have any parameters which a lines style have, similarly text under the points style can have any parameter which a texts style can have.
Refer tangram-es PR: https://github.com/tangrams/tangram-es/pull/791
We noticed this because outdoor-style started using text:visible parameter and we were getting a bunch of warning on ES.

Clarification: text:visible does work for the standalone text style, but you are correct that it does not work for text nested under points, nor for outline (nested under lines).
I didn't necessarily intend for this to work (since visible is handled as generically either a draw group-level or style layer-level parameter), but I see that it follows from the way the docs are phrase, and is a reasonable commonsense interpretation.
So I'm open to adding that behavior, if there's consensus that it's desirable.
cc @blair1618 @nvkelso

The reason we found out about this, was because it was in use in the outdoor-style.  Yes, sometimes we might want to only show the icon part of the point label, but not it's icon. This junk style code is me trying to accomplish that.

require icon & maybe text

(default in v0.8)
icons: { visible: true } on the point style implies child text symbolizer is also optionally visible.

require icon & require text

@bcamper you said this is not currently supported?
if the icon can fit, but the text can't then whole thing should optionally fail
would require new optional property in point > text symbolizer.
Rational: There are a ton of orphan icons on the outdoor style today (and other house styles) because of this, usually related to the text portion colliding a tile boundary.

require icon & hide text

sounds like this is not currently supported, focus of this issue
icons: { visible: true, text: { visible: false } }
Rational: for some features like mountain peaks where the icon itself holds up well and the name is interesting but not essential, it's nice to sometimes hide the text for a few zooms.
Workaround? Maybe setting the text size to 0 for that zoom?

no icon, require text

the only way I've found to do this so far is to disable the points block and reenable the old text symbolizer.
It's be better to optionally allow the icon of the point to be visible separate from the total combined symbolizer.

no icon, no text

icons: { visible: false }

require icon & require text
- @bcamper you said this is not currently supported?
- if the icon can fit, but the text can't then whole thing should optionally fail
- would require new optional property in point > text symbolizer.
- Rational: There are a ton of orphan icons on the outdoor style today (and other house styles) because of this, usually related to the text portion colliding a tile boundary.

I'm looking at adding this in an upcoming release.

require icon & hide text
- sounds like this is not currently supported, focus of this issue
- icons: { visible: true, text: { visible: false } }
- Rational: for some features like mountain peaks where the icon itself holds up well and the name is interesting but not essential, it's nice to sometimes hide the text for a few zooms.
- Workaround? Maybe setting the text size to 0 for that zoom?

It shouldn't be difficult to make text nested in points respect the visible property (same for outline for consistency). @tallytalwar does this pose any issues for you to add?

no icon, require text
- the only way I've found to do this so far is to disable the points block and reenable the old text symbolizer.
- It's be better to optionally allow the icon of the point to be visible separate from the total combined symbolizer.

Ah, the intention was for the standalone text style to be used for that -- it wasn't meant to be old/deprecated so much as continue to be used for such cases where you are only rendering text (like road labels too). I'd have to think more about allowing the points portion to be "empty", because there are already several assumptions that the "parent" part of that relationship is renderable.

@bcamper Already have the work done for ES. https://github.com/tangrams/tangram-es/pull/791. Was waiting on a decision to be taken for its official support :D.

visible support for points > text and lines > outline is added in https://github.com/tangrams/tangram/commit/053451d76d34c8f9a08d78654e0883fbb25fa815 & https://github.com/tangrams/tangram/commit/345044906d7f4f007b410ee8a1036721c8561f8a, and will be in the next release, 0.8.2.

Released in v0.8.2.


Move Cert Manager to operators directory

Move cert manager to operators directory and install via gitops as it's required for tekton-results.

@CathalOConnorRH I'm afraid a rebase is required.

@CathalOConnorRH I'm afraid a rebase is required.

Rebased now.


Use a configuration first method (modifiers and states) for Title

Relates directly back to #273

I'm inclined to leave this one as-is. We could write 2 vars for line-height and 2 vars for font-weight, then reference those 2 vars in the modifiers instead of having a unique line-height and font-weight var per modifier, but this seems like something a person might want more granular customization of than that. What do you think @mattnolting @christiemolloy?

Maybe keep the variables in the head as is, so that a person can still customize them. But in the body I think you could group the modifiers that share line height and font weight. That seems consistent with the updates we've been making.
But either way is good honestly.

@christiemolloy I ran this by @mceledonia and he said just group 'em for this and the pf-c-content titles. woot!

@christiemolloy I'm just going to leave this one since we haven't decided what we're going to do re: combining vars. Does it look like there is anything else to fix to you with this component? If not feel free to close it out.

I don't see anything else. Closing the issue


Fix PropertyMethodAssignment test

The test was testing the name of a method. With newer versions of
Node, Node uses the ES6 semantics for inferring the function name
which leads to test failures.

@johnjbarton TBR
@mathiasrw FYI


New control interface, extending worker/controller for multi-gpu/node

Working towards multi-gpu: Use collectives module from pygpu
Working towards multi-node: Use shared memory and MPI after NCCL collectives from pygpu
It needs Theano/libgpuarray#224
How does it seem so far? @abergeron

This does three things which I'm not so sure of:

break the interface for Controller and Worker.  It will require adaptation of all existing code (again) which people will not be happy about.
Force a dependency on mpi4py which currently serves no purpose.
Force a dependency on pygpu even if you don't want to use it.

Instead of doing that, you should make an alternate implementation of worker and controller for synchronous training (the previous one only supports asynchronous) and make the dependencies optional.

Can you please address my general comments on this?  Even if you don't have code for it, just a textual answer is ok.

You are right. Sorry for answering late.

I will refactor properly so that code using the existing interface won't break. I will make some internal changes though that will not affect their functionality, e.g. another worker spawning method and error handling and clean up (that is a necessity for multi-node, so instead of having two ways doing this I will migrate the old one too)
I need mpi4py as an MPI wrapper in order to have multi-node support. See here, I have revised some things though, but in general it is the same. Of course it should be a optional dependency needed if someone wants to use multi-node supported with the new API.
You are right. This should be an optional dependency. But it is required to use the new API as it has the gpucomm functions that allow multi-gpu through NCCL

@abergeron I need some help. Is pygpu.GpuArray or theano variable a hashable type? Because I want to map GpuArrays to shared memory buffers in a host. link
Also, how can I access internal pygpu.GpuArray of a theano gpu variable in order to use it with pygpu collectives interface?

The new code looks good and doesn't remove the previous interface.  However even moving classes around breaks the import.
To fix this create a channel.py file that imports what was previously defined it that file.

I have done this but I hesitate to push the commit. Do you have a few minutes to chat about something? I will push this commit in 5 minutes. Will you be available?

It concerns compatibility issues with the previous interface. In the new code, I change the way things are setup and finalized in order to support multi-node and to have tighter control over errors mainly. But, this means I have to change things too in the examples so that they work too with the new launcher (without necessarily using the new interface). I do not want to break things or compel others to do a lot of changes into their work, but I think we should provide a uniform and single way to do things (write new worker files, custom controller files or launch the training procedure). So I want to propose some things and I want to know if you agree with these, before I start changing code.

@abergeron I think I have completed the synchronous SGD task. How does it seem to you?

With the last two nits (comment and missing line), I believe this is ready for a merge but I'll run the tests myself on Monday to make sure.
I assume that you ran them yourself and fixed the problems you encountered :)

I will upload in a gist a bash script for msub that I use to run tests in Helios.
Until Monday, I will have updated documentation too. Things for single-node run as expected. Although I have encountered things in multi-node that make my life difficult right now: making the deployment work with the PBS which as I understand now, it uses MPI to dispatch jobs too.

Barring issues with multi-node, can I assume that this is finished?

I would go to merge now if the answer is yes. Would there be something else
to prevent the merge now of this PR?
thanks
On Mon, Aug 22, 2016 at 2:36 PM, abergeron<EMAIL_ADDRESS>wrote:

Barring issues with multi-node, can I assume that this is finished?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/mila-udem/platoon/pull/66#issuecomment-241507710, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AALC-44Uyy_v7SGKWVus2sRNdl9Z-QeXks5qiew4gaJpZM4JOXc8
.

Apart from testing the multi-node case, everything else is finished in this PR and tested. Please wait as I double-check and execute the tests once more.
Will you have time to discuss about future feature in Platoon?
Also, should I bump up the version to 0.6.0 in setup.py?

@nouiz @abergeron Everything is alright.

What do you want to discuss? How much time you think it will take?
I'm completly busy Wednesday and Friday this week. So if it is short, we
could do it Tuesdays or Thursday. What is your timezone and availability
time again?
Fred
On Mon, Aug 22, 2016 at 3:40 PM, Christos Tsirigotis <
<EMAIL_ADDRESS>wrote:

@nouiz https://github.com/nouiz @abergeron
https://github.com/abergeron Everything is alright.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mila-udem/platoon/pull/66#issuecomment-241525896, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AALC-6p0BcJ5QfFjEaASc1NtIByikpVQks5qifsxgaJpZM4JOXc8
.

On Mon, Aug 22, 2016 at 12:55 PM, Frédéric Bastien<EMAIL_ADDRESS>
wrote:

What do you want to discuss? How much time you think it will take?
I'm completly busy Wednesday and Friday this week. So if it is short, we
could do it Tuesdays or Thursday. What is your timezone and availability
time again?

Fred, is this message for me?

Fred
On Mon, Aug 22, 2016 at 3:40 PM, Christos Tsirigotis <
<EMAIL_ADDRESS>wrote:

@nouiz https://github.com/nouiz @abergeron
https://github.com/abergeron Everything is alright.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mila-udem/platoon/pull/66#issuecomment-241525896,
or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AALC-
6p0BcJ5QfFjEaASc1NtIByikpVQks5qifsxgaJpZM4JOXc8>

.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/mila-udem/platoon/pull/66#issuecomment-241529907, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABAK51DVcnh1BXjyiGSQ64o2CG_9VfTtks5qif6VgaJpZM4JOXc8
.

@nouiz I would like to prepare the final report. If it is to be merged, I would like to ask you to merge this PR soon enough please so I can use the links from Platoon, because in 20 hours I will have to have submitted the final evaluation.

Thanks. merging.

@yaoli, no it was for christos
On Mon, Aug 22, 2016 at 4:07 PM, yaoli<EMAIL_ADDRESS>wrote:

On Mon, Aug 22, 2016 at 12:55 PM, Frédéric Bastien <
<EMAIL_ADDRESS>
wrote:

What do you want to discuss? How much time you think it will take?
I'm completly busy Wednesday and Friday this week. So if it is short, we
could do it Tuesdays or Thursday. What is your timezone and availability
time again?

Fred, is this message for me?

Fred
On Mon, Aug 22, 2016 at 3:40 PM, Christos Tsirigotis <
<EMAIL_ADDRESS>wrote:

@nouiz https://github.com/nouiz @abergeron
https://github.com/abergeron Everything is alright.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mila-udem/platoon/pull/66#issuecomment-241525896,
or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AALC-
6p0BcJ5QfFjEaASc1NtIByikpVQks5qifsxgaJpZM4JOXc8>

.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/mila-udem/platoon/pull/66#issuecomment-241529907,
or mute
the thread
<https://github.com/notifications/unsubscribe-auth/
ABAK51DVcnh1BXjyiGSQ64o2CG_9VfTtks5qif6VgaJpZM4JOXc8>

.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mila-udem/platoon/pull/66#issuecomment-241533467, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AALC-0erY6AQiqlSBYm7sG-MxDe7w2UIks5qigGXgaJpZM4JOXc8
.


passport dependency with passport-saml dependency

Trying to use passport 0.3.0
Getting error: TypeError: Cannot set property 'user' of undefined [TypeError: Cannot set property 'user' of undefined] TypeError: Cannot set property 'user' of undefined
at /project/node_modules/passport-saml-encrypted/node_modules/passport/lib/http/request.js:50:35
Seem's to be need same fix as fixed on https://github.com/bergie/passport-saml/commit/010f2d4373719cf34b6dc241d51e76d8e4b1c2f2

I'm up to my neck right now, and can't really verify.
Would you mind checking if this works on this repo as well? (If it does, and you send me a PR, I'd be very grateful! )

Ok. Done and tested on openidp.feide.no


TypeScript coding guidelines say to not prefix interface with I

Bug Report

TSLint version: 3.15.1
TypeScript version: 2.5.3
Running TSLint via: (pick one) CLI

TypeScript code being linted
// code snippet
interface TypeConfig { }

Actual behavior
[tslint] interface name must start with a capitalized I (interface-name)
Microsoft suggests interface names to not  be prefixed with an I - https://github.com/Microsoft/TypeScript/wiki/Coding-guidelines#names

Note that these are the guidelines for writing the typescript compiler itself. It's not meant to be a style guide for any other typescript project out there.
If you want to adopt this style guide for your projects, you can just change the rule's config in your tslint.json to "interface-name" : [true, "never"]

50 cent from a passer-by:
"interface-name" : [true, "never-prefix"] instead of
"interface-name" : [true, "never"]
🙌
rule docs

I tried turning off the "I" prefix by adding interface-name to the default tslint.json that's generated by Create React App. However, it doesn't seem to work. Maybe I'm putting it in the wrong location?
{
  "extends": [
    "tslint:recommended",
    "tslint-react",
    "tslint-config-prettier"
  ],
  "linterOptions": {
    "exclude": [
      "config/**/*.js",
      "node_modules/**/*.ts"
    ]
  },
  "interface-name" : [true, "never-prefix"]
}

Nevermind, I figured it out:
{
  "extends": [
    "tslint:recommended",
    "tslint-react",
    "tslint-config-prettier"
  ],
  "linterOptions": {
    "exclude": [
      "config/**/*.js",
      "node_modules/**/*.ts"
    ]
  },
  "rules": {
    "interface-name" : [true, "never-prefix"]
  }
}

For some reason "interface-name": [true, "never"] doesn't work for me. But "interface-name": false works fine

@vasilev-alex "interface-name": [true, "never-prefix"] empasis on never-prefix

I'm still getting "interface name must start with a capitalized I"
this is my tslint.json:
{
  "rules": {
    "interface-name": [
      true,
      "never-prefix"
    ]
  },
  "extends": [
    "tslint:recommended",
    "tslint-react",
    "tslint-config-prettier"
  ],
  "linterOptions": {
    "exclude": [
      "config/**/*.js",
      "node_modules/**/*.ts",
      "coverage/lcov-report/*.js"
    ]
  }
}

What am I doing wrong?

@offg777 Nothing obvious. Link to repo?

The config looks correct to me - are you sure theres nothing else overriding it? The rule is definitely working properly for me

@offg777

"interface name must start with a capitalized I"

From where do you get this error? From your IDE or when running a CLI?
And I don't know about the exact cascading rules of tslint, but have you tried putting the "rules" block underneath "extends"?

厉害

Does eslint support that already?

Setting
"rules": {
    "interface-name": [
      true,
      "never-prefix"
    ]
  },

results in
Configuration for rule "interface-name" is invalid
Maybe this rule exists for a reason? Should every interface have a meaningful name obviously different from an implementing class?

 Obtrusion of your own naming conventions in programming tools is a bug of the product developer neural network ;)

Good thing TSLint is deprecated and this issue is no longer relevant, then! #4534


Panel Does Not Render When The First Datasource Does Not Have Data

I've noticed an issue where the Business Text panel renders the default content if the first (Infinity) data source does not have data. The only configuration needed to reproduce the issue is to create 2 or more Infinity data sources, and to change the Render Template to All Data.
The following JSON would cause the Business Text panel to render the default content:
[
  [],
  [{"some":"some"}],
  [{"data":"data"}],
]

The following JSON would behave as expected and the Business Text panel will render the content using the handlebars and Javascript code:
[
  [{"some":"some"}],
  [],
  [{"data":"data"}],
]

I have also attached an exported sample dashboard. The Business Charts panel does not seem to suffer from this problem.
Business Text With Multiple Infinity Datasources-1724948651025.json

@bliu13 Thank you for submitting the issue. We will investigate and update in the upcoming release.

I initially thought this issue only applied to the Business Text plugin, but it looks like all built-in and plugin visualizations are behaving the same way on the latest versions of Grafana.

@bliu13 Grafana Cloud or OSS? There is a difference between them in the latest versions. Grafana Cloud has Scenes Dashboards enabled by default.

@bliu13 Thank you. We will look into it.


[5.0] Add new Verify Post Size middleware

So I was doing some AJAX file uploading, and kept getting random TokenMismatchException errors. After much debugging - I realised the issue related to post_max_size.
If the file I was attempted to upload was greater than my post_max_size - PHP clears the super globals of $_POST and $_FILE - but still sends the server the POST request (which is now empty).
The problem is that the empty post hits Laravel, and when doing CSRF detection there is no _token - so you'll get a TokenMismatchException, leading you on wild goose chases as to why your forms are not sending correct tokens.
It also means that the user will be shown an incorrect error. Rather than be told they are uploading a file that is way to large - it just throws a 500 error with the wrong message.
I did some investigating - and there is a way to detect this situation. I propose we include this as part of the HTTP Middleware.
Originally mentioned here: https://github.com/laravel/framework/issues/8202
The getPostMaxSize() function is using a script similar to the one described on the php.net page: http://php.net/manual/en/function.ini-get.php

then we need to refactor the other middleware exceptions

Which other exceptions? We don't have contracts for session as of now so that would exclude csrf token exception. Other than that why would we need to refactor Symfony Http Exception, there no reason to extends that on illuminate/contracts.

Ok - makes sense then :)


Process Isolation is very slow as compared to HyperV Containers on Server 2019

Describe the bug
Slowness in cloning source when running multiple containers simultaneously in process isolation.

Isolation Mode
Time in Git clone
Containers running in parallel
Comments

Process
9 mins
1

HyperV
8.5 mins
1

Process
21 mins
10
<-- This is the problem

HyperV
11 mins
10

As the number of containers increases on the server, the performance of container slows down significantly but only in process isolation. I am not worried about minor performance differences. The same also happens when I compile in these containers using nmake. The performance degrades in process isolation.
These 10 containers I mentioned above are triggered by a Jenkins pipeline using Kubernetes. Here is the yaml code I used:
apiVersion: v1
kind: Pod
spec:
  tolerations:
  - effect: NoSchedule
    key: custom/build-hosts
    operator: Exists
  containers:
  - name: jnlp
    image: <image link redacted>
    command:
    - powershell
    args:
    - cp -R C:\\privconf\\*  C:\\Users\\ContainerAdministrator;
    - C:\\jenkinsscript\\jenkins.ps1
    resources:
      limits:
        cpu: 12
        memory: 16Gi
      requests:
        cpu: 12
        memory: 16Gi
    env:
    - name: MY_POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
    - name: MY_HOST_NAME
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName
    volumeMounts:
    - mountPath: /privconf
      name: credential-volume
    - mountPath: /gitcache
      name: cache-volume
    - mountPath: /jenkinsscript
      name: jenkins-script
  volumes:
  - hostPath:
      path: D:/agentconf
      type: ""
    name: credential-volume
  - hostPath:
      path: D:/agentcache
      type: ""
    name: cache-volume
  - configMap:
      defaultMode: 420
      name: jenkins-script
    name: jenkins-script
  nodeSelector:
    custom/fcds: test_akshay

The HyperV Data was gathered using Docker Swarm, as K8S doesn't support HyperV Isolation.
dockerSwarm {
    label "docker-agent"
    image "<image link redacted>"
    limitsNanoCPUs<PHONE_NUMBER>0
    limitsMemoryBytes<PHONE_NUMBER>4
    reservationsNanoCPUs<PHONE_NUMBER>0
    reservationsMemoryBytes<PHONE_NUMBER>4
}

The physical host that I ran it on is a bare metal server, with 208 logical cores (104 physical cores) after Hyperthreading enabled.
To Reproduce
Please trigger 10 parallel containers on the same host at the exact same time, cloning the exact same repository, and that way you should be able to reproduce the issue.
Expected behavior
The expectation is for Process Isolation to work on par or better than HyperV Isolation.
Configuration:

Edition: Windows Server 2019
Base Image being used: jenkins/inbound-agent:3107.v665000b_51092-7-jdk11-windowsservercore-ltsc2019
Container engine: Docker
Container Engine version:

Client:
Version:           25.0.0
API version:       1.44
Go version:        go1.21.6
Git commit:        e758fe5
Built:             Thu Jan 18 17:10:49 2024
OS/Arch:           windows/amd64
Context:           default

Server: Docker Engine - Community
Engine:
 Version:          25.0.0
 API version:      1.44 (minimum version 1.24)
 Go version:       go1.21.6
 Git commit:       615dfdf
 Built:            Thu Jan 18 17:09:34 2024
 OS/Arch:          windows/amd64
 Experimental:     false

Additional context
I have verified that there is no resource over provisioning and my Windows defender is disabled, and all my processes (including git and git-lfs) and directories where source code is checked out are part of exclusion list. As mentioned here: https://github.com/microsoft/Windows-Containers/issues/149
Also verified I have the Defender fix, which was released here: https://github.com/microsoft/Windows-Containers/issues/345

Hey @saraf-akshay, could you share what you're seeing with Windows Server 2022 process isolation?
We don't ship OS level fixes anymore (only security fixes) for Windows Server 2019 now because it is now out of mainstream support: https://learn.microsoft.com/en-us/lifecycle/products/windows-server-2019

@fady-azmy-msft : Thanks for your response. I'm working on preparing a server with Server 2022. It might take a couple days. I'll keep you posted.

@fady-azmy-msft ,@ntrappe-msft : There is still slowness.
Server 2022 is a lot better with Server 2019. Server 2019 was 2x slower, whereas Server 2022 is 1.25x slower in Process Isolation as compared to HyperV Isolation when I run 10 containers in parallel on a host, (essentially trying to run host at its full capacity) with resource (CPU and Memory) restriction as showed in my first comment's yaml file.

Here is what I have experienced with process isolation compared to Hyper-V isolation. I seen cascading container failures and even containers that crash and cannot recover EVER they have to be redeployed.
https://github.com/Azure/Azure-Data-Factory-Integration-Runtime-in-Windows-Container/issues/7

Hello @Howard-Haiyang-Hao @fady-azmy-msft @ntrappe-msft
Just checking in, Any update on this?

Im now running 80+ SHIR containers with hyper-v isolation successfully with little to no issues. Without hyper isolation the max that I could run was about 25+- and that also created issues that cause the container to completely corrupt its self at random.  Please make a Linux compatible SHIR application for ADF / Synapse!

can you run this without using host paths?

This is still very much an issue for AKS users, and was raised to MSFT support (#2309150040010155) back in October of 2023. With a similar finding here: https://forums.docker.com/t/docker-slower-to-copy-files-and-run-compiler-in-server-2019-than-windows-10/113938/2
Since AKS does not support hyper-v, only process isolation.
It's on their roadmap, but no date: https://github.com/Azure/AKS/issues/1792
MSFT support eventually told us to try Linux containers since there was no resolution in sight. This was despite 4 months of back and forth with enterprise support.


Fix json tags

/kind bug

Looks like I can't override
/test pull-kueue-test-integration-main

/retest

/test pull-kueue-test-integration-main


avx512 abs: refine GCC compiler checks for _mm512{,_mask}_abs_pd

Fix check for gcc bug in abs.h .
The gcc bug is described here:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87467
It is fixed in versions 7.4.0 and 8.3.0 and above.

Thank you @thomas-schlichter ; if you know of a project using SIMDe that can be listed publicly, it would be nice to add that to https://github.com/simd-everywhere/simde/wiki/Projects-Using-SIMDe

Yes, OpenAirInterface (https://gitlab.eurecom.fr/oai/openairinterface5g/) is using it.  It is quite a big project and we are just one of many contributors, so I don't know if I would have the right to add it there myself... ;-)

Yes, OpenAirInterface (https://gitlab.eurecom.fr/oai/openairinterface5g/) is using it. It is quite a big project and we are just one of many contributors, so I don't know if I would have the right to add it there myself... ;-)

That is really cool! I've gone and added it to the list myself.


replace Meilisearch to Meilisearch core milli

Feature Request
replace search engine to Meilisearch core https://github.com/meilisearch/milli
the issues:
Milli can manage one and only one index. MeiliSearch manages the multi-index itself. Milli is unable to store updates in a store: it is the job of something else above and this is why it is only able to process one update at a time.

milli only support for one index.


DE<>GIS Automations - Phase 2

Project Description
We've set up the scaffolding to automate the push to Socrata and do validation on datasets. Now we need to implement this for all our (and GIS') products.
Outcomes
The deployments to Socrata are entirely automated for all products.
Timing
This will require a lot of coordination, and modification to existing datasets to fix issues. This will take months.
Tasks
Tracking here
(last column)
Motivations
...
Notes
...

notes from 4/23

let's use the progress indicators from the GDE biweekly
having files in edm-distributions bucket is important (whether it's for an API or manual distribution to Socrata)
Socrata dataset types

old map types block API use
to deprecate, we'd like to make an old dataset "useless" and point to the dataset that replaces it

prepend title with something like DEPRECATED
change description to point to new dataset
update with row-less data

datasets to prioritize

those that GIS is scheduled to update soon and/or monthly

next steps

ask GIS about their upcoming dataset updates
confirm automation progress goals and track (possibly in Open Data excel sheet)
ask GIS to choose a project lead to streamline comms and planning

Also adding: goal of automations is 80% by end of Q2 (July 1). 100% by Q3, per Strategic Planning

@croswell81 FYI, this is our main issue for the project.

[like]  Matthew Croswell (DCP) reacted to your message:

From: Alex Richey @.>
Sent: Wednesday, April 24, 2024 9:12:34 PM
To: NYCPlanning/data-engineering @.>
Cc: Matthew Croswell (DCP) @.>; Mention @.>
Subject: [EXTERNAL] Re: [NYCPlanning/data-engineering] DE<>GIS Automations - Phase 2 (Issue #663)
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.  Forward suspect email to<EMAIL_ADDRESS>as an attachment (Click the More button, then forward as attachment).
@croswell81https://github.com/croswell81 FYI, this is our main issue for the project.
—
Reply to this email directly, view it on GitHubhttps://github.com/NYCPlanning/data-engineering/issues/663#issuecomment-2075853066, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AICWQSV3OXM3XIHSA34TWRTY7AN4FAVCNFSM6AAAAABEM2OKJ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZVHA2TGMBWGY.
You are receiving this because you were mentioned.Message ID: @.***>


Add ability to re-prioritize recipe handlers

I would like to change the order of recipe handlers (pulverizer, sag mill, shaped crafting, shapeless crafting, bee breeding etc.) so they show up in a different order when a recipe is looked up.

Try updating NEI Integration. I believe I fixed this a while ago.


'TypeError: Cannot convert object to primitive value' thrown when trying to serialise object values with null prototypes

Some libraries that we use create objects using Object.create(null) (possibly for performance reasons). The objects therefore have no valueOf or toString functions and thus cause the serialiser to bomb at https://github.com/logentries/le_node/blob/master/src/node_modules/serialize.js#L48 with the above TypeError.

Thanks, can’t say I anticipated that. I’ll add some guards for this case.

Cool, thanks! I just realised I can put my own guards in a custom replacer. I used the following:
if (_.isObject(val) && ! (val.toString || val.valueOf)) return JSON.parse(JSON.stringify(val)); //May not be that performant


Create first_n_primes.py

This code will help us to get first n prime numbers.

Thanks for giving me this opportunity.


LSP segfaults when referencing an alias that points to a non-existing constant

Input
For some reason, I can reproduce this consistently on my machine, but it does not happen on sorbet.run.
This was happening in the Ruby LSP, so I thought it could be something related to the whole project. However, I created a brand new folder where all I have is

Gemfile with sorbet-static-and-runtime
sorbet/config with just --dir .
An example.rb file containing the following code

A = NON_EXISTING_CONSTANT
A

Observed output
Segfault.
Expected behavior
No segfault.

I can't reproduce. When I try the same thing, I get this error:
❯ /home/jez/bin/sorbet --lsp
Argument ‘--dir .’ starts with a - but has incorrect syntax. To see all available options pass --help.

I think maybe you want either `--dir=.` or

--dir
.

in the config file.

Sorry, that was a mistake in the bug report. I did use
--dir
.

It's definitely a weird one, I was wondering if it would be reproducible by others. It's also interesting that it doesn't happen when running the typechecker on the command line, only in the LSP.

I confirm that I can reproduce this with Sorbet v0.5.11034.


refactor: Removes the CSS files from the Calendar Heatmap plugin

SUMMARY
This PR removes the CSS files from the Calendar Heatmap plugin using Emotion. It also adjusts the styles to match theme configuration and existing components.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS
Check that the plugin is very similar to the previous version. We may have color, font, and opacity differences due to theme adjustments.
ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

/testenv up


Connecting API to another proxy has no effect on API key in nightly

API keys should be connected to proxies. However, connecting an API to another proxy does not update the API key or rather the user is not asked to Get API key. Tested in nightly.
Precondition: current user has an API key for original proxy.

Closing the issue, there was a step missing from the test. If all required steps are taken, works as expected.


Grey icon in tabs

As suggested by @SylvainCorlay

The docs build failure is unrelated, https://github.com/jupytercad/JupyterCAD/pull/528 should fix it


Update MNP Script

Here the link
https://raw.githubusercontent.com/MasterNodesPro/MNPCoin/master/doc/masternode.sh
there is still old walletlink inside
wget https://github.com/MasterNodesPro/MNPCoin/releases/download/v1.1.0/mnpcoin-linux-ubuntu1604-v1.1.0.zip -O mnpcoin-v1.1.0.zip
pls Change to new walletversion

This has been completed


Is it possible to create a Wavefront file from arrays?

Hi there,
I have a uv map, with their positions and indices, all these in a numpy array. Is it possible to create a .obj file from these? I want to create a square mesh with them.
Thanks

Probably simpler to use trimesh for this : https://github.com/mikedh/trimesh


Investigate one PMID assigned to duplicate and triplicate REDCap IDs (Email: 10.10.19)

Case 1: 1 PMID assigned to duplicate REDCap IDs
Case 2: 1 PMID assigned to triplicate REDCap IDs
Emailed 10.10.19

Related to #136


http: envoy rejects valid referer

Per HTTP semantics (https://httpwg.org/http-core/draft-ietf-httpbis-semantics-latest.html#field.referer), partial URI is a valid value for "referer" header. An example: Referer: /abc/com. Envoy currently always removes this header since it deems it invalid, without any way to disable the sanitization.
Envoy should not be erasing valid header values.
cc @RenjieTang

@alyssawilk @yanavlasov @mattklein123
I know this is an old issue, but I think we might be unnecessarily strict here, so I want to revisit this issue. I don't think Envoy needs to enforce RFC unless it's trying to use the header in some ways. Envoy as a proxy never parses the context (it exposes it by-value to authz and CEL, however). So why is it spending cycles (incorrectly) validating it?
Can we disable sanitization for referer and keep it private to envoy-mobile?

@RyanTheOptimist
I think we can probably remove this logic in Envoy-mobile as well?
Looking back at the original Cronet issue, referers were not passed in the URL requests.
The whole sanitization thing came as a byproduct of initializing and setting referer through GURL.
But Envoy-mobile is already propagating referer set by the application. Maybe we should leave the responsibility to the application and avoid the extra sanitization cost in Envoy?

I think that in general we care about being standards compliant so as to avoid issues where non-standards-compliant behavior causes problems. As @kyessenov points out, Envoy is incorrectly strict in this case, and so I think we should fix this (to allow both absolute and partial URIs in referrer). But I don't think we should leave the referrer completely unvalidated, as a matter of general principle. @yanavlasov What's your take on this in the context of UHV?
I think Envoy Mobile, like Envoy, should make sure that it sends standards-compliant traffic and so I'd be supportive of continuing to rely on Envoy's sanitization of this header.

Any time we validate any field we run the risk of breaking some application that is sending non-standards compliant traffic. That's true. But that's true for all fields. We regularly get papers where various servers or proxies handle different HTTP edge cases differently. Taking a proxy with non-compliant behavior X and putting it in from of a server with non-compliant behavior Y can led to surprising behavior. Is it possible that there is no server that would do anything wrong if Envoy didn't validate referrer? Maybe. But that doesn't seem like a strong enough argument to justify sending non-compliant traffic. I'm happy to hear what others think.

@RyanTheOptimist It depends on the situation. In Envoy as a sidecar, we generally can trust upstreams and downstreams to cooperate and our goal is to minimize the impact of the sidecar without a good reason. For front LBs, it makes a lot of sense to only emit sanitized headers.
It's genuinely surprising that only "referer" is sanitized. If we add a knob to control which RFCs to enforce, where would that knob live and how fine grained do we need it?

I don't think we only sanitize the referrer header. We do lots of different sanitization at different places in the code. Some of it is in quiche, some is in nghttp2, etc. The UHV project aims to unify this validation:
https://docs.google.com/document/d/1iRprAqZt3dek107LZWtBTnb2YRRhUBanBnHvHKMPdkI/edit#heading=h.xgjl2srtytjt
https://github.com/envoyproxy/envoy/issues/20261


clusterID should be cluster_id ?

No other json field inside the OSB spec has caps. It looks like the convention would be clusterID -> cluster_id.

It kind of depends on who owns the field - kube vs osb.  osb does prefer underscores but kube prefers w/o.  I went with clusterID because I viewed it as a kube thing (since its data about a kube cluster) but if the group wants it with an underscore I don't mind switching it.

My gut says requiring caps in a json payload will cause pain is why I bring it up and osb likes the _ which is why I suggest cluster_id vs clusterid but I don't really care which wins.

what if we go with clusterid ? no underscore.  While I can go either way, I do have a slight preference for kube data to follow kube's pattern.

clusterid  is fine with me like I said (I kinda dislike the _ in the current spec but don't tell anyone)

@pmorie @MHBauer you guys ok with clusterid ?

yes


Make a better looking plan page

@saikat If this is all the content you want, I'm sure @danielameyer can spice up the page with infographics. Please confirm this is all the content you want still.


logs menu seems to be broken with namespaces

Looks like the namespaces are due to multiple copies of the container: one from cyberreboot/ and the other from core/


fix: apply tsconfig.json only to matched files

fixes https://github.com/esbuild-kit/tsx/issues/132

:tada: This PR is included in version 3.12.3 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Atom gives a quick white blink when you do something

Prerequisites

[x ] Put an X between the brackets on this line if you have done all of the following:

Reproduced the problem in Safe Mode: http://flight-manual.atom.io/hacking-atom/sections/debugging/#using-safe-mode
Followed all applicable steps in the debugging guide: http://flight-manual.atom.io/hacking-atom/sections/debugging/
Checked the FAQs on the message board for common solutions: https://discuss.atom.io/c/faq
Checked that your issue isn't already filed: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aatom
Checked that there is not already an Atom package that provides the described functionality: https://atom.io/packages

Description
When you type and open the snippets, or when you save, or when you scroll quickly the page, Atom always gives a quick white blink.
Steps to Reproduce

Type or type a snippet or save or scroll quickly the page, while see with attention
You'll see the quick white blink

Expected behavior:
The quick white blink always repeats or appears when you do these things
Actual behavior:
After the quick white blink, nothings happened, it was all normal, but the quick white blink will appear or repeat.
Reproduces how often:
Millseconds
Versions
Atom         : 1.19.0
Electron    : 1.6.9
Chrome    : 56.0.2924.87
Node        : 7.4.0
apm  1.18.2
npm  3.10.10
node 6.9.5 x64
python 2.7.13
git 2.13.3
macOS Sierra X 10.12.6
Mac mini Mid 2011 Core i5 8GB RAM

I'm unable to reproduce (or my eyes just aren't catching it) - can you confirm if this behavior is new for you as of 1.19.0 and you didn't see it in 1.18.0?

I tested, nothing happened in Windows. I think it happens only on Mac.

If your eyes are not catching, because the white blink is too much quick or maybe you are not a Mac user.

Yep I tested on macOS 10.12.6 but maybe it's too quick for me?  My laptop is a newer Macbook Pro so maybe that's part of it too.
Do you think it's still so quick that a screencast or GIF wouldn't capture the behavior?

I updated to Atom 1.19.2 and I'm going to see if this behaviour still happens.

I have just tested, I caught a quick white blink in Atom 1.19.2. The problem is that it will take a lot to catch the quick white blin during the record. Therefore, I have to cut the video.

Another report in https://github.com/atom/atom/issues/15648, also mentioned that it wasn't an issue in 1.18.0 - I'll ask the team to take a look.

Great, I tried to record a white flash, but it took much time to catch the white flash red-flagged. So I gave up and I had to get used with the white flash.

here's the gif on this happening for convenience

Atom    : 1.20.0
Electron: 1.6.9
Chrome  : 56.0.2924.87
Node    : 7.4.0

apm  1.18.4
npm  3.10.10
node 6.9.5 x64
python 2.7.12
git 2.7.4

Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.3 LTS
Release:	16.04
Codename:	xenial

Observe that it is still present in Atom Beta 1.21 too.

I have the same issue on Xenial since a few versions ago.
I have the same issue with Chromium, but way less often. Maybe only when it's under heavy load, definitely when developing Ember apps.

@VulumeCode I tested to see if heavy loads were related but it doesn't seem to be the case, I just think it's an issue with the new rendering architecture introduced in 1.9

Just wanted to mention that I asked the team about this and we'd like to wait and see if an Electron upgrade tentatively planned for sometime in October fixes this issue.

Happening on both of my Ubuntu 16.04 machines as well.

Happening on Fedora 26 with latest Atom stable too.

Happening on Arch Gnome 3 and latest Atom stable.

definitely related to load. With ansible linters on a huge playbook, you'll quickly know if you're epileptic.

@flamein check if it happens on safe mode

opening a load of stuff files, it doesn't.
It might be related to a single package then, or just the load of adding several packages.

@flamein yours might be a different issue then, for me it happens on safe mode without any package or load.

hmm yes, looks more like: https://github.com/atom/atom/issues/15032
Although they might be related

+1 on Fedora 26 with Atom 1.22

Unfortunately Atom Beta 1.23.0-beta1 still has this problem, @rsese.

Upgraded today to 1.22.1 and this problem was introduced (Ubuntu 16.04)

Same issue on Atom 1.22.1 on both Ubuntu and Arch Linux, starts to flicker several times as soon as an action is launched (save, lint, format, upgrade package, etc...)

After three months, this problem is unsolved. How unhelpful!

To me it seem using F11, full screen, solves a bit but not all… But maybe I am just overly optimistic ;)

gave up on atom because of this, moved on to Visual code and I'm loving it. much faster and less buggy.

Sorry everyone - the Electron upgrade hasn't happened yet but we'll post back in here when it does happen so folks can confirm if that resolves this issue.

Flashing in 1.22.1 on MacOS...  Very annoying...

@ddon - working in Fullscreen works for Ubuntu...

@burabure, I hate Visual Studio, because it was too difficult to set PHP server, and phpcs expected me to install pear for installing PHP Codesniffer, and Pear destroyed my macOS and PHP, causing one of extensions to disappear – v8js. I tried to repair for more than 5 hours. I uninstalled and reinstalled Homebrew, and did a small clean, and the error continues the same. Then I had to a clean install of macOS.
Visual Studio Code and php are worse for PHP developers and programmers who use macOS. They give me headache.
I prefer Atom, because too easy and simple to set PHP, and I didn't have to install the damned Pear, I installed only Homebrew and composer via Homebrew.

@burabure, I didn't need to make a clean install of macOS, I had an idea and I found a solution… I removed v8js extension from php.inis and removed ext-v8.ini. I brew-uninstalled and reinstalled php with httpd and no errors finally. I adapted myself quickly to Visual Studio Code. I just will avoid pear, so Pear is the Perses, a god of destruction.

This issue is driving me nuts!

@marshallford does full screen work for you? F11 in Ubuntu?

@tvld unfortunately no.

It's weird. On my desktop, Ubuntu 16.04, i have the issue, but not in full screen mode.
On laptop with Ubuntu 17.04 all is fine all the time...

This issue persists on 1.23.1 on Ubuntu 16.04. I upgraded from 1.18, where this did not happen.

The Electron upgrade is available in master right now so will be available to try in the next beta I believe, 1.25.  If anyone who's hitting this issue on a Mac wants to give it a try now, you can download a macOS build with the Electron upgrade here:
https://6815-3228505-gh.circle-artifacts.com/0/Users/distiller/atom/out/atom-mac.zip

1.25.0-beta0 on Ubuntu 16.04 LTS, still flashing like crazy...

There sure is something not good about the combination of Atom (or Electron) on these Intel integrated graphics sets.
e.g. My benchmark for whether the white flickering is present is installing atom-beautify because this is a big package with lots of deps.
If you do this via apm at the command line it's relatively quick. Do it from the UI on one of these iGPUs on Linux and...

Atom now has 3 processes between them consuming between 200-300% of CPU time
Plus apm consuming 100% of a CPU
Install takes something like 5x longer

Objectively the only thing that a GUI install is doing that CLI apm install isn't is displaying the stripey throbber on the install button (and presumably also processing the STDOUT of apm or something so it knows when it finished).
My white flickers have gone though. Not sure if this is because I've moved that laptop to Fedora (but fairly sure it flickered) ; I'll switch to the Intel GPU on my Ubuntu laptop and give it a go.

I haven't had a chance to experiment with the new 1.24 and 1.25 builds, anyone have any good luck?

I had read somewhere that the issue could have been caused by the linter package, so I disabled the packages linter, linter-pylama, and linter-ui-default with Atom 1.23.3. This seem to have done the trick on my side, as I haven't noticed any more flickering since then.

@gilbertohasnofb, and with atom-beautify enabled?

@awilkins, the problem is that, I use a Mac and you use Linux. Do I have to compile the repository?

From what I've seen so far it's not just one package. More like when
anything does heavy work and needs to draw something on screen.
On 15 Feb 2018 12:06, "Gustavo Reis"<EMAIL_ADDRESS>wrote:

@awilkins https://github.com/awilkins, the problem is that, I use a Mac
and you use Linux. Do I have to compile the repository?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/atom/atom/issues/15323#issuecomment-365894999, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AVkC9fxeWVgLq2Ar8uFD1jv8nHzczAaNks5tVA-jgaJpZM4O4LQL
.

@gusbemacbe Yes, I do have atom-beautify enabled. And I am on Linux if that matters at all.

The 1.25 beta has the Electron upgrade - can anyone seeing this issue confirm if this is still a problem on 1.25?

I'm running on Ubuntu 17.10 and Gnome on Xorg as desktop environment, Atom don't gives white blinks.
But when I'm running on Gnome on Wayland, Atom gives white blinks.
Note : I have an Intel Kaby Lake.

@Thebarda and what Atom version are you using?

I spent a lot of time with Atom on Windows and on macOS. No white blinks/flickers. If the white blink appears, I'll report to you.

@50Wliu I'm using 1.24.0

@Thebarda please try Atom 1.25 beta as suggested above.

1.25.0-beta0 on Ubuntu 16.04 LTS, flashing.
1.25.0-beta0 on Ubuntu 17.10 (wayland), flashing.
Will give it a try on 17.10 (xorg).

@50Wliu Thanks. That solves my problem

Atom 1.24.0 on Arch Linux x86_64 (current) with Wayland: flashing
Atom 1.26.0.dev.34465 on Arch Linux x86_64 (current) with Wayland: not flashing
@50Wliu Thanks. That fixes the issue.

1.25.0-beta2 on Ubuntu 16.04 LTS still flashing.

@kmyokoyama, you need to upgrade from 1.25 to 1.26.

@gusbemacbe The v1.25.0-beta0 build includes the Electron update theorized to fix this, v1.26.0 development builds aren't necessary (but also would include that Electron update).
@kmyokoyama Your graphics driver in use, as well as the display server (X, Wayland, ?) would be helpful.

Has anyone tried this?
https://github.com/steelbrain/linter/issues/1372#issuecomment-287307487

1.25.0-beta3
Ubuntu 16.04

Xorg 1.18.4
i915_bpo GPU driver (Xorg log claims it's the i965 driver though)
DRI2 rendering mode

GPU 00:02.0 VGA compatible controller: Intel Corporation Skylake Integrated Graphics (rev 06)

Still block-blinks under load (my usual test of installing atom-beautifier came up naturally this morning because it had an update)

OR 01:00.0 3D controller: NVIDIA Corporation GM107GLM [Quadro M1000M] (rev ff)

Still steady

(this machine is great for testing this issue because it's the exact same hardware but I can flip GPUs)
I think it's much improved, it's nowhere near as pronounced now, but now there are also "blank blinks" where the text blanks out a moment in a similar pattern to the white blinks.
Going to try this from the Arch wiki : turning off vsync. The symptoms described - of Chrome having slow performance unless you use --disable-gpu - mesh very much with the other symptoms I experience with this, which is that performance of graphics updates in Atom are v.v.slow and impact other processes - e.g. installing atom-beautify can take 5-10x longer than it takes via apm install. While doing it via the GUI is slower than the command line on a non-Intel GPU box, it's more like 1.2-1.5x rather than the incredible slowness.
Think the overall takehome here is that the drivers for Intel integrated GPUs have some kind of "quirk" (bug) that Chrome/Blink particularly triggers and thus Atom is especially hard on as a hard-working multi-layout app.

Atom    : 1.25.0-beta3
Electron: 1.7.11
Chrome  : 58.0.3029.110
Node    : 7.9.0

Still flashing on Ubuntu 16. This is quite serious, I'll have to change to another editor until it's fixed I think :(

Update: @ilavidas 's suggestion to try steelbrain/linter#1372 (comment) worked for me! Hooray!

I'm finding the disable vsync workaround I linked above (and here) to be quite acceptable also - I'm still switched to my Intel GPU not the nvidia, and I've not thrown my laptop out of the window.
The comments in the linter bug link to a Phoronix post that seems to think that using the modesetting driver may cure this... sadly it's not available for Ubuntu 16.04 (which I'm sticking with at present because I'm one of those weirdos who likes the Unity desktop).

Does it still happen in DRI 3?
It fixes for me on the same kind of blink with css animations in electron

My notebook is Nvidia, but I have the drivers disabled. I'm keeping the one from Intel, and those flash's have happened quite often.

I uninstalled the NVidia driver and the error seems to have stopped.
I use Linux.

I've been using atom for about a month now and love it. I can easily duplicate the flash and ensuing crash. Search for something when that search term isn't present. The screen flashes, editor freezes and I get an alert asking me if I want to force exit, which you have to. I'm on centos 7.2. Here's the debug info when it cores. Looks like a GL issue and possible race condition:
/usr/bin/atom: line 129:  2958 Segmentation fault      (core dumped) nohup "$ATOM_PATH" --executed-from="$(pwd)" --pid=$$ "$@" > "$ATOM_HOME/nohup.out" 2>&1
[2980:0410/193817.707749:ERROR:gles2_cmd_decoder.cc(17045)] [.RenderWorker-0x2b110fdc2c00]GL ERROR :GL_INVALID_OPERATION : glBindTexImage2DCHROMIUM: no image found with the given ID
[2980:0410/193817.708047:ERROR:gles2_cmd_decoder.cc(16316)] [.RenderWorker-0x2b110fdc2c00]GL ERROR :GL_INVALID_VALUE : glCopySubTextureCHROMIUM: source texture has no level 0

As far as I can tell this is very much dependent on the load of the host running Atom. I get this reliably on OSX (latest) and lates version of Atom when the CPU is high.
It seems to be related also to the size of the file open (in terms of line) and possibly the linter in use (linter-php 1.5.1)

I think this flickering is either from Ubuntu or Electron am facing same issue  with vscode and my electron apps
Ubuntu: 17.10

Sad to say, it has happened again for me after update to Atom 1.26.1 on Fedora 27.
further info:

Display: wayland
CPU: Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz
VGA compatible controller: Intel Corporation HD Graphics 620 (rev 02)

I can't tell if these are my seizures or the editor anymore.

Okay, still there with Atom 1.27.0 on Fedora 28.
@PostholerCom: what is your setup?

Display server
CPU
Graphics
Atom --disable-gpu flag passed or not?

Electron was upgraded to 2.0.0 in Atom 1.28.0 beta.  Would anyone mind testing that version?

@50Wliu It worked everything.
@burabure can you test it for us, please? I tested it and it was everything normal. I am not sure about you.

I upgraded to 1.27.0 this morning. So far, so good. It hasn't crashed once and I've tried. I'll give it a day or two.

1.27.0 crashed on me this morning. Bummer.

@PostholerCom you have to update to 1.28-beta to test. He didn't ask you to test 1.27. He asked you to test 1.28.

I got 1.28.0 beta to crash almost immediately.
Atom    : 1.28.0-beta0
Electron: 2.0.0
Chrome  : 61.0.3163.100
Node    : 8.9.3
ps -ef:
user    8822     1  0 11:31 pts/1    00:00:00 /bin/bash /usr/bin/atom-beta
user    8824  8822  9 11:31 pts/1    00:00:15 /usr/share/atom-beta/atom --executed-from=/home/user --pid=8811
user    8826  8824  0 11:31 pts/1    00:00:00 /usr/share/atom-beta/atom --type=zygote --no-sandbox
user    8872  8824  6 11:31 pts/1    00:00:10 /usr/share/atom-beta/atom --type=gpu-process --no-sandbox --supports-dual-gpus=false --gpu-driver-bug-workarounds=1,9,27,70,84 --disable-gl-extensions=GL_KHR_blend_equation_advanced GL_KHR_blend_equation_advanced_coherent --disable-accelerated-video-decode --gpu-vendor-id=0x8086 --gpu-device-id=0x0f31 --gpu-driver-vendor --gpu-driver-version --gpu-driver-date --service-request-channel-token=E8B012139F21224CB8F6AC037396F645

@PostholerCom Strange, it worked here without crash and blink.
@50Wliu, issue this to Electron team.

Version 1.27.0 made this worse.

@PostholerCom This issue is not about crashes.  Please file a new issue.
@danielsalles as I mentioned above, it would be very helpful if you could also test Atom 1.28.0 beta.

@50Wliu 'Crash' is the wrong word and I shouldn't use it. As stated above, I search, the screen flickers white and the editor locks for about 15 seconds, then a popup asks if I want to quit. Saying no, leaves your with a locked up editor. Yes is the only response.

Atom    : 1.28.0-beta1
Electron: 2.0.1
Chrome  : 61.0.3163.100
Node    : 8.9.3

Ubuntu 17.10
Flashing really badly.
Several white flashes, menu tree and panes flickering.

Happens for me too!
Versions
Atom : 1.19.0
Electron : 1.6.9
Chrome : 56.0.2924.87
Node : 7.4.0
Arch Linux
Razer Blade Stealth 2016 Core i7 16GB RAM
I found a complaint about not being able to bind texture 0 in opengl and also get:
org.gnome.Shell.desktop[801]: Window manager warning: Buggy client sent a _NET_ACTIVE_WINDOW message with a timestamp of 0 for 0x1a00001 (Atom)
I get this stack trace if it helps:
`Jun 19 12:29:30 Rayleigh systemd-coredump[28304]: Process 28237 (electron) of user 1000 dumped core.

-- This usually indicates a programming error in the crashing program and
-- should be reported to its vendor as a bug.
`

It still seems to be present, although less then before.
now only happens under severe load, linting 4 ansible playbooks at the same time.

Happens for me:
Atom    : 1.26.1
Electron: 1.7.11
Chrome  : 58.0.3029.110
Node    : 7.9.0

Ubuntu 16.04 LTS, running on an i3-4130 with integrated graphics.
Running atom with the --disable-gpu flag seems to mitigate the issue, but normally the flicker happens when something changes in the UI, such as a popup or fetch/push etc.

@chaitanyathengdi
The version 1.26 is out of date. Check the https://atom.io/ and see that the current version is 1.29.

I still get this with atom 1.29.0
Although i have seen similar issues on Slack which might point to electron
after all. It's less noticeable on Slack for me as the background is white
anyway.
Met vriendelijke groet / Kind regards,
Finke Lamein
Devops Engineer
T:+31 6 52 17 95 63
<EMAIL_ADDRESS>*W:*www.catawiki.nl | www.catawiki.com
On Thu, 9 Aug 2018 at 11:50, Gustavo Reis<EMAIL_ADDRESS>wrote:

@chaitanyathengdi https://github.com/chaitanyathengdi
The version 1.26 is out of date. Check the https://atom.io/ and see that
the current version is 1.29.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/atom/atom/issues/15323#issuecomment-411702562, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AVkC9Vv7xysTGcdTmNbxM19OWbAj3U7hks5uPAX-gaJpZM4O4LQL
.

@gusbemacbe Still noticing on Atom 1.29.0.
Atom    : 1.29.0
Electron: 2.0.5
Chrome  : 61.0.3163.100
Node    : 8.9.3

Looks like it's finally behind me.
Atom    : 1.30.0-beta0
Electron: 2.0.5
Chrome  : 61.0.3163.100
Node    : 8.9.3

Ubuntu 18.4.1

I am having this issue as well- current version of Atom 1.26.1 on Ubutuntu 16.04 LTS
Linux gaia 4.4.0-131-generic #157-Ubuntu SMP Thu Jul 12 15:51:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
ares: "1.10.1-DEV"
atom-shell: "1.7.11"
chrome: "58.0.3029.110"
electron: "1.7.11"
http_parser: "2.7.0"
modules: "54"
node: "7.9.0"
openssl: "1.0.2k"
uv: "1.11.0"
v8: "5.8.283.38"
zlib: "1.2.11"

Some releases of Atom it goes away, then another release the issue comes back.
I get that this may be an Electron issue- however, it's hurting Atom.
I am (gasp) even considering moving to Visual Studio Code. I really like Atom, but after years of this issue not being fixed, it's get a bit much. I've been using Atom since version 0.5X.

Reverted the .drirc workaround from Arch wiki mentioned in my comment above while on 1.30.0-beta2 and, wow, the flickers are more horrible than ever. The workaround must be really good though...
Now going to try 1.31.0-beta0 because the changelog mentions some WebGL changes from Electron that got merged...
Nope, still horribly flickery.
<puts .drirc back>
Still running on

Ubuntu 16.04
Intel Skylake integrated graphics

@adavie1, ATom still uses Electron instead of Electron 2!
They should upgrade to Electron 2 and to Electron 3!
I ran Atom, using Electron 2 and 3.

I am an Android developer. I read in class 12
I am very poor please help me with some money I have a computer with 2 gb ram and Intel process I developes app on it but it's hard to run ...so please donate me some money. Bank name - UCO Bank , MY Bank account number -<PHONE_NUMBER>6969 , Swift code - UCBAINBB102 , Ifsc code - UCBA0000657 , Account holder name - Raj Singha ..I am from India . Please donate as you like.

Reasonably sure this is happening when you have enough elements at once.  I have two files open, one 1700 lines long, one 484 lines long, and that's 274KiB of HTML in the <body /> element.  This is about 10 minutes into me starting work on this, and I generally end up with 10-20 files open at once while chasing down an issue.
It seems to me that this just isn't what HTML is for, as convenient as it seems.

I believe Atom 1.30.0 finally solved this, so please check if you're on that version.

I can confirm it's not at all fixed on 1.13.0.
Just open up some large files, perhaps even have two of the same file open in different panes, and scroll around, and perhaps click in some places.  Currently half my treeview failed to repaint, and I have several large squares of solid background colour where there should be text and such.

@Asday, you have confounded 1.13 with 1.30. The version 1.13 is too too old and is no longer supported. Please switch to 1.30.

I was still seeing this issue on Atom 1.32-beta0. I was able to fix it by changing the DRI vsync settings.
Create a file called ~/.drirc with the contents:
<device screen="0" driver="dri2">
	<application name="Default">
		<option name="vblank_mode" value="0"/>
	</application>
</device>

According to this random reddit post this setting will disable vsync. If this causes issues with other apps you should be able to make this specific to atom by adding an executable=atom option to the application tag, but I haven't tested that. See here for more info on this config file).
Before this change I was seeing the flickering quite frequently (probably at least once a minute), but after creating this file (and rebooting, though suspect that's not necessary) I haven't seen any flickering.
I got the idea from the arch wiki.

Also cross-linking #14776 because it doesn't seem to be otherwise marked as a duplicate.

Here's the solution I've found.
If I go into Edit->Preferences and change 'Color Profile' to sRGB, then allow atom to reload at the prompt, everything works great. BUT, when I normally close atom and open it again I get the same flickering mess. If I go through the steps again, it is fine.
@ssfrr I tried your fix without success. But thanks for the tip!

Still happening on 1.31.2
Fortunately, the above hack still works.

More in-depth work has been done, and the flickering returned.  I didn't need to set sRGB each time, but if the flickering starts, switching the profile between sRGB and default and letting Atom restart itself seems to assuage the issue.

Tried out the sRGB fix and it doesn't seem to work for me.  Atom is now almost unusable with all plugins disabled the flickering occurs constantly (almost every time I open the command pallet for example). :-1:
Atom    : 1.31.2
Electron: 2.0.7
Chrome  : 61.0.3163.100
Node    : 8.9.3
OS      : Fedora 28
Desktop : GNOME
CPU     : i7-6500U
Graphics: Intel HD 520
Ram     : 12 GiB

Also going to add this screenshot of Atom Startup with only a theme and systax theme enabled.

@ryanvade have you tried @ssfrr's workaround? Seems like this may be caused by a fundamental issue in chromium with vsync on intel graphics. Here is a link to the original issue. It fixed my issues with other electron apps but not done too much testing on atom yet

@ssfrr's solution worked perfectly for me! Ubuntu 14.04.5, Atom 1.31.2.

@rmasp98 I tried the workaround suggested by @ssfrr , the flashing has been reduced slightly but is not eliminated. But it is at least usable now (the white screen on startup hasn't changed but that disappears quickly).

Any news on this, or is it systemic?

Atom: 1.34.0
Electron: 2.0.16
Chrome: 61.0.3163.100
Node: 8.9.3

Fedora 29
Quite often.

For me, this issue is resolved.
Ubuntu 18.04
Atom 1.34.0

@gusbemacbe I think there are many people still experiencing this issue, myself included, and the fixes and workarounds proposed here did not work on my side. I think this issue should be left opened, particularly given the number of affected people.

I were just about to comment what the fix was since it got closed. I still experience the issue and the fix only works sometimes for me. (not most times, so quite annoying)

@gusbemacbe Come on dude, one guy says he doesn't have the problem any more on a 124 comment issue, and you close it?
Still happening for me.  Please reopen the issue.

I have tried @ssfrr's suggestion once again, and this time it seems to have fixed the issue for me. That said, this solution is far from ideal as 1) it did not seem to work with my previous GPU and 2) Atom should be able to handle vsync IMO, like every other application.

I would like to reiterate, this is still happening.  Mint 18, Atom 1.35.1 x64.
Please reopen the issue.

Thanks everyone for the feedback.
As this is an underlying Electron issue and isn't something that is within our control, the Atom team isn't going to be working on this issue other than working to get Atom updated to a more current version of Electron. Since updating to a new version of Electron is being tracked elsewhere, you may want to subscribe there for updates. Since this issue isn't something we're going to be working on individually, we're going to leave it closed.
Thanks again for all the help and information.

Has this already been fixed in a later version of Electron, or do we need to open an issue with them?
Or is there already an issue open with them?

I was on Intel Gpu, switched to Nvidia Gpu on my dell xps 15, no flicker now

Before I used wayland on Linux, after returning to Xorg this problem solved. I think it's one of the causes of flicker.
As solved for me, I'm mutating the thread.
Good luck guys.

I'm on AMD Radeon and XOrg, so it looks like neither of them are a factor.

Is there a way to force the "Restart atom" popup to appear?  Or some command to restart atom?
My startup procedure is:

Launch atom;
Switch to the other color profile;
Restart;
Work.

It doesn't matter which profile I'm on, the important thing appears to be either the switching or the restarting, and I'd like to narrow it down.

AFAIR ctrl-shift-F5 restarts Atom

Thanks @awilkins, that's exactly what I was looking for.  Doesn't solve the flickering issue, meaining it is something to do with the colour profile switch.

It seems to be fixed in 1.39.1 x64.  I'll update if that's not correct.


REMOVE QUESTION: Questionnaire - Will the results of the project be published:

REMOVE QUESTION: Questionnaire - Will the results of the project be published:

Done #260


[Addition]: PantherX selfwitnessing cluster (9HS)

Hotspot b58 Addresses
1127y8QQsngTTeiMs8bizrt1UhKsu7CBnm3AcjwSwHQa8BtUizjv
112dUHz5i6RrLE7WAKEriApq42z6MT6P8rvumzzQ5qHt8u3TSng5
112M6oJkPCxXJd7PBiYc2Fp7z2uzrAQScMD8hsiVRXvMbTvQsjkN
112WcmHRgWh98CS5uSgy9FBWZrkqyzVChdGLq6G9mxcSymq8bfCH
11Fs3t76toJvsVpaX9KUdhANNDmSeffLbHBjUsHMdnZHKg48hbA
11WNYS93KVhEDVtS6q4Dyk2Wsg9VXuN3ez4aaucYyPwWxGTrCDc
11xLdsUoN2UH1wAMRgu9XzAbA6YFgwyJXjUSLP5uHS6oxmXxJh5
11ZpWemZVzCoiGm496LsntrroVy5VrLBgcWESV8HgEtdMK7S19s
11ZS94FC1mbo6CjtX72FLXt14FKHe1BjkoR5vLWhFK5ncEXjy49
Discord Handle
No response
Reason(s)
These Hotspots are only witnessing their own brand, have very often a unlikely (perfect) placement, same assertion date-range and usually there are enough real hotspots arround but these do not get witnessed at all.
Below some proof of the clusters:

High clustering Co and clique size. Asserted in 2 blocks. Same owner 1.0 and same maker 1.0

see #6200


Multidelegation

Allow stake keys with derivation paths ending in 2/index for any index, not just 0 as previously.
Show change outputs that send funds to addresses using keys with index > 0 (since not all SW wallets support them).
IMHO https://github.com/LedgerHQ/app-cardano/pull/54 should be processed before this, then the base branch of this PR should be changed to master.

@janmazak I think I'll close #54.
I'll merge this one into develop and cherry pick it to a new branch release_5.0.2 (base branch being release_5.0.1) with #53, if that's ok with you.

@sgliner-ledger There'll be a small conflict with develop that has to be resolved: one of the many commits on develop did some refactor of the function bip44_isOrdinaryStakingKeyPath. I'm not sure how to proceed with this, but just switching the base branch is not enough (on top of develop, it would be something like this: https://github.com/vacuumlabs/ledger-app-cardano-shelley/pull/199/files#diff-aa6ff0a28b607b65946266d9692b37e0019de370a298c709e154ac50d6df3fcdR222 )

@janmazak Does this need to be integrated in develop ? If not let's just merge this into release_5.0.2, unless I am missing something ?

Yes, the changes are also supposed to appear in future releases (i.e. should appear in develop too).

Ok then can you change the target of this PR to release-5.0.2 and create another PR targetting develop with the conflict fixed ? Thanks !

Ok then can you change the target of this PR to release-5.0.2 and create another PR targetting develop with the conflict fixed ? Thanks !

All done. See https://github.com/LedgerHQ/app-cardano/pull/56

Linked to https://ledgerhq.atlassian.net/browse/B2CA-1370


Request for release

Hello,
Its been a while since #21 has been merged in; along with four more PR's merged in this year adding features.
Would be great to have a new release tagged out with these changes if possible?
Are there any blocker issues preventing a new release I can look at helping with?
Sorry to create more noise 😅

Hey @pinkforest any interest in doing this? There are a couple PRs (#34, #37) to get merged as well. I've reviewed both and they are ready.

Thanks @Ralim and Thanks @Serial-ATA - I was going to add the publish to the org so you can release as well but seemingly forgot.
I'm behind on things but will get to it after next week's weekend - I am stuffed atm

@Serial-ATA could you do the honors ? :pray: e.g. send a PR with changelog + version bump

Sure, I'll get it done shortly.

Hi everyone, happy new year! Any chance we can get this pushed through in the next couple of weeks? 🙏

There was a release so closing this


HPX counters crashing the application

Using HPX counters in a std::thread makes the program crash.
I create an additional thread to sample some HPX counters with a certain frequency using:
sampler_thread = std::thread(&my_sampling_funct, this);
Then if I create/use a counter in my_sampling_funct, the program crashes.

HPX Version: latest
Platform (compiler, OS): gcc 7.2.0, CentOS 7.4.1708

Could you provide us with a small reproducing test case for this. The way you describe it does not allow for any diagnostics or even fixing. Thanks.

Could you give any kind of test case. I am not understand even where it will crash so don't know how to fix it. Thanks

@RANJEET16520 I didn't ask you to fix it. I asked you to give us some code that allows us to reproduce the problem you're seeing.

Hi all, I'm sorry that I wasn't clear enough when I opened the issue. Please find below a simplified example where this error occurs.
#include <unistd.h>

#include <hpx/hpx_init.hpp>
#include <hpx/include/performance_counters.hpp>
#include <iostream>
#include <thread>

hpx::id_type idle_rate_counter_;

void my_sampling_funct(void)
{

    // Create one counter per locality
    static const char * idle_counter_name = "/threads{locality#%d/total}/idle-rate";

    const std::uint32_t prefix = hpx::get_locality_id();

    std::cerr << "Registering counter " << boost::str(boost::format(idle_counter_name) % prefix) << std::endl;
    idle_rate_counter_ = hpx::performance_counters::get_counter(
            boost::str(boost::format(idle_counter_name) % prefix));

    // Start the counter
    hpx::performance_counters::stubs::performance_counter::start(hpx::launch::sync, idle_rate_counter_);

    // Get the value
    hpx::performance_counters::counter_value idle_value = hpx::performance_counters::stubs::performance_counter::get_value(
                hpx::launch::sync, idle_rate_counter_);

    double idle_rate_ = idle_value.get_value<double>() * 0.01;

    std::cout << "Idle rate: " << idle_rate_ << std::endl;

    // Stop the counter
    hpx::performance_counters::stubs::performance_counter::stop(hpx::launch::sync, idle_rate_counter_);

}

int hpx_main(int argc, char **argv)
{

    // Create one additional OS-thread
    std::thread sampler_thread = std::thread(&my_sampling_funct);

    sampler_thread.join();

    hpx::finalize();

    //std::terminate();
    return 0;
}

int main(int argc, char **argv)
{
    return hpx::init();
}

Please don't hesitate to ask me again if you need further assistance. Thanks!

Thanks sir
On 10-Mar-2018 6:38 PM, "xaguilar"<EMAIL_ADDRESS>wrote:

Hi all, I'm sorry that I wasn't clear enough when I opened the issue.
Please find below a simplified example where this error occurs.
#include <unistd.h>
#include <hpx/hpx_init.hpp>
#include <hpx/include/performance_counters.hpp>
#include
#include
hpx::id_type idle_rate_counter_;
void my_sampling_funct(void)
{
// Create one counter per locality
static const char * idle_counter_name = "/threads{locality#%d/total}/idle-rate";

const std::uint32_t prefix = hpx::get_locality_id();

std::cerr << "Registering counter " << boost::str(boost::format(idle_counter_name) % prefix) << std::endl;
idle_rate_counter_ = hpx::performance_counters::get_counter(
        boost::str(boost::format(idle_counter_name) % prefix));

// Start the counter
hpx::performance_counters::stubs::performance_counter::start(hpx::launch::sync, idle_rate_counter_);

// Get the value
hpx::performance_counters::counter_value idle_value = hpx::performance_counters::stubs::performance_counter::get_value(
            hpx::launch::sync, idle_rate_counter_);

double idle_rate_ = idle_value.get_value<double>() * 0.01;

std::cout << "Idle rate: " << idle_rate_ << std::endl;

// Stop the counter
hpx::performance_counters::stubs::performance_counter::stop(hpx::launch::sync, idle_rate_counter_);

}
int hpx_main(int argc, char **argv)
{
// Create one additional OS-thread
std::thread sampler_thread = std::thread(&my_sampling_funct);

sampler_thread.join();

hpx::finalize();

//std::terminate();
return 0;

}
int main(int argc, char **argv)
{
return hpx::init();
}
Please don't hesitate to ask me again if you need further assistance.
Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/STEllAR-GROUP/hpx/issues/3224#issuecomment-372028553,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Aep1itv3Gx3vhWEe6kkwWzsHiseLo4P9ks5tc9BXgaJpZM4SkXqq
.

@RANJEET16520 Thanks for the code snippet. I'm not sure why you decided to create a new (kernel-) thread for your test function. I would advise not to do that. You should create an HPX thread instead. Either simply switch std::thread for hpx::thread or use hpx::async to launch your function on a new HPX thread. I personally prefer the latter as directly using ...::thread does not (conceptually) scale well.
Generally, it is not possible to directly invoke (most of) HPX's functionality from arbitrary kernel-threads. For this to be possible those threads have to be registered with the runtime. I however admit that the generated error message could be more descriptive. I'll make a note to work on this.

Hi, thanks a lot for your explanation. I wasn't aware that blending kernel threads with HPX wasn't a good idea. I needed to sample the counters every certain period and as I already had a thread doing some work at a certain frequency, I just added the code in there. In any case, knowing now that doing this is not good, I'll move all that code to a function used together with hpx::util::interval_time (I wasn't aware either of such HPX utility actually) and I'll remove the additional thread creation. Thanks again!

Hello Sir,
This program shows an error related to it's header file.
May be I am not able to compile this program,.
I am trying to compile this program from many days but it continuously
showing errors.
Thanks.
On 10-Mar-2018 6:38 PM, "xaguilar"<EMAIL_ADDRESS>wrote:

Hi all, I'm sorry that I wasn't clear enough when I opened the issue.
Please find below a simplified example where this error occurs.
#include <unistd.h>
#include <hpx/hpx_init.hpp>
#include <hpx/include/performance_counters.hpp>
#include
#include
hpx::id_type idle_rate_counter_;
void my_sampling_funct(void)
{
// Create one counter per locality
static const char * idle_counter_name = "/threads{locality#%d/total}/idle-rate";

const std::uint32_t prefix = hpx::get_locality_id();

std::cerr << "Registering counter " << boost::str(boost::format(idle_counter_name) % prefix) << std::endl;
idle_rate_counter_ = hpx::performance_counters::get_counter(
        boost::str(boost::format(idle_counter_name) % prefix));

// Start the counter
hpx::performance_counters::stubs::performance_counter::start(hpx::launch::sync, idle_rate_counter_);

// Get the value
hpx::performance_counters::counter_value idle_value = hpx::performance_counters::stubs::performance_counter::get_value(
            hpx::launch::sync, idle_rate_counter_);

double idle_rate_ = idle_value.get_value<double>() * 0.01;

std::cout << "Idle rate: " << idle_rate_ << std::endl;

// Stop the counter
hpx::performance_counters::stubs::performance_counter::stop(hpx::launch::sync, idle_rate_counter_);

}
int hpx_main(int argc, char **argv)
{
// Create one additional OS-thread
std::thread sampler_thread = std::thread(&my_sampling_funct);

sampler_thread.join();

hpx::finalize();

//std::terminate();
return 0;

}
int main(int argc, char **argv)
{
return hpx::init();
}
Please don't hesitate to ask me again if you need further assistance.
Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/STEllAR-GROUP/hpx/issues/3224#issuecomment-372028553,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Aep1itv3Gx3vhWEe6kkwWzsHiseLo4P9ks5tc9BXgaJpZM4SkXqq
.

Which header file? What errors do you have? I could compile this small example in my environment without any problems.

@RANJEET16520 you'll have to be a bit more specific about what errors you're seeing. Also, in order to help we'd need to see the code that causes errors for you (or better, a minimal self-contained example that demonstrates the issue you're seeing).

Sir this error is shown by header files.
I am compiling this code from a week and it still not able to give any
output, except this error.

@RANJEET16520 You're saying "...except this error". What error specifically? Can you paste it somewhere?