Add ParentJoin KNN support

A join within Lucene is built by adding child-docs and parent-docs in order. Since our vector field already supports sparse indexing, it should be able to support parent join indexing.
However, when searching for the closest k, it is still the k nearest children vectors with no way to join back to the parent.
This commit adds this ability through some significant changes:

New leaf reader function that allows a collector for knn results
The knn results can then utilize bit-sets to join back to the parent id

This type of support is critical for nearest passage retrieval over larger documents. Generally, you want the top-k documents and knowledge of the nearest passages over each top-k document. Lucene's join functionality is a nice fit for this.
This does not replace the need for multi-valued vectors, which is important for other ranking methods (e.g. colbert token embeddings). But, it could be used in the case when metadata about the passage embedding must be stored (e.g. the related passage).

From a quick look, this lower level KNN collection API looks interesting. It has currently a high surface - presumably because extending the queue was easier to have a working prototype, which is cool - I'm curious how much leaner it can be made. It feels like we'd need at least collect(int docID, float similarity), float minSimilarity() and TopDocs topDocs(), would it be enough or is there more?

@alessandrobenedetti I took some of your ideas on deduplicating vector IDs based on some other id for this PR. If this work continues, I think some of it can transfer to the native multi-vector support in Lucene.

would it be enough or is there more?

I will dig a bit more on making this cleaner.
My biggest performance concerns are around keeping track of the heap-index -> ID and shuffling those around so often and resolving the docId by vector ordinal on every push.

@jpountz I took another shot at the KnnResults interface. I restricted the abstract and @Override methods to narrow the API. Additionally, I disconnected it from the queue, but it still has a queue object internally that sub-classes can utilize.

@jpountz my original benchmarks were flawed. There was a bug in my testing. Nested is actually 80% slower (or 1.8x times) than the current search times.
I am investigating the current possible causes.

@jpountz
Here are two JFRs. Both are force-merged to a single segment. One is nested (which each parent only have a single nested doc) the other is not-nested (so densely packed vectors and using the regular query).
My tests show the current implementation (keeping track of hashes while building heap) is between 30-50% slower when doing nested queries.
jfrs.zip

Here are two more JFRs, one with nested count of 128, the other with 512.
From what I can tell, the majority of the time is spent correcting the vector ordinal & getting the parent doc id. The caching stack overhead seems much smaller than that overhead.
many-nested.zip

One interesting side-effect of this change is that it gives us an entry point for supporting timeouts, I believe that we could do this via KnnCollector#earlyTerminated?

🤔 yeah, it totally could if the collector has some signal indicating timeout, we can exit :)

Hi @benwtrent I'm definitely curious to have a look at what you did here, but before diving in to the details (it's big!) I'd like to have a clearer sense of the problem you're solving. Is there a GH issue I'm missing that has more of a writeup? What confuses me is, I would have expected something like ToParentBlockJoinQuery(parentBitSet, KnnVectorQuery()) to more or less work already. Clearly I'm missing something, but maybe you could explain it clearly for the slower ones among us :)

Thanks for digging in @msokolov!

I'd like to have a clearer sense of the problem you're solving.

This PR solves a similar, but different problem to: https://github.com/apache/lucene/issues/12313
Text embedding models have a token limit, so when processing larger text inputs, they need to be divided into passages. These passages share a common parent document. When users search for the "top-k" documents, they expect the initial parent document as the result, not just individual passages.
A "multi-value" vector only partially solves this. The user will need to know the nearest passages across those documents to use in retrieval augmented generation. Multi-value fields cannot solve this as metadata needs to be associated with each vector to tie them to an originating passage, or better yet, some field containing the passage text itself. join seemed like a natural place to tackle this.

We get the top-k parent documents (e.g. the users larger chunk of text)
And can still get the nearest passages from that deduplicated set of parent documents

Not to mention the nice flexibility we get (filtering on passage metadata, filtering on parent documents, hybrid scoring on the parent or child level, etc.)

What confuses me is, I would have expected something like ToParentBlockJoinQuery(parentBitSet, KnnVectorQuery()) to more or less work already.

The main issue is that it won't return the correct number of parent documents when the user requests the top-k parents based on their children vectors. If there are multiple children per parent, this approach may return fewer than k parent documents.

The main issue is that it won't return the correct number of parent documents when the user requests the top-k parents based on their children vectors. If there are multiple children per parent, this approach may return fewer than k parent documents.

Thanks I see now. So this is kind of similar in spirit to the existing problem where we want the top K documents (by vector distance) satisfying some constraints and we have to choose whether to find some higher number of nearest docs (K') in the hopes that at least K of them will satisfy the constraints (post-filtering), or whether to apply the filters while searching, guaranteeing top K. I just want to note that both approaches have merit; it's a tradeoff depending on how restrictive the filters are, but for not-very-restrictive filters, post-filtering can outperform. In this case I guess there is a similar tradeoff relating to how many child documents there typically are. If it's a small number (say c children per parent), it may be better to use KNN search with K' = c * K.  It would be interesting to compare these two approaches to see if we can provide some guidance or even some kind of api that chooses?

I agree that there is similarity in that in both cases it boils down to whether or not you can accept having less than k hits. However the degradation is brutal with filtering as you either need to evaluate the filter across the entire segment to load it into a bitset (not great for both runtime (if the filter cardinality is high) and memory usage) or linearly scan all filter matches (not great either). Here the degradation is much more graceful as you only pay some overhead for vectors that get collected. For filtering, I could see a case for requesting k'>k vectors and then do post filtering. For this case I think I would always want to use this feature, potentially combined with the visitLimit option to protect against worst-case conditions like a million child docs per parent that would make collisions frequent.

@jpountz I think I see what you mean re: the cost of evaluating filters. We have been working on a solution where the most common and restrictive filters are cached and can be applied as pre-filters without needing to recompute for every query, while also joining with other constraints (post-filtering). This has led to increased yield (more high-scoring matches) at acceptable cost increase in many cases but in some cases we see increased costs without much better yield (as compared to applying all the filters as post-filters). So I feel like if the parent doc filter is highly restrictive there could be a similar tradeoff in this case ie even when the cost of the parent doc filter computation is not a consideration you might still see better recall/latency tradeoff using overcollection/post-filtering. I don't think this has any implications for this change really, but I do think it would be interesting to see some evaluation comparing.

If it's a small number (say c children per parent), it may be better to use KNN search with K' = c * K. It would be interesting to compare these two approaches to see if we can provide some guidance or even some kind of api that chooses?

@msokolov I have had similar thoughts. I think this type of optimization could be done even if there aren't any child filters. What about the case where every parent has a single child? It's definitely better to gather the vectors from the sparse index and then join back (.e.g post filter) on the parent docs.
The calculation would be something like K' = max(c) * K and then join after gathering the results. This could be revisited after this initial work is completed.

@msokolov let me know if there are further changes required.

Thanks @benwtrent for this work! I finally had the chance to take a look.
It's a lot and I see it's already merged, so I don't have any meaningful comment at the moment, but if I have time I'll dive into it in the future! (mostly when and if I resume the work on multi-valued, for which I am still waiting for fundings).
The work here drastically changes the way also my Pull Request should look like right now.
As a side note, do you happen to have any performance benchmark? I am quite curious as I always label nested docs approaches in Lucene to be 'slow', but having some facts (that potentially contradicts my statement) would be super cool!

The work here drastically changes the way also my Pull Request should look like right now.

Yes, I am sorry about that. But the good news is that the integration for multi-value vectors has some nicer APIs to take advantage of (e.g. KnnCollector) and it could possibly copy/paste the deduplicating nearest neighbor min-heap implementation.

As a side note, do you happen to have any performance benchmark?

The following test was completed over 139004 documents with 768 float32 dimensions.
The statistics for the nested value distributions:
1944 total unique documents
62.0 median number of nested values
71.50411522633745 mean number of nested values
309 max number of nested values
1 min number of nested values
2156.9469722481676 variance of nested values
|                                        50th percentile latency |          knn-search-10-100 |   3.10031     |     ms |
|                                        90th percentile latency |          knn-search-10-100 |   3.5629      |     ms |
|                                        99th percentile latency |          knn-search-10-100 |   4.60912     |     ms |
|                                      99.9th percentile latency |          knn-search-10-100 |  14.322       |     ms |
|                                       100th percentile latency |          knn-search-10-100 |  72.6463      |     ms |
|                                        50th percentile latency |   knn-nested-search-10-100 |   6.2615      |     ms |
|                                        90th percentile latency |   knn-nested-search-10-100 |   6.95849     |     ms |
|                                        99th percentile latency |   knn-nested-search-10-100 |   7.8881      |     ms |
|                                      99.9th percentile latency |   knn-nested-search-10-100 |  12.0871      |     ms |
|                                       100th percentile latency |   knn-nested-search-10-100 |  57.9238      |     ms |
|                                        50th percentile latency |        knn-search-100-1000 |   7.30288     |     ms |
|                                        90th percentile latency |        knn-search-100-1000 |   8.18694     |     ms |
|                                        99th percentile latency |        knn-search-100-1000 |   9.23673     |     ms |
|                                      99.9th percentile latency |        knn-search-100-1000 |  18.7072      |     ms |
|                                       100th percentile latency |        knn-search-100-1000 |  23.8712      |     ms |
|                                        50th percentile latency | knn-search-nested-100-1000 |  26.6446      |     ms |
|                                        90th percentile latency | knn-search-nested-100-1000 |  38.2561      |     ms |
|                                        99th percentile latency | knn-search-nested-100-1000 |  44.3627      |     ms |
|                                      99.9th percentile latency | knn-search-nested-100-1000 |  51.1843      |     ms |
|                                       100th percentile latency | knn-search-nested-100-1000 |  52.0864      |     ms |

GASP! Nested seems 2x to 4x slower!
But, keep in mind, we are eagerly joining! When I dug into the difference, I discovered that eagerly joining on this dataset meant we were visiting 3x to 5x more vectors. Consequently doing 3-5x more vector comparisons and deeper exploration of the graph. This lines up really nicely with the performance difference.
Since HNSW is log(n) the overall constant overhead of nested seems rather minor compared to the need to gather nearest vectors.
I am not sure these numbers are reflective of other nested/block-joining operations (like a term search).

The work here drastically changes the way also my Pull Request should look like right now.

Yes, I am sorry about that. But the good news is that the integration for multi-value vectors has some nicer APIs to take advantage of (e.g. KnnCollector) and it could possibly copy/paste the deduplicating nearest neighbor min-heap implementation.

No worries at all! My work is still paused, looking for sponsors, so no harm! When I resume it as you said I may find benefits (and do improvements) to the new data structures added (I admint I got lost in the amount of KnnCollectors and similar classes added, but I'm super curious to explore each of them thoroughfully.

As a side note, do you happen to have any performance benchmark?

The following test was completed over 139004 documents with 768 float32 dimensions.
The statistics for the nested value distributions:
1944 total unique documents 62.0 median number of nested values 71.50411522633745 mean number of nested values 309 max number of nested values 1 min number of nested values 2156.9469722481676 variance of nested values
|                                        50th percentile latency |          knn-search-10-100 |   3.10031     |     ms |
|                                        90th percentile latency |          knn-search-10-100 |   3.5629      |     ms |
|                                        99th percentile latency |          knn-search-10-100 |   4.60912     |     ms |
|                                      99.9th percentile latency |          knn-search-10-100 |  14.322       |     ms |
|                                       100th percentile latency |          knn-search-10-100 |  72.6463      |     ms |
|                                        50th percentile latency |   knn-nested-search-10-100 |   6.2615      |     ms |
|                                        90th percentile latency |   knn-nested-search-10-100 |   6.95849     |     ms |
|                                        99th percentile latency |   knn-nested-search-10-100 |   7.8881      |     ms |
|                                      99.9th percentile latency |   knn-nested-search-10-100 |  12.0871      |     ms |
|                                       100th percentile latency |   knn-nested-search-10-100 |  57.9238      |     ms |
|                                        50th percentile latency |        knn-search-100-1000 |   7.30288     |     ms |
|                                        90th percentile latency |        knn-search-100-1000 |   8.18694     |     ms |
|                                        99th percentile latency |        knn-search-100-1000 |   9.23673     |     ms |
|                                      99.9th percentile latency |        knn-search-100-1000 |  18.7072      |     ms |
|                                       100th percentile latency |        knn-search-100-1000 |  23.8712      |     ms |
|                                        50th percentile latency | knn-search-nested-100-1000 |  26.6446      |     ms |
|                                        90th percentile latency | knn-search-nested-100-1000 |  38.2561      |     ms |
|                                        99th percentile latency | knn-search-nested-100-1000 |  44.3627      |     ms |
|                                      99.9th percentile latency | knn-search-nested-100-1000 |  51.1843      |     ms |
|                                       100th percentile latency | knn-search-nested-100-1000 |  52.0864      |     ms |

GASP! Nested seems 2x to 4x slower!
But, keep in mind, we are eagerly joining! When I dug into the difference, I discovered that eagerly joining on this dataset meant we were visiting 3x to 5x more vectors. Consequently doing 3-5x more vector comparisons and deeper exploration of the graph. This lines up really nicely with the performance difference.
Since HNSW is log(n) the overall constant overhead of nested seems rather minor compared to the need to gather nearest vectors.
I am not sure these numbers are reflective of other nested/block-joining operations (like a term search).
Interesting and thanks for the heads up, I hope to investigate this further as well in the future!

@david-sitsky sorry for the confusion, it was renamed DiversifyingChildren*KnnVectorQuery

Hello,
I hope this message finds you well. I am reaching out regarding the recent enhancement described in https://lucene.apache.org/core/9_8_0/changes/Changes.html#v9.8.0.new_features GitHub issue #12434, which introduces the KnnCollector to LeafReader and KnnVectorReader for custom collection of vector search results.
I am currently using a KNN query in the following format:
{!knn f=vector topK=5}[-0.08779877, ... , 0.009549321]
I would like to leverage the new KnnCollector to obtain scores from child documents. Specifically, I would appreciate an example illustrating how to:
Apply a filter query (fq) on the parent document.
Retrieve both the parent document and child document with the child's score.
Your assistance with this matter would be immensely helpful. Thank you in advance for your time and support.
Best regards,
Gaurav Jayswal

Hello @benwtrent @alessandrobenedetti , I'm seeking assistance in comprehending how to calculate the total score for a parent document, sort it in descending order by the score, and additionally, only compute matched child documents with it's score. Your guidance is appreciated. Here's the Solr query I've written.
{!parent which=vector_status_s:P}{!knn f=vector topK=5}[-0.11524472,-0.015096951,-0.02416302,-0.042546727,-0.026223207,-0.024288718,-0.06648887,-0.019747224,-0.15247388,0.017357947,-0.07997921,-0.0044639846,0.026787976,0.04674327,-0.025140975,0.022736596,0.038513225,-0.007743583,0.008468372,-0.010870436,-0.049723275,0.052544124,-0.01040933,-0.021913795,0.055262197,0.03814301,-0.030052986,0.011873034,0.014196521,-0.045679808,-0.030343458,0.0027234245,0.067535505,-0.0065462803,0.014386235,0.0815663,0.034734566,0.076742254,-0.047816984,-0.06893713,-0.00575254,-0.0065407623,0.032530636,-0.047053427,-0.045342736,0.05458816,-0.0036792993,-0.119853236,0.0621119,0.05868241,-0.026309596,-0.07372244,0.098664746,-0.10785703,0.058371536,0.11598263,0.0035437504,0.034718044,-0.008448179,-0.015374862,0.11060915,-0.034994677,-0.07641869,-0.018799445,0.06742509,0.032640636,-0.0805842,0.056232687,-0.055578843,-0.09173681,0.055953458,-0.093306944,-0.005010136,-0.027816057,0.069614165,0.017868416,0.064177155,0.033978168,0.0015766103,-0.024414165,-0.03577587,0.0456416,-0.02556829,-0.019146064,-0.061762888,0.0881951,0.021560378,-0.021047728,0.012747205,0.022434331,0.018475346,-0.046412993,-0.04435442,0.04168245,0.058627237,0.0135493,-0.06728198,0.07312625,-0.05581615,0.04042105,-0.011703705,0.08142669,-0.02410761,-0.091190875,-0.028388929,-0.032626763,-0.05041609,0.06420719,0.046886727,0.06463699,0.035878975,-0.001365671,-0.069236726,-0.06430995,0.03162415,-0.07464388,-0.03160973,0.04416939,-0.024402045,-0.014191367,0.07411146,0.104716584,-0.048191704,-0.005468206,0.03430066,2.9004048E-4,0.026272327,1.70109E-33,0.029819172,0.06773484,-0.00742313,0.071461275,0.07202007,0.020473966,0.038026482,0.08200619,0.02915628,-0.012804087,0.059027206,0.089257926,-0.06499567,-0.0025720217,-0.046283025,-0.07170174,-0.046748683,-0.112400465,-0.035749055,-0.0014416518,0.020358646,0.046174794,-0.052782748,0.058808744,0.054796845,-0.060931858,0.108469814,0.033284586,-0.03280937,0.024645116,-0.0805344,0.034969736,-0.032420795,0.004894371,-0.026090093,0.0041577136,-0.04494302,-0.037979875,-0.03771678,-0.02735791,-0.09234106,0.002481496,-0.07598019,-3.466485E-4,-0.06326391,-0.00995851,0.035185788,0.07036043,0.10276299,-0.059668124,-0.0083749425,0.018889284,0.027138462,0.07999683,-0.019039072,-0.061816998,0.017313186,-0.051152766,0.039306473,-0.06226915,0.01846972,0.04963292,0.020879248,0.14606093,-0.09989049,-0.005342353,0.06521556,-0.024311792,0.041433204,0.066303164,0.022143282,-0.035711862,0.009525745,-0.057548966,-0.0806604,0.0028130244,-0.010393038,0.112975076,-0.0018174571,0.030156288,-0.02206562,0.04948243,0.032909732,0.01555169,0.014569762,0.0039269393,0.017587842,-0.0105174305,0.016612422,0.077120796,-0.042344075,-0.06468686,-0.036919363,0.03040028,-0.019409828,-3.251894E-33,0.040625818,0.016581528,-0.021927938,0.021516323,0.03471849,-0.045520667,-0.017614665,-0.07006911,0.0071153007,0.0025327879,0.005648088,0.04241776,-0.010652379,-0.023858054,0.012913902,-0.027931714,0.004287367,-0.027264068,0.061210893,-0.039421324,0.12714641,0.03849749,-0.094134904,-0.007945886,0.04947471,-0.056016088,-0.087781586,0.0797169,-0.025210094,-0.06051712,-0.019571543,-0.060732212,-0.012113058,0.017380701,-0.06130845,0.06727084,-0.033202328,0.091200486,0.02490767,-0.07749138,0.05946457,-0.02085205,-0.0049607577,0.050158907,-0.024974424,-0.054097354,-0.034355577,-0.0655865,-0.06576413,0.056954406,0.05969364,0.08660039,0.0020785942,-0.09242396,-0.052793916,0.034658235,0.03479103,0.0055098752,0.00982389,0.11976097,0.03207969,-0.012991179,0.05993113,0.037464924,-0.05148737,0.0068404386,0.07629449,-0.0012071802,-0.05490101,0.061263554,-0.008928003,-0.004517294,0.03881187,-0.030366309,-0.032273546,0.0091122035,0.0014949276,0.031635325,-0.028407004,-0.069446735,0.036516335,-0.014874138,0.049066026,0.12900232,0.05830397,0.06482125,0.06983362,0.041088205,0.0348922,0.07437936,-0.011469576,-0.043531597,-0.009242795,-0.060591742,-0.11750616,-5.111815E-8,-0.015631394,-0.027901249,-0.123293586,-0.05102494,-0.012230896,-0.0066172937,0.025514597,0.033083055,-0.022466997,0.059927735,0.09024767,-0.039916337,-0.09570077,0.0348283,0.020531913,-0.071214065,-0.023437353,0.013603563,-0.100605406,-0.067646325,0.031216368,0.0102580255,-0.017372983,0.031901423,-0.032343123,0.011557923,-0.019640395,0.0059259743,0.030634984,0.023833074,-0.082269505,0.0863291,-0.014290165,-0.027523914,-0.018137619,-0.030911706,0.01040259,0.01957091,-0.028549556,-0.04439533,-0.061055064,-0.005775817,0.025976641,0.008309728,-0.030016769,-0.025356589,-0.07005577,-0.02789409,0.028691122,-0.013100756,-0.018772617,0.037944615,0.0311197,0.009778543,0.0055453815,-0.01825943,-0.022618072,-0.07403188,0.03977728,-0.08186114,-0.0038845171,-0.054700762,0.0027615656,0.054946393] fl=*,[child]


added support for bt on api 31

ok, since incallmanager sucks they dont support api level 31 so app crashes since incallmanager requires bluetooth ...
anyway, added support for new BLUETOOTH_CONNECT on api 31,  added symetrical methods for permissions in js that it already has for cam and mic and added way to grab bt permission that is required for platform and make it all work even when user wont grant permission ...

LGTM, just wondering what exactly gives us saving permission status in shared prefs instead of asking OS about it again? If it was granted or denied system will not ask the user again but return true/false immediately just like we do it right now or did I miss something?

Correct but system doesn't ask, we trigger the permission request and android doesn't tell u whether user granted or denied. By default all permissions are denied, so since we manage permissions, we need to know whether it's denied by default or by user. For first time we ask user once they say, we don't anymore, they can change it in os app preferences

Ita about managing denied status, we can not determine if it was denied by user or its first time, hence the flag in shared prefa


eth balance requests being sent to satstack

Hi, I just installed satstack and got it up and running, very happy to be able to use my full node with ledger live :slightly_smiling_face: :tada:
When I look in the satstack logs, I see requests being made to eth related endpoints (e.g. /blockchain/v3/eth/addresses/0xADDR/balance), what exactly is happening to these requests? Are the requests forwarded to ledgers backend? Is ledger-live operating in a degraded state for non bitcoin assets if I have EXPLORER set to point to satstack?

@xwvvvvwx Thanks for taking satstack for a spin! 🙌
We're aware of this issue - rest assured that you won't experience this once https://github.com/LedgerHQ/ledger-live-desktop/pull/3291 is merged, which includes a native integration (easy setup, slick UX, etc) for satstack.
Let's keep this open until the official release, in case someone else runs into it.

Should be good with Ledger Live 2.18.0! Check out the updated usage instructions here (see recommended setup).

awesome! Thanks :pray: :sparkles:


Fix for a bogus issue #207 leads to regression in a working code

In the issue #207 the user dropped default database without a ClickHouse restart.
Which is not supported by CH, see source code on global context.
User must set the new default database name in the server settings.
And user must restart the whole server.
Now in a default scenario of c = create_client()
c.database is None and not default
Which leads to any code relying on the default to fail.

Do you know of any code actually relying on a value of Client.database?. None of the client methods rely on this value since all functionality must operate correctly in the user's default database which ClickHouse will use if the database parameter is not specified.  If the user actually needs that value, they can substitute the currentDatabase() function in their query or use a client.command(SELECT currentDatabase()) to get it.  (The better argument is that Client.database should be a private _database property, since it's intended use is internal functionality, but that would be even more of a breaking change.)
Also, in addition to the situation where the default was dropped without resetting the server wide default_database setting, the previous code was broken even if the just default_database for the specified user was dropped.  While in theory the operator should not do that, it's cleaner and more user friendly to remove the unnecessary and possibly invalid currentDatabase() call, and not break existing clients or client applications with confusing error messages.


Graphrag：The recall effect is not ideal

Description
Recently, I have been using Graphrag to index patent documents. The embedding model used is BGE-M3, and the document is divided into paragraphs with some additional segmentation rules: small paragraphs are merged, large paragraphs are divided by 384 tokens, and overlap=120
After indexing 30 patent data, the effectiveness of the recalled documents was still acceptable, but after reaching 1000 patent data, for the test set, not a single one was successfully recalled
The code during debugging is as follows
docs = self.document_collection.search(query_type='hybrid').vector(query_embedding).text(query).limit(k).rerank(reranker=Rerank_model).to_list()

At the same time, I also tested many different text segmentation methods and embedding models, and even after reaching 1000 articles, the results were still not ideal
Link
No response

hi @minglong-huang
looks like you were querying on a dynamic dataset, I'd guess this is caused by a bug already fixed, which is contained in latest lancedb release v0.16.0.
additionally, it's recommended to call optimize to achieve better search performance for dynamic dataset

Thank you, I will give it a try


Add Slack Integration

It is hard to follow changes.
This is also a test for slack integration💎

Done


test reporter: code cleanup

What this PR does

Remove duplicate VM snapshot function
Skip all volume snapshot log collections when volume snapshots are not available
Remove unused function parameters
Add print functions to avoid duplication of writers and '\n's, and unhandled error warnings
Improve some messages

Fixes #
Why we need it and why it was done in this way
The following tradeoffs were made:
The following alternatives were considered:
Links to places where the discussion took place:
Special notes for your reviewer

Checklist
This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR.
Approvers are expected to review this list.

[ ] Design: A design document was considered and is present (link) or not required
[ ] PR: The PR description is expressive enough and will help future contributors
[ ] Code: Write code that humans can understand and Keep it simple
[ ] Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
[ ] Upgrade: Impact of this change on upgrade flows was considered and addressed if required
[ ] Testing: New code requires new unit tests. New features and bug fixes require at least on e2e test
[ ] Documentation: A user-guide update was considered and is present (link) or not required. You want a user-guide update if it's a user facing feature / API change.
[ ] Community: Announcement to kubevirt-dev was considered

Release note

NONE

/sig code-quality

Thank you for the clean-up! It looks good, only one doubt from my side.
Recently, it was discussed to change the artifacts path from .../k8s-reporter/1/... to .../k8s-reporter/worker1/.... Maybe this is outside of the scope of the PR... @xpivarc FYI

Yeah I'd rather keep that as a separate PR. It's a small change but it can do a lot of damage if anything external relies on the current path...

/lgtm

Thank you!
/lgtm


Updated validations.py python script.

Fixed the behavior of validate_user function in validations.py
Closes: #1
Fixes #<issue_number_goes_here>

It's a good idea to open an issue first for discussion.

[ ] Tests pass
[ ] Appropriate changes to README are included in PR

Added a change in the condition of validations.py python script.


Is there a way to get client.identity within a handler?

I am looking for a way to access the identity within a function, currently I am wrapping the handlers like below, but I feel like there should be an easier way?
client.handle("StartTransaction", ({params})=>{
    return StartTransaction({params}, client.identity)
})

Hi @VincentWillats ,
Sorry, perhaps I don't understand exactly what you're trying to do - but isn't the client already in scope anyway?
I assume you have some code that looks like this (simplistic example):
async function StartTransaction({params}, clientIdentity) {
    // register the transaction parameters and get a transaction ID to return
    const transactionId = await transactionDatabase.register(
        clientIdentity,
        params.connectorId,
        params.idTag,
        params.meterStart
    )

    // return the transaction ID
    return {
        transactionId,
        idTagInfo: { status: 'Accepted' }
    }
}

client.handle("StartTransaction", ({params})=>{
    return StartTransaction({params}, client.identity)
})

This could be re-written as simply:
client.handle("StartTransaction", async ({params})=>{
    // register the transaction parameters and get a transaction ID to return
    const transactionId = await transactionDatabase.register(
        client.identity,
        params.connectorId,
        params.idTag,
        params.meterStart
    )

    // return the transaction ID
    return {
        transactionId,
        idTagInfo: { status: 'Accepted' }
    }
})

Does this help?

Apologies in my example that would be right yes, but in my actual code base I have the functions split out in other files, since adding all the logic for 1.6 and 2.0.1 in that single file is a bit bulky.

I see; So your StartTransaction function is imported from another file.
In order to avoid "forwarding" these calls one by one to each function, you might want to use a wildcard handler in an arrangement like so...
const BootNotification = require('./bootnotification.js')
const StatusNotification = require('./statusnotification.js')
const StartTransaction = require('./starttransaction.js')
const StopTransaction = require('./stoptransaction.js')

const handlers = {
    BootNotification,
    StatusNotification,
    StartTransaction,
    StopTransaction,
}

client.handle(({method, params}) => {
    if (!handlers.hasOwnProperty(method)) {
        throw createRPCError("NotImplemented")
    }

    const handler = handlers[method]
    return handler({params}, client.identity)
})

That is a much cleaner way, thank you for the suggestion.


Spanish and Guarani filtering

Figure out why we're filtering out that much sentences for Spanish (~38%) and Guarani (~50%)

Small tip: always be precise about the data source, in this example this was only related to the public bi-text, and the answer was already provided (I know that our Discord channel is a bit overwhelming because of me hah) - duplication filtering occurred here.


TypeScript Intellisense Not Working Across Files

Hi! I've been trying to get TypeScript Intellisense to work with multiple files using the demo, but it's not happening. I added a tsconfig file thinking it might help, but no luck there either.
My main goal is to export a function from one file and have VSCode suggest to auto-import it. After that, I want to make sure VSCode can see the node_modules I install and check types correctly.
I saw this project called jside that seems to have everything working, including TypeScript Intellisense. They're doing something different with how files are set up, I think, but I can't figure out exactly what makes it work there and not in my case - demo project.
Any ideas on what I might be missing or doing wrong?

Thanks a lot, this was very helpful. Adding the headers fixed the intellisense across files issue for me and the node_modules work out of the box now - except for a few issues.
btw, do you know why extensionHost.worker.js tries to call the npm registry for every module under node_modules folder? Like for example if I have the uuid module, it would send a GET request to https://registry.npmjs.org/uuid. This is causing some issues since I have private modules mounted under node_modules but the worker keeps trying to call the npm registry and npm reject the request!

The npm default extensions calls registry.npmjs.org to get some metadata about packages that are used in the project, are you using it?

I don't have the npm extensions installed. I tried to install it and set npm.fetchOnlinePackageInfo to false but it didn't change anything!

Then I'm not able to reproduce nor seeing any code from this repo referencing registry.npmjs.org
Can you provide a simple reproduction repo?

It looks like it's not an issue with this library but with the VSCode typescript extension then, right?

Yes, that's right, VSCode did not complete this feature in the web version of the extension.
(You must disable ATA so that types are not downloaded automatically: https://github.com/cemalgnlts/jside/blob/main/src/workspace/userConfiguration.json#L38)

@cemalgnlts Yeah, and I tried to do that but the extension fetches the types anyway!

Sometimes (maybe because of the way I use it) when I delete IndexedDB and refresh the page, the configuration file starts working correctly.

I was able to disable this behavior by commenting out this subscription https://github.com/microsoft/vscode/blob/main/extensions/typescript-language-features/src/extension.browser.ts#L108
Do you think this is something that should be patched in this package? I understand if it's not and I can publish my changes under a different package.

I was able to disable this behavior by commenting out this subscription https://github.com/microsoft/vscode/blob/main/extensions/typescript-language-features/src/extension.browser.ts#L108
Do you think this is something that should be patched in this package? I understand if it's not and I can publish my changes under a different package.

Probably not. It's not a bug but a feature you don't want

I discovered that you can force it to read using declare module and including all *.d.ts files in the node_modules folder.
Screen recording 2024-03-14 12.28.56.webm


maj 2019 tasks, step 1

idata:

subtract data sets (e.g. background)
rebin to a certain number of equal bins
maybe: rebin using exp bin size, this is used by a few sans people
map data points by some function
rescale axis
constructor using axis and values from literals

ifunc:

exp fit fct
vectorized parameter values shown in node inspector
fix: vectorized plot data

misc:

fix: idx editor dual mode
unpack vectorized data set from sim scan sweep

(And this actually works, see below)

Vectorised dataset support by zip file is now implemented, needs testing @climbcat

Very good, I will test that!
Four new nodes have been added to ifl and uploaded to elearn1, where the case has been sketched out.

@willend
I have tested the .zip method as you requested above. If I run MATLAB manually, it works fine, both on my system and on elearn1. However, if I run this functionality through the worker process, I get the same error message in both cases:
T2 : idata_dd540157495b4c80a963c2b5209af340 = iData('http://sim.e-neutrons.org/static/data/jaga15_SANSsimpleSpheres_20190513_135017/rdetector.zip');
wget http://sim.e-neutrons.org/static/data/jaga15_SANSsimpleSpheres_20190513_135017/rdetector.zip -O /tmp/tp84994144570907190.zip
iLoad: Can't extract file "/tmp/tp84994144570907190.zip" (unzip).
Warning: Function UNZIP requires Java.

And indeed, the zip method does not work when used through the interface. There must be something different on the the virtualized system?

status update:
The following are DONE:

subtract data sets (e.g. background)
vectorized parameter values shown in node inspector
fix: vectorized plot data

The following are HALF:

map data points by some function
rescale axis (actually, map_axis_1)
unpack vectorized data set from sim scan sweep

The following are suggested SKIP:

maybe: rebin using exp bin size, this is used by a few sans people
fix: idx editor dual mode

The following are new:

Create Log and Square models as ifuncs
Enable construction-time model determination as default parameters

Log and Square axis/signal IData->IData mappings ("operators") have been created, and the map and map_axis functions have been discarded in favour of a From_model.

rebin iFit code approximately: b=interp(a, new_axis) , where new_axis=linspace(min(a.x),max(a.x),newbins)

Most of the tasks have been done and implemented, this task can be closed.


Support numpy or Google docstrings

Are there any plans to do this? Is this already implemented?

We could add a new flag --convention same as pydocstyle does.

Would be a very nice feature !

Would be a very nice feature !

While this is still open, I wonder what do people currently use to format their numpy/Google docstrings? I could not find any alternatives to docformatter, and it's hard to believe developers are doing this by hand and not losing their sanity in the process.

I could not find any alternatives to docformatter

@ubalklen there is also https://github.com/dadadel/pyment

The last time I tried it, it had a problem that running pyment -w . (-w is for formatting files in-place) multiple times doesn't produce the same result (it keeps adding docstring for parameters that already have docstrings)
But that should be a solvable problem.

any update on this?

@Goldziher I recently started helping out with maintenance.  I've been focusing on merging all the open PRs and addressing issues that were related to PEP 257 conventions.  Once these are addressed, I'll release a new version (v1.5.0) of docformatter to PyPi.  After that I plan to tackle the group of issues related to the various docstring styles (#43, #60, #68, #70, #78, #79).  Once that group of issues is addressed, I'll release v1.6.0.  Hopefully that'll happen in the next month or so.

@Goldziher I recently started helping out with maintenance.  I've been focusing on merging all the open PRs and addressing issues that were related to PEP 257 conventions.  Once these are addressed, I'll release a new version (v1.5.0) of docformatter to PyPi.  After that I plan to tackle the group of issues related to the various docstring styles (#43, #60, #68, #70, #78, #79).  Once that group of issues is addressed, I'll release v1.6.0.  Hopefully that'll happen in the next month or so.

Sounds great. Lemme know when it's ready to test on a codebase 😉

Now that I'm thinking of it, I just stumbled upon the docstring_parser library. @weibullguy, are you aware of it? Is it something you ever considered at one point? It seems to be a good library to abstract the support of ReST, Google, Numpydoc-style and Epydoc styles.

@kdeldycke I am aware of it now, thanks!  It looks promising for dealing with the various syntax flavors.

This would be a great feature.
Am about to convert a library from sphinx-style to google, and will be rewrapping a bunch of lines by hand.
Another library that parses, if you’re not already aware, is Griffe, which is used by mkdocstrings.

Would be very needed !!!

This has been marked as low urgency, but for me, the lack of support for Google doscstrings means I cannot use docformatter.

what do people currently use to format their .../Google docstrings?

We did it by hand.  And we liked it!  (Though now that I'm no longer under G's thumb, I just started looking for such a tool and stumbled onto this issue.)

I could not find any alternatives to docformatter...

I just discovered https://github.com/heavenshell/py-doq which generates NEW docstrings, but explicitly will not update https://github.com/heavenshell/py-doq/issues/30
It seems to me that the various tools out there all touch on the subject, but none hits all of the bullet points (at least the ones I want):

Create docstrings from scratch with the same (exactly, or kind of) reasoning as pylint will use to complain (e.g., min-length, no-docstring-rgx)
Create docstrings that are almost-but-not-quite to documented specs (or generally, claim to follow documented guidelines for any X, except the rules they don't like and not let you easily make them right)
Update docstrings as parms are added/removed/reordered
Find the variety of config files consistently (IMO, the only thing pyproject.toml did was fulfill https://xkcd.com/927/, and even the it is suspect)


NIFI-5900 Add SelectJson processor

Overview
The goal of this PR is to further fortify NiFi when working with large JSON files.  As noted in the NiFi overview, systems will invariably receive "data that is too big, too small, too fast, too slow, corrupt, wrong, or in the wrong format."  In the case of "too big", NiFi (or any JVM) can continue just fine and handle large files with ease if it does so in a streaming fashion, but the current JSON processors use a DOM approach that is limited by available heap space.  This PR recommends the addition of a SelectJson processor that can be employed when large JSON files are expected or possible.
The current EvaluateJsonPath and SplitJson processors both leverage the Jayway JsonPath library.  The Jayway implementation has excellent support for JSON Path expressions, but requires that the entire JSON file be loaded into memory.  It builds a document object model (DOM) before evaluating the targeted JSON Path.  This is already noted as a "System Resource Consideration" in the documentation for the SplitJson processor, and the same is true for EvaluateJsonPath.
The proposed SelectJson processor uses an alternate library called JsonSurfer to evaluate a JSON Path without loading the whole document into memory all at once, similar to SAX implementations for XML processing. This allows for near-constant memory usage, independent of file size, as shown in the following test results:

The trade-off is between heap space usage and JSON Path functionality.  The SelectJson processor supports almost all of JSON Path, with a few limitations mentioned in the @CapabilityDescription.  For full JSON Path support and/or multiple JSON Path expressions, EvaluateJsonPath and/or SplitJson processor should be used.  When memory conservation is important, the SelectJson processor should be used.
Licensing
The JsonSurfer library is covered by the MIT License which is compatible with Apache 2.0.
Testing
This PR is a follow-on from #3414 in which I proposed a similar solution that required extenseive unit testing.  Tests from that PR were adapted and preserved for this PR, even though many of them are testing the JsonSurf library.  This is a much simpler PR since the path processing is handled in a third-party library.
As for the memory statistics noted above, they were gathered using the same methodology described in #3414.  For posterity, here's a python script to generate JSON files of arbitrary size:
import uuid

(I, J, K) = (1, 8737, 3)
with open('out.json', 'w') as f:
    f.write("[")
    for i in range(0,I):
        f.write("[")
        for j in range(0,J):
            f.write("[")
            for k in range(0,K):
                f.write('"' + str(uuid.uuid4()) + '"');
                if (k < K - 1):
                    f.write(",")
            f.write("],\n" if j < J - 1 else "]\n")
        f.write("],\n" if i < I - 1 else "]\n")
    f.write("]\n")

How to use SelectJson Processor
Given an incoming FlowFile and a valid JSON Path setting, SelectJson will send one or more FlowFiles to the selected relation, and the original FlowFile will be sent to the original relation.  If JSON Path did not match any object or array in the document, then the document will be passed to the failure relation.
JSON Path Examples
Here is a sample JSON file, followed by JSON Path expressions and the content of the FlowFiles that would be output from the SplitLargeJson processor.
Sample JSON:
[
  {
    "name": "Seattle",
    "weather": [
      {
        "main": "Snow",
        "description": "light snow"
      }
    ]
  },
  {
    "name": "Washington, DC",
    "weather": [
      {
        "main": "Mist",
        "description": "mist"
      },
      {
        "main": "Fog",
        "description": "fog"
      }
    ]
  }
]

JSON Path Expression: $[1].weather.*

FlowFile 0: {"main":"Mist","description":"mist"}
FlowFile 1: {"main":"Fog","description":"fog"}

JSON Path Expression: $[1].name

FlowFile 0: "Washington, DC"

JSON Path Expression: $[*]['weather'][*]['main']

FlowFile 0: "Snow"
FlowFile 1: "Mist"
FlowFile 2: "Fog"

Checklist

[x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?

[x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

[x] Is your initial contribution a single, squashed commit?

[ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
(Note: mvn clean install completes without error after disabling FileBasedClusterNodeFirewallTest and DBCPServiceTest.
Adding -Pcontrib-check fails , but it appears to fail on master branch too)

[x] Have you written or updated unit tests to verify your changes?

[x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

[x] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?

[x] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?

[x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

See Also
SplitLargeJson: #3414
StreamingJsonReader: #3222

@ottobackwards This approach (SelectJson) is simpler than my original approach (SplitLargeJson -- #3414 ) and provides the same system resource advantages, with one exception: If the targeted JSON excerpts themselves are large, then I'm fairly certain the SplitLargeJson approach would be better.  This is because the JsonSurfer library does not provide a means by which to supply an OutputStream to its bind signature... as far as I can see.
In other words, the required heap space is independent of incoming FlowFile size, for both SelectJson and SplitLargeJson, but if the outgoing FlowFile size is also large, then the SplitLargeJson approach is currently a better approach, from a system resource perspective.

Note that this PR is closely related to #3414.  I don't intend for both PRs to be accepted.  They are two different approaches to solve the same problem.  Here are the high level questions/options that I can see:

Create a new SplitJsonProcessor that uses javax.json (PR 3414)
Create a new SelectJson that uses JsonSurfer (this PR)
Keep only SplitJson and optionally employ a streaming approach, backed by javax.json, when a new property is set
Keep only SplitJson and optionally employ a streaming approach, backed by JsonSurfer, when a new property is set

The later two options involve possible incompatibilities between the underlying libraries (javax.json and JsonSurfer) and the exiting processor behaviors.  My vote after considering this for a while is to keep the EvaluateJsonPath processor, keep the SplitJson processor, and add the SelectJson processor.

These PR descriptions are amazingly helpful, thanks!
I haven't followed the other PR closely, so not sure if this was already discussed, but is there significant benefit to using SelectJson/SplitLargeJson vs. the record processors like QueryRecord/PartitionRecord/SplitRecord with a JsonTreeReader and JsonRecordWriter?
The original json processors like SplitJson and EvaluateJsonPath existed early on, way before the record concept, and seems like most of their purpose has been replaced by the record concept.

@bbende I am not familiar with the split/partition records.  The issues I think in the past would have been not having a schema, and the jsonpath vs record query language functionality.
I know needing a schema is gone now, although I don't know if there is parity.

So, I don't think split record supports a query.  Where as split json can reach into a json structure and find an internal array and split that  etc etc

@bbende Thank you!  As for your comment: It could be that record processing is the "new normal", in which case #3222 is an important PR.  In either case, however, as long as the EvaluateJsonPath and SplitJson exist in NiFi, I think there is argument to provide memory-efficient counterparts.
@ottobackwards I haven't written much because it seems we're still waiting for review by a different group of people?  I would be happy to make identified changes if folks think that one of these two PRs (#3455 or #3414) would be approved.  However, if neither PR looks very promising or likely to be accepted (that's OK of course, no problem!) then I don't want to spend any more time on this.

@ottobackwards If an invalid JSON Path is submitted, the SelectJson processor would be in an invalid state.  This PR relies on the underlying JsonSurfer for validation of the JSON path and I don't know whether that library will filter out / report the few edge cases where a path is valid but not supported.  I could look into that further depending on overall trajectory, as mentioned above and on the other PR.

@bbende what would you suggest to move these two PR's forward ( or one of them rather )?  I think that on of them should land, but the decision between using the lib or hand rolled json path capability is something that I think needs more community involvement.

I think Matty B and Mark are the most involved with the record related processors, so they might be able to offer the best guidance. From a high-level, it seems like maybe we should have both approaches (this PR and Mike's streaming reader).

@ottobackwards and/or @bbende Do you know what my next step would be, to push this PR forward?  bbende mentioned "Matty B" or "Mark".  See also my comment from May 21st.
Thank you for any help or suggestions.

Besides de-conflicting, from my POV, I think that there needs to be some consensus on which of the approaches makes for sense for the project.  I think that should come from the committers.
I personally would vote for this PR, using the surfer, as it has less maintenance burden.

@ottobackwards Yeah I agree that this PR would probably have less maintenance burden.  That's why I closed PR #3414 a few months ago.  I would be happy to merge or to close this PR.  I'd prefer the former but just want to go one way or another.
In my opinion, both this PR and PR #3222 should be merged.  This would allow for streaming json processing from either a record-based processing model or the "traditional"/original processing model.  I think #3222 and this PR complement one another, but they're not dependent on each other.  I did not submit PR #3222 and it is apparently closed.  I submitted PR #3414 but I closed that one in favor of this one.  Therefore, I think this PR should be merged and @MikeThomsen can decide separately whether he would like to resubmit his work.

+1 from me

@arenger hope you are holding out patience wise on this.  Hopefully some traction soon ( although it looks like it will be post release )

@ottobackwards Thanks Otto, yeah I look forward to hearing whether or not folks want to include this.  If they do, it should be all ready to go.

closing due to inactivity but the effort/description/discussion are solid.  i suspect we'll have to implement a stale PR bot soon and this would get swept.  But great descriptions here.  I really wonder if the record processors take care of this well enough now.  Either way that is where the effort should go on a go forward basis for something like this.


cmake fails

Report

[x] I have searched existing issues and this is not a duplicate

Issues and Steps to Reproduce
in new osx installation:
mkdir build
cd build
cmake ..
->
You have called ADD_LIBRARY for library yogacore without any source files. This typically indicates a problem with your CMakeLists.txt file
-- Configuring done
CMake Error: Cannot determine link language for target "yogacore".
CMake Error: CMake can not determine linker language for target: yogacore
Expected Behavior
Create working makefiles
Actual Behavior
make
make[2]: *** No rule to make target `CMakeFiles/yogacore.dir/build'.  Stop.
make[1]: *** [CMakeFiles/yogacore.dir/all] Error 2
make: *** [all] Error 2
Link to Code
I guess this may have someting to do with changing Yoga.c to Yoga.cpp (why the change?).

Fixed via #676. Please reopen if you have further issues.


[no ci]add DingTalk QR code

Why
Add DingTalk discussion group.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.chufeng.qy seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


[v6.0] Image caching when using the image decoder

Related forum thread.
Image decoder calls open_cb multiple times in order to display a single image. In each case the decoder decodes the entire image.
This is even worse with animation - on every frame open_cb is called multiple times.
A non trivial image format (PNG or JPG for example) takes time to decode.
On architectures with enough ram to store a decoded image, it does not make sense to decode the same image again and again.
I suggest adding a caching logic into lvgl img to decode an image only once, until a different image is set (by lv_img_set_src).
Every time a new image is set (by lv_img_set_src), the cache should be invalidated.
When testing this, my experience shows a very significant performance improvement when the decoded image is cached.

I've just realised that caching this way is not working well with more than one image. Let's say there are 2 pcs  200x200 PNG images. The VDB is set to 20 lines and now the whole screen is being refreshed. In this case, when drawing the first chunk (first 20 lines of the display) both images needs to be decoded. Therefore caching is pointless with more than one image.
If you have enough RAM you can call lv_img_decoder_open in your code to get the decoded image and build an lv_img_dsc_t variable with the decoded data. It would give the biggest flexibility.
Or you can use a screen sized VDB. In would ensure that an image can be drawn with one open/close because the screen always will be refreshed in one big chunk. However, the images still need to be decoded on every new frame...

@kisvegabor With enough ram why not cache multiple images? We can set a threshold on the number of images (or number of KB) we would like to reserve for the cache.

If you have enough RAM you can call lv_img_decoder_open in your code to get the decoded image and build an lv_img_dsc_t variable with the decoded data. It would give the biggest flexibility.

You don't need lvgl image decoder if all you want is to decode the image beforehand and set_src the decoded image.
More generally - What is the use case of lvgl image decoder? Is anyone using the image decoder today? If yes, how did they handle this problem?
My point is that I'm not sure image decoder is usable at all for any practical purpose if the same image is decoded multiple times. It would just become too slow and waste significant amount of cycles, to the point where I'm not sure anyone would want to use it.

More generally - What is the use case of lvgl image decoder? Is anyone using the image decoder today? If yes, how did they handle this problem?

They can be used to decode any custom image format. They don't have to be as complex as a PNG or JPG. E.g. a simple RLE coding or so.
The image decoder is an abstraction. You can buffer/cache images as you want. E.g. if you have RAM to store 10 uncompressed icons you can cache the last 10 decompressed icons but you can have 100 PNG icons in the flash each stored as an lv_img_dsc_t variable.  I really didn't consider the situation when an image uses a user created lv_img_dsc_t and the user "silently" changes the image data. I think it's a special case but of course, I'm open to improve it. However, as it's related to how buffering works in the decoder, maybe a custom function clear the cache or drop only one image would be the clearest solution.

Is anyone using the image decoder today?

I don't know.  :)

They can be used to decode any custom image format.

In my opinion, a decoder without cache would have very narrow usability.
It would only be used for the simplest encodings where decoding is so fast that you don't care it's called many times for the same image.
But I don't think it would be practical to use an image decoder without caching for any non trivial encoding (PNG, JPG, etc.).

2. The image decoder is an abstraction. You can buffer/cache images as you want

@kisvegabor I agree that caching could be implemented either on the image decoder or on lvgl. But does it make sense to implement caching multiple times for each image decoder?
Imagine - that in the future you have multiple decoders for various image formats. Would each decoder implement its own caching? Would the user need to configure caching parameters individually for each decoder? (enable/disable and threshold)
Why not implement caching only once in lvgl?  then the user would have a centralized control over the caching configuration.

I really didn't consider the situation when an image uses a user created lv_img_dsc_t and the user "silently" changes the image data.

If you insist not implemeting caching in lvgl, and implement it in each decoder instead, I still think that the trigger for invalidating the cache for an individual image should be a call to lv_img_set_src.
The simplest implementation would just require lv_img_set_src to clear some flag that could be set and examined by the decoder.

I see your points. A cache like "don't close the last X images" can work. The cache entries can be timestamped and always the oldest would be overwritten.
However, it would be very unpredictable regarding memory usage. Let's say "X=3" so the last three images are left open. If the last three images are simple built images then the cached memory size is 0 bytes. But if three PNG images then it could be huge. It would be nice to have a memory limit besides a number of opened images but it's not trivial to tell the memory used by an opened image. From the above example, it's clear that the used memory is not related to the image size.
So if you agree, I can easily add this caching mechanism. If the user wants his own caching then he can set "X = 0".
The cache clean on lv_img_set_src should be optional because typically lv_img_dsc_ts are not modified in runtime. Thus cleaning the cache on every lv_img_set_src would cause unnecessary cache cleans.
What do you think?

However, it would be very unpredictable regarding memory usage.

I agree. I also think that memory limit is better than image count.
What if we let the user configure the cache size in KB and track image sizes? built in images would consume 0 KB, while other images would consume according to data_size in lv_img_dsc_t. The caching logic would cache an image only when adding its size to the total (current) cache size does not exceed the configured cache size.

The cache clean on lv_img_set_src should be optional because typically lv_img_dsc_ts are not modified in runtime. Thus cleaning the cache on every lv_img_set_src would cause unnecessary cache cleans.

I'm not sure I understand this. Why would it cause unnecessary cache cleans? Usually image data is only set once using lv_img_set_src per image. If you call lv_img_set_src twice for the same image, it probably means you changed the image source, isn't it?
Why not update the "invalidate cache" flag in lv_img_decoder_dsc_t every time lv_img_set_src is called?
lv_img_decoder_dsc_t is kept per image per decoder right? so we would know when image data has changed.
This would allow a decoder to know if it could use the cache or not, in case we decide implementing the cache in the decoder. And the same flag could be used for the generic caching logic as well, when it's
enabled.
Does it make sense?

... while other images would consume according to data_size in lv_img_dsc_t. The caching logic would cache an image only when adding its size to the total (current) cache size does not exceed the configured cache size.

It's not entirely true. Imagine that a PNG file is decoded from an SD card. E.g lv_img_set_src(img, "S:pic.png"). In this case, lodepng first reads the whole file, allocates the output buffer and allocates other buffers to process the image. So what if you let 100kB for caching and 20 kB left. The image is 19kB so it seems ok, but a lot more memory can be allocated during decoding.
Or if decocder_line_read_cb is used probably the image doesn't need to be fully decoded. Maybe a line can be returned immediately. Maybe the previous line is also required to decode (e.g. they are XORed). So there are even more unknown things here.
Or when built-in images are loaded as files there is nothing to cache because they can be read directly line-by-line, so it seems 0 kB cache is used. The library can think that any number of images from files with built-in color format can be opened. But if the file is left open then it also consumes memory.
In light of these uncertainties, I think memory limit can't work. It's even difficult to predict the used memory in the decoder.
IMO it might be the user's responsibility to ensure that he will have enough memory even if the 3 largest images are loaded at the same time. (assume he allows 3 images to be cached)

I'm not sure I understand this. Why would it cause unnecessary cache cleans? Usually image data is only set once using lv_img_set_src per image. If you call lv_img_set_src twice for the same image, it probably means you changed the image source, isn't it?

You can use the same image source in multiple image objects. It's quite common. Think about icons or any type of symbols. For example, a water system which has a lot of valves. In this case, only two image sources are used (valve_open and valve_close) but for many valves.  Cleaning the cache always when a valve changes state wouldn't be reasonable because we still have the same two images.
Or imagine a file system browser with file and folder icons. It's the same case.
So when happens this issue?

You have an lv_img_dsc_t variable (call it dsc)
Store some image data in  dsc->data field
Use the dsc in an image object with lv_img_set_src
Change the content of dsc->data to an other image
Set the image again with lv_img_set_src
And the cache is not invalidated because dsc is still the same and the image is not updated.

So my question is once you already work with dsc->data why is it a big deal to call an function like lv_img_decoder_invalidate_src(&dsc)? It seems quite straight forward to me. You need to notify the library if you change an alredy used resource.

So my question is once you already work with dsc->data why is it a big deal to call an function like lv_img_decoder_invalidate_src(&dsc)? It seems quite straight forward to me. You need to notify the library if you change an alredy used resource.

@kisvegabor I see your point. I didn't think of the case of using the same image source on multiple image objects, but as you put it, it is probably very common.
So I'm convinced the best solution is, as you suggested, expect the user to call an invalidate_src function.
I noticed you called the function "lv_img_decoder_invalidate_src".
Does it have to be a decoder function? Suppose the user is not aware of the decoder, he is just setting sources to images and those sources could be either PNG or JPG images. Maybe the user is not even aware whether the image source he is holding is PNG or JPG. To invalidate the cache, if it's a decoder function, we would need to associate an image source with the right decoder.
Or maybe you meant that this would be some global function unrelated to specific decoder?

Or maybe you meant that this would be some global function unrelated to specific decoder?

Yes. For example

lv_img_decoder_invalidate_src(&my_dsc)
lv_img_decoder_invalidate_src("S:pic.png")

The image drawing module can have an array to hold the opened lv_img_decoder_dsc_t variables. lv_img_decoder_invalidate_src would check the decoder dscs in the array and if a dsc's source matches it closes that one.
It's really not the best name as the cache should be managed by the drawing module. lv_img_draw_invalidate_src doesn't sounds much better. Do you have any idea?
Anyway, independently from the name it seems we agreed on how this feature should work. Or is there anything else?

if a dsc's source matches it closes that one.

@kisvegabor Closing an image is invalidating it? Will you keep images open until they are invalidated? (because I think that today they are repeatedly opened and closed).
Will you call open only once for an image, until it's invalidated (or deleted)?

Anyway, independently from the name it seems we agreed on how this feature should work. Or is there anything else?

Yes, I think we agree.
Regarding the name, how about lv_img_invalidate_src?

Closing an image is invalidating it?  Will you keep images open until they are invalidated?  Will you call open only once for an image, until it's invalidated (or deleted)?

Yes, for all of them. :)

Regarding the name, how about lv_img_invalidate_src?

It might give the false impression of this function is related to the image object, however, is something more generic and lower level.

@amirgon @kisvegabor If the function is related to the decoding logic, then I think for consistency, it's best to call it lv_img_decoder_invalidate_src. It makes it clear that it relates to the decoding logic.
Someone who isn't aware of the decoder can still use the function when they see it written in the documentation.

I implemented it in lv_img_cache.c. When it opens an image it measures the time to open and try to keep the slow to open images in the cache. I added a life time parameter which means images used earlier than this time are considered unused and more probable to overwritten in the cache.
I thought it's worth to be in a new module so I used the lv_img_cache_invalidate_src name.
The default cache size and life time can be set in lv_conf.h but can be changed in run time too.
What do you think?
@amirgon Can you test the performance?

@kisvegabor I didn't check it yet, but I noticed one problem:
https://github.com/littlevgl/lvgl/blob/a9c818a6efa7762202489e6ef7bffbfebb053e07/src/lv_draw/lv_img_cache.c#L27
img_cache must be declared as gc root, like the global linked lists, otherwise it will be collected by gc prematurely.

@amirgon
I've fixed it. See ca1f21a
I'm not sure if it's valid: lv_img_cache_entry_t * cache = LV_GC_ROOT(_lv_img_cache_array);
I also updated the caching policy which decides which cache entry to (re)use. I removed the life time parameter because it's not used in the new algorithm.
It works like this:

every cache entry has a life and a time_to_open parameter
every time lv_img_cache_open is called the life of all cache entries is decreased by 1. (make them older)
if the image is found in the cache it's life is increased by time_to_open to make it "more alive"
if there is no more space in the cache the entry with the smallest life will be reused. (the oldest)

This way the images which are more difficult to open or used frequently will live longer.

@kisvegabor It works great! Using the decoder with large enough cache, gives the same user experience as using native TRUE_COLOR_ALPHA image directly.
I only checked performance with two images, though. I did not check how the cache is behaving when dynamically adding more images, whether aging behaves correctly etc.
Maybe we need to think of a test bench. It's not trivial to check it right.

@amirgon

It works great! Using the decoder with large enough cache, gives the same user experience as using native TRUE_COLOR_ALPHA image directly.

Glad to hear that! :)

I only checked performance with two images, though. I did not check how the cache is behaving when dynamically adding more images, whether aging behaves correctly etc.

I was testing it with the demo UI using its bubble wallpaper, the dices PNG image, and a mouse cursor.
So three images: 2 built-in, 1 PNG. I set the cache size to 2 and moved the cursor over the dices. This way all 3 images were drawn but the PNG was opened only once and the other two were opened/closed continuously. So basically it seemed working.

Maybe we need to think of a test bench. It's not trivial to check it right.

Yes, a more advanced test would be great but it's really not trivial.
To have a predictable test we should use lv_img_cache_open directly instead of clicking/dragging the images.  What do you think can we test it with 3 images and cache size = 2?
We can create 3 dummy image decoders (for the three images) with three different delays inside their open function, call lv_img_cache_open a lot of times by pseudo-randomly selecting an image to open and finally check which two images are cached.

Yes, a more advanced test would be great but it's really not trivial.
To have a predictable test we should use lv_img_cache_open directly instead of clicking/dragging the images. What do you think can we test it with 3 images and cache size = 2?
We can create 3 dummy image decoders (for the three images) with three different delays inside their open function, call lv_img_cache_open a lot of times by pseudo-randomly selecting an image to open and finally check which two images are cached.

Yes as you said - testing it right is not going to be trivial.
We can try what you suggest.

@kisvegabor
https://github.com/littlevgl/lvgl/blob/d79df213a16d2e18de1c2e074c31de3855568150/src/lv_draw/lv_img_decoder.h#L40
Why didn't you defined LV_IMG_DECODER_OPEN_FAIL as NULL?
Casting a pointer to -1 is awkward and causes a problem in the Micropython binding.
I support converting None to NULL, but I would like to avoid converting any integer value to pointer, as this would cause a segfault if someone puts integer instead of a pointer by mistake.

@kisvegabor Here is another idea for the image decoder:
Image loading might be slow. If an image is read from the network - could be very slow.
Today everything is synchronous - everything is blocked until the image is loaded.
How about supporting asynchronous image loading?
Something like this:

open_cb reports it started downloading the image asynchronously, and returns.
lvgl displays some placeholder (spinner?) instead of the image
When the image completed downloading, the system signals this to lvgl
lvgl replaces the image placeholder by the image.

What do you think?

Why didn't you defined LV_IMG_DECODER_OPEN_FAIL as NULL?

It's really awkward. I updated it. Not it returns LV_RES_OK/INV and the image data is set in dsc->imd_data.

How about supporting asynchronous image loading?

Sounds interesting. I've found an easy way to support it. Now, if the image can't be drawn the "No data" text is displayed as a placeholder. I added a the dsc->error_msg field to set a custom placeholder.
If the error_msg is not NULL then it will be displayed instead of the image. The decoder_open function still needs to return LV_RES_OK.
When the image arrives you can invalidate the cache entry and now open the image normally.

I added a the dsc->error_msg field to set a custom placeholder.
If the error_msg is not NULL then it will be displayed instead of the image.

@kisvegabor What if I want to display a spinner or another image as a placeholder, instead of a label?

(btw, it would require a change in the way you measure opening time in the caching logic)

I don't think so. Open time still shows how fast it was to open the image. If it has placeholder, it's fast to open and it's still valid to say "this image was easy to open, let's cache something more complicated instead".

When the image arrive from the network and I invalidate the cache to display it, how would you know how much time it took to fetch it from the network? I would want it to have high score in the cache because it took very long to fetch it from the network.

What if I want to display a spinner or another image as a placeholder, instead of a label?

A spinner would be too complex. A label simply can be drawn on the fly (instead of the image) which consumes no extra memory and means no extra complexity (like an addition animation).  So for now, I'd like to keep the text version.

When the image arrive from the network and I invalidate the cache to display it, how would you know how much time it took to fetch it from the network? I would want it to have high score in the cache because it took very long to fetch it from the network.

I see. I updated the image cache module. Now you time_to_open is part of the decoder_dsc. In your decoder_open you can set time_to_open to any value. If left 0, LittlevGL set the measured time.
It seems quite flexible and easy to use for me. What do you think?
Here is the updated png decoder:
png_decoder.zip

A spinner would be too complex.

@kisvegabor could you explain the difficulty? Isn't it just another lvgl object that is instantiated in such occasion, and occupies the same coordinates?

Here is the updated png decoder:
png_decoder.zip

Why don't you add it to github, instead of zip? I would suggest using lodepng repo as git submodule.
btw, I already have this on lv_binding_micropython/driver/png/, it's built by lv_micropython and I have a working pure-python decoder (that uses lodepng) on https://github.com/littlevgl/lv_binding_micropython/blob/dev-6.0/examples/example2.py

I think instantiation of another object may be the complexity.

I think instantiation of another object may be the complexity.

@embeddedt Why is this a problem? You can have a function pointer to a "default placeholder" init function. That function would create an configure the placeholder object.
If the user wants a different placeholder, all he needs to do is to implement such function and register it in that function pointer. (You may need also a "deinit" function to free resources when the placeholder is removed).
I think it would be nice if the default placeholder displayed a spinner, but it could display a label by default if you think it's better, as long as the user has the power to configure a different placeholder.

I think it would be nice if the default placeholder displayed a spinner, but it could display a label by default if you think it's better, as long as the user has the power to configure a different placeholder.

IMO loading images from the web is a very special case so adding generic support for it would be overkill.
In addition, the placeholder_init function would be called when drawing is in progress and you can't create new object while drawing because it would mess up all the previous optimizations and calculations about the process of rendering.
However, it seems you to me you can do it manually:

Create an image
Create a preloader object on it
When the image has arrived remove the preloader.

@amirgon
I'm trying to close the pending issue before the release. As I see the original and important issue has been solved by the implementation of caching.
We can continue the discussion about the placeholder if you don't like the above propoesed solution but it's already independent from caching itself. So if you'd like to improve the placeholder, please open an other issue for it.
And thank you for the contribution, caching was really a missing feature!
BTW, I've written about it here: https://littlevgl.com/sphinx/overview/image.html#image-caching


[WIP] Switch to using PyPy3-6.0.0

A PR to discuss if this seems like something we should do.
The basic motivation behind this PR is I was trying to look at what was making our XMLRPC APIs so slow, and it appears that fundamentally the issue is just that the XMLRPC APIs are poorly designed (duh) and rendering the huge XML structure that some of the APIs return is taking up the bulk of the time spent.
I'm pretty sure that PyPy is likely to speed up the performance of the entire code base, but I haven't actually went through view by view to get a sense of whether it did or not.
So the biggest downsides to switching to PyPy:

Big change to our runtime, which comes with it risk of something exploding in Production that we didn't find in testing.
PyPy3 is still on Python 3.5 (although it has some features, like f-strings). This wasn't a big deal in our code base (see the diff) but 3.5 is getting old enough it feels like people might stop dropping support soon?

I believe PyPy is actively working on 3.6 support, and #pypy suggests it is nearly done.

Some C extensions just don't work (although cpyext is getting better compatibility all the time), and when they do work they are often times slow.

Using Pure Python instead of C is often times just as fast, and when it's not there is typically a CFFI version of the library that can be used instead.

CPython has the bulk of the real world use, so PyPy might have more edge cases?
Performance is going to be less consistent due to JIT warmup.

I don't honestly think our performance is that consistent right now as it is.

Tests are likely slower?

PyPy's JIT works best when you're running the same code over and over.. tests do the exact opposite.

PyPI running on PyPy, double the confusion, double the fun. It's PYPYI!

The big upside is a pretty healthy performance boost (10x on XMLRPC list_packages() API when running locally, without any effort besides porting to PyPy itself).
This isn't actually fully working as of now, I see an error being generated from celery when running locally, and there are ~74 test failures (A lot of which look like fallout from going from 3.6 -> 3.5). Throwing this up to give folks a chance to comment, and to think about what a possible migration to PyPy might look like.

@pypa/warehouse-committers

PyPy is a great computer program.

PyPI running on PyPy, double the confusion, double the fun.

I'm probably more excited about seeing this happen, than is reasonable.

This is currently blocked on pypy/pypy#2912 either that being fixed in PyPy itself or in one of {PyCurl, Kombu}, or worked around in Warehouse.

ISTM installing PyPy 3 6.0.0 via pyenv when the Travis Job starts should work (but I haven't tested that).

Alright, I've updated this so that we're using a nightly version of PyPy which has the above bug fixed, and I've verified that Celery is working with it.
I've already gotten all of the tests working fine, so that means the main thing left is figuring out how to work in CI. We can just try to install PyPy3 as part of the Travis job, but that is yet another place we have to update what version of Python we're using.
Since our real deployment now uses Docker Containers, and I believe (@ewdurbin can confirm) that it uses a docker container as built by our Dockerfile, that it may make the most sense to just build our docker container in CI and run our tests inside of that. That will give us far more consistent behavior between local, CI, and production.

Just a side note: it might make sense to use CircleCI instead of Travis, given that it has better support for testing via Docker containers right away.

Oh, another solution to speeding this up would be to move some of the stuff we do into a "base image" of sorts that we manage as an independent repository. This wouldn't be able to be any of the Python level dependencies, but we could potentially have a "warehouse base" image that looked something like:
FROM debian:jessie-slim as build

RUN set -x \
    && apt-get update \
    && apt-get install --no-install-recommends -y \
        bzip2 build-essential libexpat1-dev libffi-dev libxml2-dev libxslt-dev libpq-dev libcurl4-openssl-dev libssl-dev

ADD http://buildbot.pypy.org/nightly/py3.5/pypy-c-jit-95312-3bb86a88e1d3-linux64.tar.bz2 /tmp/pypy.tar.bz2
RUN set -x \
    && echo "9d6e7c11645833794f5f207051331f1835f1ba10aec3c9aaa0ae0612e8829b4e  /tmp/pypy.tar.bz2" | sha256sum -c - \
    && mkdir -p /opt/pypy \
    && tar xjf /tmp/pypy.tar.bz2 --strip-components=1 -C /opt/pypy
ENV PATH="/opt/pypy/bin:${PATH}"

This would increase our final image size a little bit (since we'd have the dev headers installed in production, instead of just the libs themselves) but I don't think by a meaningful amount. It would make things like bumping the version of Python or adding a new C dependency harder to do, since we'd have to manage that inside of another repository, but maybe it would be worth it?

This is currently blocked on https://github.com/ambv/black/issues/613, where our use of f-strings (which PyPy3 supports even though it is a 3.6 feature and PyPy3 is 3.5) is causing black to flag the file as Python 3.6+, which causes it to use features which are not Python 3.5 compatible, which then causes PyPy to fail to parse that file because it doesn't support all of the Syntax from Python 3.6.
There are only a few ways around this:

Remove all of our use of f-strings so we are "Pure" 3.5 compatible.

I really don't want to do this, f-strings are a really nice improvement and PyPy supports them fine. Also a PyPy3 that supports 3.6 is hopefully in not the super far future, where it would suck to have to go through and undo all of the undoing of f-strings.

Patch black to add a --no-py36 flag that we can use to force < Python 3.6 (I assume it would still handle the fact that there are f-strings, just wouldn't format around them).

I'm not sure if black will accept this configuration option or not.

Wait until a PyPy3 that supports 3.6 comes out.

We can do this today with nightlies, and I think an alpha will be out soon?

Disable the numeric literals rule in black (one of two rules PyPy3 fails on, and the only one the current code base actually triggers) and hope that we don't trigger the other rule before a 3.6 supporting PyPy3 comes out.

Personally the only options that feel reasonable to me are patching black and waiting until a 3.6 PyPy3 comes out. I'm going to play with these two options and see what I can make happen.

PyPy3.5 v6.0 is now supported in Xenial boxes: https://github.com/cherrypy/cheroot/commit/cfee0e13e660f4f67a37c1928a737dd9ccc05fec

Closing this, it's bit rotted at this point and I don't think we're actually going to end up doing it. We're on CPython 3.7 now and will likely switch to CPython 3.8 when it comes out as well.
At this point we're likely to get more traction on getting the new APIs out which don't have the fundamental problems that the XMLRPC APIs do, and then maybe we will make a translation layer for the, and that can run in PyPy.


edit GETTING_STARTED.md with proper case

The example gave an incorrect solution as the test was looking for 'Hello, World!' not 'Hello, world!'

Thanks @gilmatic !


Bad Request for DeploySPLists Execution

I'm following Deployment guide for building access app template, but when I try to run DeploySPLists flow I have this error:
Error
Action 'Add_RequestBuildingAccess_to_Default_View' failed
O modo de exibição especificado é inválido. (The specified view mode is invalid.).
Any Solution?

I actually found a workaround myself.
The problem was in flow steps to add created fields to default view list. So I deleted these steps, let the flow create the lists and fields and then I added the fields to default view mannually.


Add missing action types

Motivation and Context
Can't set up response driven reminders.
What has changed
Added missing action types.
How to test?
Opsit.
Links
Trello: https://trello.com/c/4M3rxFYD

 Add missing action types to Ops tool for Yogi


Trying to get the dependence analysis right in the presence of friends and --cmi

I've had several attempts at getting this code right ...
I think this one may finally be okay. Could use another set of eyes to check it out.
The main bit to focus on starts from around line 837 in FStar.Parser.Dep.fs

This looks good to me. Summary of our lunch discussion:

this is correct, in that it "upgrades" the graph to make sure all the incoming edges onto a node are of the kind FriendImplementation, if at least one of them is, thus ensuring proper dependency discovery
this traversal could be made more efficient, by:

keeping a data structure on the side that marks each node that is friended by someone
treating each interface edge u --> v where v is friended by someone as a friendimplementation edge
computing the transitive closure of the graph using the two bullet points above.

The optimization can happen later. Dependency analysis seems to be dominated by reading & parsing the files.


Remove repeated logs that can rate-limit P4RT.

Remove repeated logs that can rate-limit P4RT.
Build and Test Logs:
rkavitha@0cad235d15a5:/sonic/src/sonic-p4rt/sonic-pins$ bazel build $BAZEL_BUILD_OPTS ...
INFO: Analyzed 235 targets (0 packages loaded, 0 targets configured).
INFO: Found 235 targets...
INFO: Elapsed time: 0.224s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
rkavitha@0cad235d15a5:/sonic/src/sonic-p4rt/sonic-pins$ bazel test $BAZEL_BUILD_OPTS ...
INFO: Analyzed 235 targets (0 packages loaded, 0 targets configured).
INFO: Found 152 targets and 83 test targets...
INFO: Elapsed time: 20.273s, Critical Path: 5.04s
INFO: 23 processes: 1 internal, 8 linux-sandbox, 14 local.
INFO: Build completed successfully, 23 total actions
//gutil:collections_test                                        (cached) PASSED in 0.2s
//gutil:io_test                                                 (cached) PASSED in 0.7s
//gutil:proto_matchers_test                                     (cached) PASSED in 0.3s
//gutil:proto_test                                              (cached) PASSED in 0.2s
//gutil:status_matchers_test                                    (cached) PASSED in 0.2s
//gutil:table_entry_key_test                                    (cached) PASSED in 0.0s
//p4_pdpi:ir_tools_test                                         (cached) PASSED in 0.4s
//p4_pdpi/netaddr:ipv4_address_and_network_address_test         (cached) PASSED in 0.2s
//p4_pdpi/netaddr:ipv6_address_test                             (cached) PASSED in 0.2s
//p4_pdpi/netaddr:mac_address_test                              (cached) PASSED in 0.2s
//p4_pdpi/packetlib:packetlib_fuzzer_test                       (cached) PASSED in 15.4s
//p4_pdpi/packetlib:packetlib_test                              (cached) PASSED in 0.1s
//p4_pdpi/packetlib:packetlib_test_runner                       (cached) PASSED in 0.1s
//p4_pdpi/packetlib:packetlib_unit_test                         (cached) PASSED in 0.8s
//p4_pdpi/string_encodings:bit_string_test                      (cached) PASSED in 0.2s
//p4_pdpi/string_encodings:byte_string_test                     (cached) PASSED in 0.2s
//p4_pdpi/string_encodings:decimal_string_test                  (cached) PASSED in 0.1s
//p4_pdpi/string_encodings:decimal_string_test_runner           (cached) PASSED in 0.0s
//p4_pdpi/string_encodings:hex_string_test                      (cached) PASSED in 0.0s
//p4_pdpi/string_encodings:hex_string_test_runner               (cached) PASSED in 0.1s
//p4_pdpi/string_encodings:readable_byte_string_test            (cached) PASSED in 0.2s
//p4_pdpi/testing:helper_function_test                          (cached) PASSED in 0.3s
//p4_pdpi/testing:info_test                                     (cached) PASSED in 0.1s
//p4_pdpi/testing:info_test_runner                              (cached) PASSED in 0.0s
//p4_pdpi/testing:main_pd_test                                  (cached) PASSED in 0.0s
//p4_pdpi/testing:mock_p4_runtime_server_test                   (cached) PASSED in 0.3s
//p4_pdpi/testing:packet_io_test                                (cached) PASSED in 0.2s
//p4_pdpi/testing:packet_io_test_runner                         (cached) PASSED in 0.0s
//p4_pdpi/testing:rpc_test                                      (cached) PASSED in 0.1s
//p4_pdpi/testing:rpc_test_runner                               (cached) PASSED in 0.1s
//p4_pdpi/testing:sequencing_test                               (cached) PASSED in 0.4s
//p4_pdpi/testing:sequencing_test_runner                        (cached) PASSED in 0.0s
//p4_pdpi/testing:table_entry_gunit_test                        (cached) PASSED in 0.3s
//p4_pdpi/testing:table_entry_test                              (cached) PASSED in 0.1s
//p4_pdpi/testing:table_entry_test_runner                       (cached) PASSED in 0.4s
//p4_pdpi/utils:annotation_parser_test                          (cached) PASSED in 0.2s
//p4_pdpi/utils:ir_test                                         (cached) PASSED in 0.3s
//p4rt_app/p4runtime:ir_translation_test                        (cached) PASSED in 0.4s
//p4rt_app/p4runtime:p4info_verification_schema_test            (cached) PASSED in 0.4s
//p4rt_app/p4runtime:p4info_verification_test                   (cached) PASSED in 0.4s
//p4rt_app/p4runtime:packetio_helpers_test                      (cached) PASSED in 0.5s
//p4rt_app/sonic:app_db_acl_def_table_manager_test              (cached) PASSED in 0.4s
//p4rt_app/sonic:app_db_to_pdpi_ir_translator_test              (cached) PASSED in 0.4s
//p4rt_app/sonic:packetio_impl_test                             (cached) PASSED in 0.3s
//p4rt_app/sonic:packetio_port_test                             (cached) PASSED in 0.3s
//p4rt_app/sonic:response_handler_test                          (cached) PASSED in 0.7s
//p4rt_app/sonic:state_verification_test                        (cached) PASSED in 0.2s
//p4rt_app/sonic/adapters:fake_sonic_db_table_test              (cached) PASSED in 0.1s
//p4rt_app/tests/lib:app_db_entry_builder_test                  (cached) PASSED in 0.0s
//p4rt_app/utils:event_data_tracker_test                        (cached) PASSED in 0.0s
//p4rt_app/utils:table_utility_test                             (cached) PASSED in 0.3s
//sai_p4/instantiations/google:clos_stage_test                  (cached) PASSED in 0.2s
//sai_p4/instantiations/google:fabric_border_router_p4info_up_to_date_test (cached) PASSED in 0.0s
//sai_p4/instantiations/google:middleblock_p4info_up_to_date_test (cached) PASSED in 0.0s
//sai_p4/instantiations/google:sai_p4info_fetcher_test          (cached) PASSED in 0.3s
//sai_p4/instantiations/google:sai_p4info_test                  (cached) PASSED in 0.5s
//sai_p4/instantiations/google:sai_pd_proto_test                (cached) PASSED in 0.0s
//sai_p4/instantiations/google:sai_pd_util_test                 (cached) PASSED in 0.2s
//sai_p4/instantiations/google:union_p4info_up_to_date_test     (cached) PASSED in 0.1s
//sai_p4/instantiations/google:wbb_p4info_up_to_date_test       (cached) PASSED in 0.1s
//sai_p4/tools:p4info_tools_test                                (cached) PASSED in 0.3s
//p4rt_app/event_monitoring:app_state_db_port_table_event_test           PASSED in 0.5s
//p4rt_app/event_monitoring:app_state_db_send_to_ingress_port_table_event_test PASSED in 0.8s
//p4rt_app/event_monitoring:config_db_node_cfg_table_event_test          PASSED in 0.8s
//p4rt_app/event_monitoring:config_db_port_table_event_test              PASSED in 0.8s
//p4rt_app/event_monitoring:state_verification_events_test               PASSED in 0.8s
//p4rt_app/sonic:app_db_manager_test                                     PASSED in 0.4s
//p4rt_app/sonic:vrf_entry_translation_test                              PASSED in 0.4s
//p4rt_app/tests:acl_table_test                                          PASSED in 0.8s
//p4rt_app/tests:action_set_test                                         PASSED in 0.6s
//p4rt_app/tests:api_access_test                                         PASSED in 0.6s
//p4rt_app/tests:arbitration_test                                        PASSED in 0.6s
//p4rt_app/tests:fixed_l3_tables_test                                    PASSED in 1.7s
//p4rt_app/tests:forwarding_pipeline_config_test                         PASSED in 2.1s
//p4rt_app/tests:grpc_behavior_test                                      PASSED in 5.0s
//p4rt_app/tests:p4_constraints_test                                     PASSED in 0.2s
//p4rt_app/tests:p4_constraints_test_runner                              PASSED in 0.1s
//p4rt_app/tests:p4_programs_test                                        PASSED in 0.8s
//p4rt_app/tests:packetio_test                                           PASSED in 2.9s
//p4rt_app/tests:port_name_and_id_test                                   PASSED in 0.7s
//p4rt_app/tests:response_path_test                                      PASSED in 1.3s
//p4rt_app/tests:role_test                                               PASSED in 0.6s
//p4rt_app/tests:state_verification_test                                 PASSED in 0.8s
Executed 22 out of 83 tests: 83 tests pass.
There were tests whose specified size is too big. Use the --test_verbose_timeoutINFO: Build completed successfully, 23 total actions

@kishanps, Could you please review.
Thanks.


gradle: Fix executable bit

This fixes the gradle script not being marked as executable for Linux and macOS systems.

Since gradlew is a large file + executable, GitHub won't display the changes. I don't want to merge (just out of potential safety) unless I can see what was changed... I'll approve if I can see what was changed.
I know its probably safe since other people have merged your PRs, I'm also just curious as to what was changed since I might start doing this in the future.


Tabbing in snippets is broken

Seem to have traced this https://github.com/atom/snippets/issues/53 down to the emmet package. When I disable the package tab stops work as expected.

Want to back this issue? Place a bounty on it! We accept bounties via Bountysource.

Maybe related to #146.


Additional platforms

Hi! This plugin is incredibly powerful, thank you for your hard work on it. Do you have plans to develop support for other platforms? PS5/XBOX? How difficult would it be, if feasible at all?

Thanks a lot !
I do not plan on supporting consoles but I'm pretty sure it would "just work" if you tried (with maybe some minor modifications). The code is console and API agnostic, you just need a performant GPU.
Hope this helps, let me know if you make it work.