feat(collector): add additional ports and explicitly define OTLP endpoints

This PR:

enables Jaeger compact thrift protocol to serve as drop in replacement for the Jaeger Agent

Hey @secustor, thank you for fixing it! Can you please keep only OTLP fix in this PR and bump version to 0.7.2 to keep atomic updates?
About thrift compact. It doesn't seem to be a popular format. To be honest I'm not convinced that we should add it by default. Do you use if yourself?

@dmitryax I have made a break out PR for the endpoint fixes.
Regarding thrift compact. This is the protocol which Jaeger instrumentations use to communicate with their node agent.
As Jaeger is deprecating their own Agent in favor of the OTL Collector it makes sense IMO to enable this by default, to give such user a better migration experience.  https://www.jaegertracing.io/docs/1.27/deployment/#agent
That way their integrations simply "work" instead of looking up the implementation details of the collector and jaeger

@secustor Ok, sounds good. Let's add it by default. Could you please update this PR?


Thanos Receive memory usage spikes extremely high after restart (9GB before restart, 93GB after)

Thanos, Prometheus and Golang version used:
bin/thanos --version
thanos, version 0.10.1 (branch: HEAD, revision: bdcc35842f30ada375f65aaf748a104b43d56672)
  build user:       circleci@4e51e880cd24
  build date:       20200124-07:36:32
  go version:       go1.13.1

Object Storage Provider: MinIO
What happened: After a restart of the VM running the Thanos Receive component, the memory of my Thanos Receive exploded from around 9GB (based on ps and go_memstats_sys_bytes stats) to 93GB!
What you expected to happen:  Thanos Receive should be able to restart using the same amount of memory it was using before the restart.
How to reproduce it (as minimally and precisely as possible):  Run a relatively large workload and restart Thanos.  Watch the memory usage.
Full logs to relevant components:
Anything else we need to know:
Originally my VM's memory size was only 32GB.  Thanos Receive OOM'ed after a restart, so I increased to 64GB on the VM.  Thanos Receive OOM'ed again, so I went to 128GB.  The other processes on the VM (MinIO, Thanos Store/Compact/Query, Prometheus) are contributing < 1GB to the system memory usage.
I have 12 vCPUs on this system.
I am running a scale test workload that generates 2.19 million datapoints each minute.  The workload has the ability to add cardinality to the time series.  However, in order to rule that out, I have been running with a static 2.19 million time series (metric name/label combinations).  One datapoint per time series each minute.  I cleared out my test environment to ensure that old time series were not polluting the test.  My original theory that cardinality of time series fluctuating was causing the issue (perhaps replaying back everything and loading all of the time series in its history) was incorrect based on this.

During the test, the go_memstats_alloc_bytes fluctuates between 4 and 8GB.  However, after the restart it exploded to 88GB.  While the internal allocation does reset back to the 4-8GB range, the system RSS go_memstats_sys_bytes does not (not a surprise).  It remains around 93GB.

I have some additional script automation to monitor system stats with ps, iotop, sar etc.  From this I can see that the CPU usage of Thanos Receive was very high.  The column CPU % is the percentage of a single CPU core (so 1081% is 10.81 cores).  Using iotop I can see that the majority of the disk IO was a large amount of writes by Thanos Receive (P-Read and P-Write are KB of writes per minute).
Thanos Receive system statistics:

MinIO did a large amount of work for a few minutes at the end of Thanos Receives' startup.  Mostly disk writes (around 23GB of them).
MinIO system statistics:

Environment:

OS (e.g. from /etc/os-release): Red Hat Enterprise Linux Server release 7.7 (Maipo)
Kernel (e.g. uname -a): 3.10.0-1062.el7.x86_64 #1 SMP Thu Jul 18 20:25:13 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Hi @rdzimmer, unfortunately this is currently a known issue in Thanos receive. This occurs whenever the WAL must be replayed to write a whole block to disk. Normally this occurs during a restart if there is an incomplete block. I think we might have another open issue on this topic, so maybe we can consolidate te discussion. Nevertheless, it is an implementation issue in the Prometheus library we are using to read the WAL. We are actively investigating, as there’s no reason that a restart should be the limiting factor in the system :p. We’re also happy to review a contribution :)
-Lucas

Thanks @squat!  Looks like https://github.com/thanos-io/thanos/issues/1624 is probably the duplicate you're referencing.  Not sure how I missed it yesterday (once I added "WAL" to the search it was obvious).  Happy to close this and keep using that one.  I'll see if there's any contribute I or my team can make.

One thought I just had, I'm guessing the reason I saw mostly writes and not reads was because in this case I didn't reboot the VM, just Thanos Receive.  So the data was most likely still in system file cache.  I'll setup another test and do a full reboot to confirm.  Not sure if that really helps things, but at least explains the results better.

Experiencing the same problem. After the restart the memory consumption went from 6Gi to 47Gi and then crashes due to insufficient memory on the node.

I'm facing the same problem. After restart the memory went from 14Gi to 64Gi and then also crashed  due OOMkilled by system.


Date/time metadata in some photos not being read in uploader

A user says that starting on Nov 22 or so, their photos' date/time are not being read when using /observations/upload. I can replicate with this photo of theirs.
FWIW, they say this only happens with photos they have cropped.

I suspect 8f120dbf48683a56340fb7d25de351321e85bc0f may have fixed this. Can you still replicate it?

No, it's working now.


Update hf_decoder_model.py

remove explicit target_modules; PEFT would detect it automatically.

see issue in #125

Hi Pingchuan,
Thank you for your contribution!


Add configuration for waits/sleeps (linear or exponential) between retries

Using elasticsearch-net in our Elasticsearch Client.
We are receiving intermittent MaxRetryExceptions that are not exposing any InnerExceptions (is set to null). We know that these retries are likely due to a network issue that is beyond our control anyway.
Instead of returning nothing for these attempted requests, we would rather return something after ~1 second (worst case).
Currently, elasticsearch-net retries up to our MaxRetries (10) in less than 200ms. We would like the retries to have a configurable back-off to allow for the network issue to correct before throwing MaxRetryExceptions. This way we have the option to wait between retries to fit our needs.

Hey @mjohnst thanks for opening this issue, will let it bake over the weekend and think about this.
My first initial reaction is that this kind of logic might be best handled in the most opaque fashion around the client call in your application layer using something like Poly which has support for this baked in.
Retries are really meant to retry on different nodes in the cluster but still fail as fast as possible.

Hi @Mpdreamz thanks for the response!
We are relatively confident that the error causing these retries is at the elasticsearch-net layer of the application since the requests are failing in a matter of milliseconds (we'd expect that a network level failure would take longer (x10 for each retry)).
Given this, we would prefer a option to set in elasticsearch-net for the back-off over using something like Poly that would encapsulate our requests, but I appreciate you guys thinking about it :)

Can you tell me a bit about your connection pool and how its set up? Are you using sniffing?
If you are not seeing any innerexceptions my first gut feeling is that the client is seeing 503 or 502 responses could this be the case?

I've added additional logging to the RetryRequestAsyc method to hopefully get some more information, but the only message we get is TraceSource="w3wp.exe Could this be a 502 or 503 surfacing?
I also added a increasing wait during retries (100ms * retry count), but the error persisted to our max retry count with these waits. This occurs intermittently for a few hours then not at all for up to a few days.
We are using sniffing for our connections.
Here is our code setting up the connection pool. Let me know if any other details would help.
private bool UpdateEndpoints(ElasticClient seedClient)
{
    try
    {
        var uris = this.GetHttpEndpoints(seedClient).ToList();

        if (uris.Count == 0)
        {
            Trace.TraceError("Unable to retrieve any IPs from seed endpoints");
            return false;
        }

        if (uris.All(uri => this.clientUris.Contains(uri)) && this.clientUris.All(clientUri => uris.Contains(clientUri)))
        {
            return true;
        }
        else
        {
            Trace.TraceInformation("Client endpoints updated");
            this.clientUris.Clear();
            foreach (var uri in uris) this.clientUris.Add(uri);
        }

        var pool = new StaticConnectionPool(uris);
        var settings = new ConnectionSettings(pool);
        if (this.enableDebugInfo)
        {
            settings.EnableMetrics(true);
            settings.ExposeRawResponse(true);
        }
        settings.MaximumRetries(this.numRetries);
        settings.SetDefaultPropertyNameInferrer(p => p);
        this.elasticClient = new ElasticClient(settings);

    }
    catch (Exception ex)
    {
        Trace.TraceError("Unable to update Elasticsearch endpoints: {0}; retrying", ex.ToString());
        return false;
    }
    finally
    {
        // Trigger next update
        if (this.updateEndpointsTimer != null)
            this.updateEndpointsTimer.Change(this.updateEndpointsInterval, Timeout.InfiniteTimeSpan);
    }
    return true;
}

HI @mjohnst,
I'm interested to see your GetHttpEndpoints() method, we also have a SniffingConnection that will sniff each node until one responds with the current cluster information.
http://nest.azurewebsites.net/elasticsearch-net/cluster-failover.html
Setting SniffOnStartup will fetch all the nodes from the cluster from the seed nodes you pass in the constructor. Setting SniffOnConnectionFault will resniff every time a failed request happens
You can also set SniffLifeSpan to resniff every this.updateEndpointsInterval.
We are planning to make the sniff itself fully customizable in 2.0 but this is something we could totally backport in a 1.8 release.

Since our nodes have data, master, and http endpoints, we use this method to filter our node endpoints to just http.
Here is our GetHttpEndpoints method:
private IEnumerable<Uri> GetHttpEndpoints(ElasticClient seedClient)
{
    var nodesInfo = GetElasticsearchNodesInfo(seedClient);

    if (nodesInfo != null)
    {
        var uris =
            nodesInfo.Values
                .Where(
                    n =>
                        n.Settings.ContainsKey("node") &&
                        n.Settings["node"].Value["http"] == "true" &&
                        n.HttpAddress.StartsWith("inet[/") &&
                        n.HttpAddress.EndsWith("]"))
                .Select(n => "http://" + n.HttpAddress.Substring(6, n.HttpAddress.Length - 7));

        foreach (var uriString in uris)
        {
            Uri uri;
            if (Uri.TryCreate(uriString, UriKind.Absolute, out uri))
            {
                yield return uri;
            }
        }
    }
}

public virtual Dictionary<string, NodeInfo> GetElasticsearchNodesInfo(ElasticClient seedClient)
{
    if (seedClient != null)
        return seedClient.NodesInfo().Nodes;
    else
        return null;
}

Hey @Mpdreamz, thought I'd give an update to where we are and see if you had any suggestions to help us further expose what's going wrong in our client.
We tried to mitigate the MaxRetryExceptions by switching from sniffing to static (using the default Azure load balancer). Since then, we've stopped seeing the MaxRetryExceptions, but have been seeing 503 return codes in their place. You mentioned above that you suspected this could be the root cause. Do you have any thoughts on how we could further diagnose this or expose more information about what's causing the 503? (we've also contacted our Azure support to see if they can help)
Thanks!

Thanks for the update @mjohnst, much appreciated.
Elasticsearch will throw a 503 if the node you are currently talking too can't find its master, It used to also throw a 503 on threadpool rejections (too many requests) but since mid 2014 ES switched 429 for that.
https://github.com/elastic/elasticsearch/issues/6627
Another reason for the a 503 could be when all shards are in a transient recovery state and none can handle a search request (very rare).
Or it could be the azure loadbalancer or another piece of infra.
On ConnectionSettings there is a SetConnectionStatusHandler that you can pass a Func that will get called each time a good or bad response is being created. You could use that to specifically listen for 503's and log them so we can see what happens? Be sure to also set KeepRawResponses on Connectionsettings so that you can see it on response.ConnectionStatus

Adding some logging via the SetConnectionStatusHandler today. I'll let you know if we expose any more details.

We were able to get a bit more information from the logging.
We can now see this exception is part of the problem.
The IElasticsearchResponse RawResponse:
{"error":"ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];]","status":503}

If you have any ideas about this, I'd be happy to hear :)

Hey @mjohnst are you able to locate the full elasticsearch exception from the elasticsearch log itself?

We had to add some more logging to narrow down what node it was happening on, but we did find the root cause to why the node was unavailable in its logs:
failed to send join request to master...
reason...RemoteTransportException [internal:discovery/zen/join]]...
nested: NotSerializableTransportException[org.elasticsearch.transport.ConnectTransportException]

Restarting the node's machine fixed this issue (we had trouble restarting just elasticsearch on the node).
This seems like an issue beyond Elasticsearch-Net, but it would be a great mitigation if our requests would retry on a different node instead of hitting a similarly "dead" node until max retries is hit.
Do you know if there's a mirror for this issue https://github.com/elastic/elasticsearch/issues/11709 in NEST/ES.Net ElasticsearchClients. If so, do you know whereabouts we could make changes in the code to force selection of a new node when there's a 503 or ClusterBlockException?

Hi @mjohnst sorry for the disconnect on this issue.
NEST already retries on 503, we have quite extensive unit tests in place to see that we do. Here is one where we test a ping succeeding but the subsequent call returning in a 503 so we should fallover
So I'm still wondering what sequence of events cause a 503, Can you share your connection settings?

I don't think there's any issue with NEST not retrying on 503s at all, but potentially not retrying on different nodes (like issue I referenced in my last post). We were definitely seeing the same node being hit 10 times (our max retry) in a row with 503s returning each time.
The 503 in our case looks like it was just surfaced from the node being unable to communicate with the master. We're really sure how this happened, but restarting the node has fixed it.
Our connection settings are pretty much the defaults. We have just a few settings configured:
settings.SetDefaultPropertyNameInferrer(p => p);
settings.EnableMetrics(true);
settings.ExposeRawResponse(true);
settings.MaximumRetries(numRetries); # 10 from XML below

Some more settings in XML that we read:
<Setting name="LatencyThreshold" value="5000" />
<Setting name="EsUpdateIntervalInSeconds" value="120" />
<Setting name="EsMaxRetries" value="10" />

Hi @mjohnst closing this issue for now.
If you are still having issues please reopen a new ticket and we would love to help you get to the bottom of this.
NEST 2.0 will have a lot of improvements with regards to testability and exposure of what goes on under the covers.


Custom Epic User Intro Sounds

Custom Intro Sounds

Specific Media Folder
FileName = Twitch Name
Detect first text in chat
Plays matching sound
Some limit to sound length (slightly longer than !shock, e.g. 2 sec)
User has to submit via PR

Stretch goals

Command to trigger after

Only by user !myintro

Outros also

{name}_intro.wav and {name}_outro.wav
!myintro !myoutro

Admin/Mod cmds

!user_{name}_{intro/outro}

Didn't get a chance to talk through it more during the game stream but here's what I've begun doing to implement functionality for viewer sounds...

Detect when someone joins the chat
Check if they have a join sound defined in the effects configuration
Play the sound

Effects Configuration example: https://github.com/clarkio/ttv-chat-light/blob/master/src/server/effects.json#L72
Code to try and activate a "join" sound: https://github.com/clarkio/ttv-chat-light/blob/master/src/server/twitch-chat.ts#L94
Let me know what you all think. This can also be done for when someone leaves.
The one potential issue I could see with this is if someone just wants to peek into the stream but not really be noticed. I suppose a solution to that is to not have a join/depart sound configured for yourself (assuming people will choose to make a PR to add themselves [used themichaeljolley and roberttables as guinea pigs so far]).

Finished in https://github.com/clarkio/ttv-chat-light/commit/afec9722fa060b301e83c8076a237d7d464fb0bb
Closing


NetworkImage changes headers

The NetworkImage class changes the headers I provided.
More specifically it appended "Dart/2.0 (dart:io), " before what I set as User-Agent. I want my server to only serve requests from my application so I want to enforce my custom User-Agent. Of course I could allow a User-Agent with "Dart/2.0 (dart:io), " as prefix but the problem here is that I don't know if "Dart/2.0 (dart:io), " is going to change soon.

This isn't set by the NetworkImage, but by the Dart sdk's HttpClient.  Presumably this wouldn't change without a bump in the Dart sdk version.
cc @zanderso

@zocker1996  If you want to you can override the HttpClient created by NetworkAsset and NetworkImage using HttpOverrides.  This will give you a change to set the user agent manually using the API mentioned above.

I assume the suggestions solve this issue.


Fix offset in Backends and QueryHandlers

content-browser-sylius uses Pagerfanta since there are only Backends, but layouts-sylius use result from Doctrine Query Builder since there are only QueryHandlers. layouts-sylius-bitbag is a merge between the two - having both QueryHandlers and Backends. I have decided to use getAdapter method because of its simplicity. In versions from 3.2.0 to 3.6.2 it says that the method getAdapter will be deprecated, but the deprecation message was removed in version 3.7.0, current version is 4.2.0 and the getAdapter method still exists without any deprecation messages. getSlice method has the same effect as creating a new query with groupBy(entity.id)->setFirstResult(x)->setMaxResults(y) (groupBy is necessary because of the multiple joins happening across the query). So instead of creating new queries and updating a lot of code for the same effect seemed dubious.

Decided to go with the query builders because when using Pagerfanta the error is thrown when three conditions are met: limit, orderBy and fetchToMany association. The error thrown is "Cannot select distinct identifiers from query with LIMIT and ORDER BY on a column from a fetch joined to-many association. Use output walkers." And the output walkers are set to false by default because of the performance issues -  see here

Thanks @igorkrz !


InfluxDB Cloud: reports

Proposal:
Reports that can be sent via different endpoints on specified schedule automatically. Reports can be in different formats and contain different data. Examples:

screenshot of dashboard cells within specified time range can be sent to specified email address
result of Flux query can be sent to HTTP endpoint in JSON format

It's different than alerts. Alerts are sent on some conditional. But reports are supposed to be sent without conditions, by schedule.
New feature can be placed in renamed Alerts section like this:

Use case:
Examples of use cases:

report containing image of sales amounts of a company which is sent every Friday to specified email addresses
JSON containing bucket data is sent to existing REST API as body of a request

@melanchall thank you!


[support request] : when to get things ready

Hi, thanks for the library.
When following examples, this works pretty easy. The near rendering is pretty neat 👍
But as soon as I needed some undocummented uses, it becomes a bit less easy.
I need to retrieve the Mesh inside a Text, but this is not 'instantaneous'
let newText = new ThreeMeshUI.Text({
        content: "I'm a text!",
      });

console.log("New Text ",newText.children.length); // no children created yet

// Even after trying somethings that would'nt make sense ( i guess )
newText.updateLayout();
console.log("New Text ",newText.children.length); // nope, still 0 children

I ended up using some vicious ways to finally get it
let newText = new ThreeMeshUI.Text({
        content: "I'm a text!",
      });

requestAnimationFrame( ()=>{
    console.log( newText.children.length ); // Ah, finally get it !

But I would like to better understand how things works, before generalising this nextFrame way into my code.
Thanks !

Hi, thanks for you interest in the lib !
There is a callback called after any update, it's component.onAfterUpdate, which you can also use on Text components.
See this fiddle : https://jsfiddle.net/felixmariotto/L8z1c4wj/
Out of curiosity, why do you need to get the characters meshes ?

Thx for the fiddle.
It looks a bit weird, the onAfterUpdate runs at least two times, but sometimes it goes to up to three times.
Children are not set in the first onAfterUpdate but usually on the second.

It was on chrome at least.
I need to retrieve the Mesh for its usual meshes properties.
The text I create must go on a specific Layer/Selection on a specific .renderOrder
In my current understanding of Three, .renderOrder of THREE.Object3D do not cascade the value to each of its children. So I need to explicitely tell each Mesh need to go on which layer at which order.
I also animate the material opacity property, which is not instantly created too.
By the way, Im unable to animate opacity from the Material property newText.fontMaterial.opacity. Even with .uniformsNeedUpdate=true.
I finally worked around using uniforms properties newText.fontMaterial.uniforms.u_opacity.
I might be completely out of scope of the library, even I really really like all the layouting possibilities its offers, I just need to place titles ( as plane ) into a 3d world for now. And perform actions just as I would do on simple planes. I used another library before, but it lacks maintenance. So I give this library a try.

It looks a bit weird, the onAfterUpdate runs at least two times, but sometimes it goes to up to three times.
Children are not set in the first onAfterUpdate but usually on the second.

onAfterUpdate is called everytime the component is updated. When you instantiate the component, it's updated. When you update the component's parent's size, it's updated. Lastly, when the font is loaded, three-mesh-ui knows how to layout the content, so it's updated. That's why the text layout is not instantanuous, how could it be computed before to load the font data ?

I also animate the material opacity property, which is not instantly created too.
By the way, Im unable to animate opacity from the Material property newText.fontMaterial.opacity. Even with .uniformsNeedUpdate=true.
I finally worked around using uniforms properties newText.fontMaterial.uniforms.u_opacity.

Seems to work well on my side, here is a fiddle : https://jsfiddle.net/felixmariotto/2kwbxcL4/24/

Also setting the mesh layer was the very reason onAfterUpdate was introduced, see #70


VC Admin UI- Create New VC's

As an admin ,
I want to create list of VC's
so that the engagement has all the VC even if the downstream data fetch from EPICFLOW is not implemented

MA + Carly need to talk to Geoff. MA wants to know she is working from the right list so her wireframe is mostly correct.

What do we need to make the table look like?


[Feature request] Variable results as conditions

It would be nice to have the ability to get a result from a variable and use it as a condition for another. Like this:{icon:[targeted_block_id]}. I've tried to do this with conditionals but nothing has worked, and the docs don't have anything on this.

I have ideas for that, but it probably won't be next update. Though target block, fluid, and entity icons would be good additions (can't believe it's not already a thing)

Target block/fluid icons have been added in the v4 beta (entities are kinda hard though)


Panic handling: Report and halt.

As discussed on the #6946 call, we should do two things differently in event of a panic:

Ensure the application stops running, as continued execution is UB and can cause the appearance of strange bugs.
Report the failure to the user.

Both of these things could be achieved by altering our panic hook to navigate to a new "crash" page. Note that this API should not attempt to pass information to the JS handler, as it is possible that state is so corrupted that we cannot export heap-allocated objects to JS correctly.

But I understand that the "crash page" can have info about the stack trace from Rust, right ? Ofc not displayed by default, but displayed after clicking "details" or something like that.

Merging with #7376


wink-mqtt.js will not run - Aborted

So I ran npm install, copied all the files over, and modified the init. I noticed it wasn't running after the reboot so I manually started it with the script. Still wasn't running after that. I manually started it with "node wink-mqtt.js" while I was in the directory and it said 'Aborted' and quit out. I threw some console prints in there and it's executing them. I have zero experience with Node unfortunately, so I can't do much debugging. Any ideas?

Oh I forgot to add - I'm on 2.66 firmware, and I did modify the IP address of the broker to be correct.

Haven't tested on 2.66 yet, will give it a shot and get back to you. They
may have moved around somethings with the new version. If you can get me
more debug information from the log saved at /var/log/mqtt.log
On Tue, Aug 2, 2016 at 10:52 AM bazhip<EMAIL_ADDRESS>wrote:

Oh I forgot to add - I'm on 2.66 firmware, and I did modify the IP address
of the broker to be correct.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/danielolson13/wink-mqtt/issues/9#issuecomment-236929292,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEs5GKZP-ibLaKjcnPdR83SYKP8JOuW8ks5qb1mYgaJpZM4JasI0
.

 Turns out it is running (doh) and trying to spawn it again gave the "Aborted".  I didn't see it in 'top' so I thought it wasn't. Double checked with ps and there it is.
I was rather brief in my initial post. What lead me to not thinking it was running and trying to start it again was that no messages are getting published on state changes. I had this up and running nicely on 2.49, but long story short I wasn't using it for quite some time, I upgraded and reinstalled today to get it going again.
I've done some more digging and here is what I found:
On the initial boot, it is publishing out the initial states (although uncorrectly. Some things that are off are publishing as on, and some that are on are publishing as off)
Your program is correctly tailing the log file and checking the database when it reads there is a state change. What's interesting though is that it is not getting the correct results back from the query (I believe). I uncommented the console.log where it prints out the values from the query, and they are all the same as the initial publish. Since they are always the same, it is never publishing out the values.
When I publish TO it ( mosquitto_pub -t 'home/4/1/set' -m 'ON' ) it fires everything off and the database query comes back as:
runApron() -u,-m,4,-t,1,-v,ON
PUBLISHING: home/4/1v: ON
Publish status home 4 1
state change
checkdb()
home/4/1:ON:OFF
different#############################################
PUBLISHING: home/4/1v: OFF
home/4/2:254:254
home/5/1:OFF:OFF
home/5/2:254:254
(I added some extra debugging)
Then when I turn a DIFFERENT light off, it goes back to OFF
state change
checkdb()
home/4/1:OFF:OFF
home/4/2:254:254
home/5/1:OFF:OFF
home/5/2:254:254
So the status is only changing when I push an update to it through MQTT, and it is only like that temporarily. It never changes the values when I do it through the app.
I wonder if something changed in the database? Maybe it's just something with my hub? It is reading correctly in the wink app though.

Thanks for digging into this deeper. My solution is by no means perfect. If I remember right the database stores a GET and SET value for each attribute value. The program may be using the wrong value from the database, i.e. GET when it should be SET. Can you see the values in the database change when you turn a device on and off with the app?
Now that 2.66 has "local control" it may be better to look at how Wink is implementing that and add MQTT support on top of that.
The reason I originally wrote this (and probably why you're trying to get it to work) is because I was looking for faster readings from sensors and lights to be more responsive. The API based HomeAssistant solution is getting better, but not quite instantaneous like local control with the wink app.

Yeah, I might just have to sit down and learn node myself :( Unfortunately I'm running OpenHAB2 rather than HomeAssistant. I might give that a try though.
Anywho, no I can't see it updating. I noticed that the database file /var/lib/databases/apron.db hadn't been modified in about an hour as well. I'm assuming that was the last time your program had touched it.
I did a find -name *.db on the filesystem and I found a /tmp/database/apron-2.db that does update when I make changes.
I pointed your program at the other database and it is correctly updating now, but I feel like using the tmp database is not the correct solution...

Interesting, good investigative work. They may have changed the database
location again, it changed from 2.19 to 2.49. I think they were trying to
reduce the writes to the flash memory. Storing the database in tmp or RAM
would extend the life of the flash memory. Just looked at my /tmp/database
directory and I have apron-14.db, apron-15.db and apron-16.db. After a
reboot I only have apron-2.db. Once that database grows it must create a
new file.
Need to write some code to reference the most recent or highest number
database file for 2.66 firmware
I don't really know node either, but I know enough javascript and google-fu
to be dangerous.
On Tue, Aug 2, 2016 at 1:30 PM bazhip<EMAIL_ADDRESS>wrote:

Yeah, I might just have to sit down and learn node myself :( Unfortunately
I'm running OpenHAB2 rather than HomeAssistant. I might give that a try
though.
Anywho, no I can't see it updating. I noticed that the database file
/var/lib/databases/apron.db hadn't been modified in about an hour as
well. I'm assuming that was the last time your program had touched it.
I did a find -name *.db on the filesystem and I found a
/tmp/database/apron-2.db that does update when I make changes.
I pointed your program at the other database and it is correctly updating
now, but I feel like using the tmp database is not the correct solution...
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/danielolson13/wink-mqtt/issues/9#issuecomment-236979304,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEs5GNEuqhNLW-ehJqDrQtvBIq5CXJODks5qb37BgaJpZM4JasI0
.

Are you saying you had 14, 15, and 16 on 2.66 or is it still there on 2.49?

Same issue here; upgraded to 2.66 today and wink-mqtt stopped working. After troubleshooting for several hours, have decided to give it a rest for a bit. Daniel, any help would be appreciated.

FYI, below are my databases:

Yeah, follow the comment thread. Modify the script and point the database path at the one in /tmp/ They moved it again in 2.66. looks like they auto increment as well once it gets too big. The script needs to be modified to look for the database with the largest number, or do what I do and when a new database is created, just reboot lol

Yep, that works. Did that after my previous post. the one in /tmp (currently apron-2.db) is working. I may create a timer to reboot the hub every day or so (not sure if that is enough). How often are you rebooting?

Iunno, maybe once a week? If you do it daily at some freakish hour of the night, you should be fine. I just don't have time to learn node right now to modify the script.

So I changed something in /opt/wink-mqtt/wink-mqtt.js file and something happened. I changed this:
// ******* apron database location for 2.19 firmware ********
var aprondatabase = '/database/apron.db';
// ******* apron database location for 2.49 firmware ************
var aprondatabase = '/var/lib/database/apron.db';

to this:
// ******* apron database location for 2.19 firmware ********
var aprondatabase = '/tmp/database/apron-2.db';
// ******* apron database location for 2.49 firmware ************
var aprondatabase = '/tmp/database/apron-2.db';

Now when I restart the Wink, the apron-2.db file is not created, and my Hub doesn't connect to the cloud API even though it's not disabled in the Hosts file.
When I run aprontest -l I get:
[root@flex-dvt ~]# aprontest -l
connect() failed: No such file or directory
connect() failed: No such file or directory
Found 11 devices in database...
Couldn't get devices

Found 1 ZIGBEE groups in database...
GROUP ID |             NAME |
       2 |  IrjUh1pQjQS1MSo |
[root@flex-dvt ~]# nano /opt/wink-mqtt/wink-mqtt.js
[root@flex-dvt ~]# aprontest -l
connect() failed: No such file or directory
connect() failed: No such file or directory
Found 11 devices in database...
Couldn't get devices

Found 1 ZIGBEE groups in database...
GROUP ID |             NAME |
       2 |  IrjUh1pQjQS1MSo |

What did I do and how can I undo it?

Issue fixed


Stop buildinging NewtonSoft.Json for NetStandard1.*

Related to https://github.com/dotnet/source-build/issues/4482
Test Build (Microsoft internal link)

I will wait to merge until stage 2 of the test build finishes even though stage 1 should be sufficient.


Mapbox: implement predefined user location

Mapbox predefined user location display is used by some applications to
automatically show user location on maps.
The two required methods from GoogleMap API are isMyLocationEnabled and
setMyLocationEnabled, which are implemented here by using
LocationComponent class.
The initialization must be performed when the map style is loaded.

In Tripadvisor and Park4Night for example, the user location is not displayed. This does not happen in Uber for example. The reason is that these two apps use Mapbox predefined user location display utilities, which is not implemented in GoogleMap.kt
The implementation proposed here works at least for these two apps.
The code should be carefully reviewed though: I had never used kotlin before..

Merged and cleaned up in 981301e9c6e236765d53389b3a7fdb689b352a80


adding cnu.edu.ph domain

@hannahcnu First stage of verification passed, official domain for Cebu Normal University is cnu.edu.ph as far as I verified. This is just an informational note. Pull request is still in the review. I appreciate your patience while waiting for the final decision.

@philipto Thanks

@hannahcnu Thank you for your patience while waiting for a review and processing your request. Pull request merged.
Please start requesting the licenses in about an hour, to let the changes to propagate through our system.


solo.sendKey(KeyEvent.KEYCODE_HOME); does not take the application to Home screen

From<EMAIL_ADDRESS>on January 20, 2012 04:10:32
What steps will reproduce the problem? 1.solo.sendKey(KeyEvent.KEYCODE_HOME); does not take the application to Home screen 2. 3. What is the expected output? What do you see instead? It should take me to home screen. What version of the product are you using? On what operating system? Robotium 3.1, ADT 15 Please provide any additional information below. Renas, In one of the post you said that it is included in Robotium 2.0 but is not working. If you make it working it helps us in our project's recovery system.
Original issue: http://code.google.com/p/robotium/issues/detail?id=213

From<EMAIL_ADDRESS>on January 22, 2012 23:20:14
Unfortunately it is a restriction of the Android platform. Nothing can be done to get around it.
Status: Invalid


Fix Strings support

Fix Strings support #12
Add tests

Looks good. I am going to need some time to test it.

Thank you!


Time format in japanese is not optimal

Time format is not localized in Japanese.
Incorrect: 14:30
Proposed: 14時30分
Code:
    final DateTimeFormatter dateTimeFormatter = new DateTimeFormatterBuilder()
            .append(DateTimeFormatter.ofLocalizedTime(FormatStyle.SHORT))
            .toFormatter(Locale.JAPAN);
    System.out.println(dateTimeFormatter.format(LocalDateTime.now()));

    DateTimeFormatter localizedFormatter = DateTimeFormatter.ofPattern("h:mm", Locale.JAPAN);
    System.out.println(LocalDateTime.now().toLocalTime().format(localizedFormatter));`

Output:
11:31
11:31

I've tested this and threetenbp produces the same output as Java 17. This is probably in line with the definition from CLDR.
Closing as nothing to change.


faster travis builds using their container infra

Using containers[1] & caching the dependant packages (using wheels) so that subsequent
travis builds cache this.
As a result was able to bring down the build time to about 52 seconds[2]
[1] http://blog.travis-ci.com/2014-12-17-faster-builds-with-container-based-infrastructure/
[2] https://travis-ci.org/theanalyst/smart_open/builds/49096656

Thanks, but does 71s (before) vs. 52s (now) spent in Travis really make a difference? Can you give more motivation for this optimization?
I'm not a big fan of fixing things that are not broken. Especially when the fix brings in more complexity (as caching invariably does).
Not to sound ungrateful but I'm afraid I'm -1 on this one. I'll leave the decision to @VincTheSecond.

Yeah atm doesn't seem worth it.. feel free to close this.

Ping @VincTheSecond : either merge or close, don't leave PRs stale.

Thank you for your proposal, but for this small project it is not really necessary. Let's keep things as simple as possible.


Terminal state update error

Hi I have declared an agent like this
agent = PPOAgent(
        states=dict(type='float', shape=(37,)),
        actions=dict(type='int', num_actions=3),
        network=[
            dict(type='dense', size=74),
            dict(type='dense', size=31)
        ],
        batching_capacity=1200,
        step_optimizer=dict(
            type='adam',
            learning_rate=1e-4
        ),
        summarizer=dict(
            directory="./board",
            steps=50,
            labels=['graph',
                'losses',
                'total-loss']
        )
    )

everything work when I do: agent.observe(reward=reward, terminal=False), but the following error shows up on: agent.observe(reward=reward, terminal=True)
InvalidArgumentError (see above for traceback): Must have updates.shape = indices.shape + params.shape[1:] or updates.shape = [], got updates.shape [60,37], indices.shape [35], params.shape [10000,37]
	 [[node ppo/observe-timestep/store/ScatterUpdate (defined at /venv/lib/python3.6/site-packages/tensorforce/core/memories/queue.py:176)  = ScatterUpdate[T=DT_FLOAT, Tindices=DT_INT32, use_locking=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](ppo/initialize/latest/initialize/state-state, ppo/observe-timestep/store/mod, ppo/strided_slice, ^ppo/observe-timestep/store/AssignSub)]]

It seems to be an error with the shapes dimensions. But I haven't managed to understand the logic behind this shape arithmetic: updates.shape = indices.shape + params.shape[1:]
Can you explain it please?

Hi,
The shape arithmetic is from the TensorFlow exception and shows which shapes don't fit together why. A likely cause for this is if you do not call observe() after every call to act(). Is that possible? Otherwise, could you post the code where you interact with the agent (act() and observe())?

That code looks alright. What error did you get with Runner, out of interest? Generally, are you using the newest version of TensorFlow and Tensorforce (GitHub version)?

Hey, I have just replicated the Runner object error.
Traceback (most recent call last):
  File "agent.v1.0.py", line 104, in <module>
    main(e+1)
  File "agent.v1.0.py", line 90, in main
    runner.run(episodes=3000, max_episode_timesteps=200, episode_finished=episode_finished)
  File "/venv/lib/python3.6/site-packages/tensorforce/execution/runner.py", line 100, in run
    action = self.agent.act(states=state, deterministic=deterministic)
  File "/venv/lib/python3.6/site-packages/tensorforce/agents/agent.py", line 148, in act
    independent=independent
  File "/venv/lib/python3.6/site-packages/tensorforce/models/model.py", line 1373, in act
    assert state.shape[0] <= self.batching_capacity
IndexError: tuple index out of range

The weird thing here is that I have a "debug" message in my env.get_state function, showing the shape of the state, but it never was printed. As if this assert state.shape[0] <= self.batching_capacity was called before the first env.get_state call. Also I modify the model.py and tried to print the cuestioned shape but I got (). Here's the source using Runner (with the example code provided in the website)
agent = PPOAgent(
        states=dict(type='float', shape=(37,)),
        actions=dict(type='int', num_actions=3),
        network=[
            dict(type='dense', size=74),
            dict(type='dense', size=31)
        ],
        batching_capacity=1200,
        step_optimizer=dict(
            type='adam',
            learning_rate=1e-4
        ),
        summarizer=dict(
            directory="./board",
            steps=50,
            labels=['graph',
                'losses',
                'total-loss']
        )
    )

    try:
        agent.restore_model("./checkpoints/")
    except Exception as e:
        logging.info("> error restoring model:"+str(e))

    # Create the runner
    runner = Runner(agent=agent, environment=env)

    # Callback function printing episode statistics
    def episode_finished(r):
        print("Finished episode {ep} after {ts} timesteps (reward: {reward})".format(ep=r.episode, ts=r.episode_timestep,
                                                                                reward=r.episode_rewards[-1]))
        return True

    # Start learning
    runner.run(episodes=3000, max_episode_timesteps=200, episode_finished=episode_finished)
    runner.close()

    # Print statistics
    print("Learning finished. Total episodes: {ep}. Average reward of last 100 episodes: {ar}.".format(
        ep=runner.episode,
        ar=np.mean(runner.episode_rewards[-100:]))
    )

I'm using the pip repository version. Let me pull the TensorForce source from Github and see if that works.

I can confirm the bug was resolved when updating to the Github version. Thanks for all. Closing the thread.


Creating a project on a new machine for the first time takes ages and eventually hangs infinitely

Discord username
No response
What type of issue is this?
Transient – Occurring only once
Is this issue blocking you from using Enso?

[ ] Yes, I can't use Enso because of this issue.

Is this a regression?

[ ] Yes, previous version of Enso did not have this issue.

What issue are you facing?
I downloaded the newest nightly on a clean machine and created a new project. I took ages to download Graal and eventually hang on

When I tried to open another project it was created quickly and run pretty smoothly. I was waiting for 30 min before doing any action.
Expected behaviour
New project creation should not hang.
How we can reproduce it?
No response
Screenshots or screencasts
No response
Logs
No response
Enso Version
Nightly 2.08
Browser or standalone distribution
Standalone distribution (local project)
Browser Version or standalone distribution
standalone
Operating System
MacOS
Operating System Version
No response
Hardware you are using
MacBookAir Apple M1 8gb MacOS Ventura 13.3.1

@sylwiabr you don't happen to have full logs, do you?

Archive.zip
@hubertp I don't remember which run was it so I am attaching zip with 2 logs

It may be due to the fact that the engine tries to download all the edition files from GitHub. You should be able to reproduce it by removing the ~/.local/share/enso/editions directory


Error while running variational_autoencoder_deconv.py example on TensorFlow

Tried to run https://github.com/fchollet/keras/blob/master/examples/variational_autoencoder_deconv.py as is.
Got exception in the generation part (on Keras 1.0.7 and TF 0.10.0rc0):

InvalidArgumentError: Conv2DSlowBackpropInput: input and out_backprop must have the same batch size
[[Node: conv2d_transpose_1 = Conv2DBackpropInput[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](conv2d_transpose_1/output_shape, transpose_9, transpose_7)]]

Full trace - https://gist.github.com/lukovkin/d2619674bb03f7bf3e161d4e1c6dab81.
Works just fine on Theano 8.2.
Please make sure that the boxes below are checked before you submit your issue. Thank you!

[X] Check that you are up-to-date with the master branch of Keras. You can update with:
pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps
[X] Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

I have also hit this.

Did you resolve this?

anyone? similar  problem here

I can confirm that the example currently works. TF 1.2, keras 2.0.5. I think this issue can be closed.

Did anybody resolve this problem? I met the same problem!
Invalid argument: Conv2DSlowBackpropInput: input and out_backprop must have the same batch size
input batch: 64 outbackprop batch: 56 batch_dim: 0


E2: Implement indexer for complex and matrix types

Since string type already implements this sort of functionality, I think it would be nice to have "indexer" for the following types:

angle
complex
matrix
matrix2
matrix4
quaternion
vector
vector2
vector4
wirelink*

Here is an example of how string indexer works (source: extended/string module):
#ifndef string:reverse()
    function string string:reverse()
    {
        if (This:length() < 2)
        {
            return This
        }
        local Reversed = ""
        for (Index = This:length(), 1, -1)
        {
            Reversed += This[Index]
        }
        return Reversed
    } # End of function string=s:reverse()
#endif

Above function reverses order of characters in a given string. If you look inside the loop, you see This[Index], which returns a single-character string that is positioned at given index for the specified string/text. Okay, but that is just for string type. Also, for string type, I think this is the same as using string=This:sub(Index, Index) function.
Now the following is what I would suggest for other types, as to what indexer getter would return and setter would assign (at least that's what get functions in extended/indexer module returns, while set functions assign a value at that index):
angle
```
number = angle[1] -- Get Pitch component of Angle (angle.p or angle.pitch).
number = angle[2] -- Get Yaw component of Angle (angle.y or angle.yaw).
number = angle[3] -- Get Roll component of Angle (angle.r or angle.roll).
```
complex
```
number = complex[1] -- Get Real component of Complex (complex.real).
number = complex[2] -- Get Imaginary component of Complex (complex.imag).
```
matrix
```
number = matrix[1] -- Get element at Row:1/Column:1 of Matrix.
number = matrix[2] -- Get element at Row:1/Column:2 of Matrix.
number = matrix[3] -- Get element at Row:1/Column:3 of Matrix.
number = matrix[4] -- Get element at Row:2/Column:1 of Matrix.
number = matrix[5] -- Get element at Row:2/Column:2 of Matrix.
number = matrix[6] -- Get element at Row:2/Column:3 of Matrix.
number = matrix[7] -- Get element at Row:3/Column:1 of Matrix.
number = matrix[8] -- Get element at Row:3/Column:2 of Matrix.
number = matrix[9] -- Get element at Row:3/Column:3 of Matrix.
```
matrix2
```
number = matrix2[1] -- Get element at Row:1/Column:1 of Matrix2.
number = matrix2[2] -- Get element at Row:1/Column:2 of Matrix2.
number = matrix2[3] -- Get element at Row:2/Column:1 of Matrix2.
number = matrix2[4] -- Get element at Row:2/Column:2 of Matrix2.
```
matrix4
```
number = matrix4[1]  -- Get element at Row:1/Column:1 of Matrix4.
number = matrix4[2]  -- Get element at Row:1/Column:2 of Matrix4.
number = matrix4[3]  -- Get element at Row:1/Column:3 of Matrix4.
number = matrix4[4]  -- Get element at Row:1/Column:4 of Matrix4.
number = matrix4[5]  -- Get element at Row:2/Column:1 of Matrix4.
number = matrix4[6]  -- Get element at Row:2/Column:2 of Matrix4.
number = matrix4[7]  -- Get element at Row:2/Column:3 of Matrix4.
number = matrix4[8]  -- Get element at Row:2/Column:4 of Matrix4.
number = matrix4[9]  -- Get element at Row:3/Column:1 of Matrix4.
number = matrix4[10] -- Get element at Row:3/Column:2 of Matrix4.
number = matrix4[11] -- Get element at Row:3/Column:3 of Matrix4.
number = matrix4[12] -- Get element at Row:3/Column:4 of Matrix4.
number = matrix4[13] -- Get element at Row:4/Column:1 of Matrix4.
number = matrix4[14] -- Get element at Row:4/Column:2 of Matrix4.
number = matrix4[15] -- Get element at Row:4/Column:3 of Matrix4.
number = matrix4[16] -- Get element at Row:4/Column:4 of Matrix4.
```
quaternion
```
number = quaternion[1] -- Get Real component of Quaternion (quaternion.real).
number = quaternion[2] -- Get I component of Quaternion (quaternion.i).
number = quaternion[3] -- Get J component of Quaternion (quaternion.j).
number = quaternion[4] -- Get K component of Quaternion (quaternion.k).
```
vector
```
number = vector[1] -- Get X component of Vector (vector.x).
number = vector[2] -- Get Y component of Vector (vector.y).
number = vector[3] -- Get Z component of Vector (vector.z).
```
vector2
```
number = vector[1] -- Get X component of Vector2 (vector.x).
number = vector[2] -- Get Y component of Vector2 (vector.y).
```
vector4
```
number = vector[1] -- Get X component of Vector4 (vector.x).
number = vector[2] -- Get Y component of Vector4 (vector.y).
number = vector[3] -- Get Z component of Vector4 (vector.z).
number = vector[4] -- Get W component of Vector4 (vector.w).
```
wirelink* (a bit special)
```
number = wirelink[address]      => wirelink:readCell(address)     -- Getter. Read (number) from address.
void   = wirelink[address] = N  => wirelink:writeCell(address, N) -- Setter. Write (number) N to address.
```

vector, vector2, vector4, quaternion, wirelink, and angle already have index operators. Both setters and getters.
Only types that don't are matrix* and complex

@bigdogmat

vector, vector2, vector4, quaternion, wirelink, and angle already have index operators. Both setters and getters.

Ah, right. https://github.com/wiremod/wire/commit/51125d1772d073eec136d5d4f61fa96d64f2a888

Also, your reverse method for strings is already implemented in the string library

Yes.. But, I remember I was on some server (that was a while ago), and when I searched for such function, it wasn't there (at that time). That's why it is (now) enclosed with #ifndef directive ;)
So, I kept it there to see how it is/was done.

wouldnt it be easier to just have toArray functions?

He wants it Python-style.

but shittons of new code for such a slim use case doesn't sound terribly efficient.
With X:toArray() you cover at least two use cases at once

Aren't they supposed to be immutable anyway?

That's another concern... Don't add setters :)
toArray should just return table.Clone(this) or what it's called

I have just run over this.. It is wirelink indexer for setting a value at address.
Now, from what I see, that works different than the actual function (WriteCell)..
I mean look at the first two checks:
WriteCell:
if not validWirelink(self, this) then return 0 end
if not this.WriteCell then return 0 end

Indexer:
if not validWirelink(self, this) then return value end
if not this.WriteCell then return value end

Seriously.. Then finally indexer calls into WriteCell (that again performs same checks), ignores return value from it, so indexer forces value while WriteCell returns either 1 or 0:
WriteCell:
if this:WriteCell(address, value) then return 1 else return 0 end

Indexer:
this:WriteCell(address, value)
return value

And just underneath that indexer setter, is the getter which is actually an alias to ReadCell:
e2function number wirelink:operator[](address) = e2function number wirelink:readCell(address)

What the F***. (That has to be cleaned-up.)

The difference here is, the function returns whether it succeeded, whereas the operator returns the value that was assigned, in order to be chainable.

So, that is valid?
Hm.. And how do you mean to chain that indexer setter?

Hm.. And how do you mean to chain that indexer setter?

link[address] = link[address_2] = link[address_3] = somevalue

@bigdogmat Okay, I see, now that makes sense. I am sorry this went off-topic now.
But, I could sworn chaining like that wasn't there when I have first attempted it like this (long time ago):
@persist A B T:table
T["C", number] = B = A = 123

https://github.com/wiremod/wire/commit/e024d79271e78938de0773844d77747468fafecb I think this introduced it, but I'm not sure.


Cppclient ssl

remove some options from bazel.rc to allow build of boringssl
implement post_ssl

Not added by your PR but if you could fix the following warning:
INFO: From Compiling cppclient/beast.cc:
cppclient/beast.cc:147:10: warning: moving a local object in a return statement prevents copy elision [-Wpessimizing-move]
  return std::move(req);
         ^
cppclient/beast.cc:147:10: note: remove std::move call here
  return std::move(req);
         ^~~~~~~~~~   ~
1 warning generated.


Image Management: Implementation for create image type, image version and get image version REST APIs

Included changes

Database changes for containerization and image management - https://jira01.corp.linkedin.com:8443/browse/BDP-3108
Implementation of create image type REST API - https://jira01.corp.linkedin.com:8443/browse/BDP-3655
Implementation of create image version REST API - https://jira01.corp.linkedin.com:8443/browse/BDP-3654
Implementation of get image version REST API - https://jira01.corp.linkedin.com:8443/browse/BDP-3712

Initial first cut commit and pull request : https://github.com/azkaban/azkaban/pull/2637
Image management REST API design doc: https://docs.google.com/document/d/1zbA8tZ7jjT3owWNPFLyhtU9A-8RKxRU6Rkqq1LA1G6o/edit#

Pull Request Test Coverage Report for Build 5618

121 of 215 (56.28%)  changed or added relevant lines in 19 files are covered.
13 unchanged lines in 5 files lost coverage.
Overall coverage decreased (-0.009%) to 42.378%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

az-core/src/main/java/azkaban/Constants.java
0
1
0.0%

azkaban-common/src/main/java/azkaban/imagemgmt/models/ImageVersion.java
31
32
96.88%

azkaban-common/src/main/java/azkaban/imagemgmt/exeception/ImageMgmtException.java
2
4
50.0%

azkaban-common/src/main/java/azkaban/imagemgmt/models/ImageType.java
24
26
92.31%

az-core/src/main/java/azkaban/utils/JSONUtils.java
2
6
33.33%

azkaban-common/src/main/java/azkaban/imagemgmt/exeception/ImageMgmtDaoException.java
0
4
0.0%

azkaban-common/src/main/java/azkaban/imagemgmt/models/ImageOwnership.java
12
18
66.67%

azkaban-web-server/src/main/java/azkaban/imagemgmt/utils/ConverterUtils.java
8
14
57.14%

azkaban-web-server/src/main/java/azkaban/imagemgmt/servlets/ImageVersionServlet.java
0
7
0.0%

azkaban-common/src/main/java/azkaban/imagemgmt/models/BaseModel.java
8
16
50.0%

Files with Coverage Reduction
New Missed Lines
%

azkaban-web-server/src/main/java/azkaban/imagemgmt/servlets/ImageTypeServlet.java
1
0%

azkaban-common/src/main/java/azkaban/jobExecutor/AbstractJob.java
2
37.93%

azkaban-web-server/src/main/java/azkaban/imagemgmt/servlets/ImageVersionServlet.java
2
0%

azkaban-exec-server/src/main/java/azkaban/execapp/JobRunner.java
3
77.03%

azkaban-common/src/main/java/azkaban/jobExecutor/AbstractProcessJob.java
5
52.58%

Totals

Change from base Build 5617:
-0.009%

Covered Lines:
14317

Relevant Lines:
33784

💛  - Coveralls

Thank you for taking care of all the comments.


Setting root span status explicitly

Enable unambiguous queries for traces with root successes or failures

please follow https://www.conventionalcommits.org/en/v1.0.0/
this PR title should've been: fix(js/plugins/google-cloud): setting root span status explicitly


Sort platforms list

As you can see:

it's currently complicated to find the correct one from the list. However, I'm not sure if it's the best approach to sort the list every time or if we should insert it correctly instead. But in the second case, that would mean updating all the entries in the DB. Not sure which solution you prefer?

probably not the place to discuss this, but is there scope to group them by platform tier?

It would change it indeed. But I don't consider it a problem: it'll remain the default platform, no need to put it at the op of the list. I was also thinking about making the current platform text bold or something along the line (in another PR).


Added notify workflow for posting test results to Slack

No review needed.

Coverage remained the same at 37.138% when pulling 39b580e0be41c0d4bc3f316b353161e7f6ec8054 on andy/add-notify into 7649ead6f0093e4f5854a606cb78458b59f38ae7 on master.


Identify use cases representing business requirements

we identified the Spectrum Auction and produced a demo, which is currently in the repo.


clarify confusing language and make consistent

@ericsampson : Thanks for your contribution! The author(s) have been notified to review your proposed change.

Excellent feedback, @ericsampson. The article in question has been broken up into smaller topics, but I applied your suggestions to About Keys.  Thank you! #please-close

Thanks @msmbaldwin ! I took a peek at the result and noticed an extra word that got left in:
"VERIFY operations should be are performed locally


🐛 Branching - Browser recalls deleted branch

cc: @UlyssesMaclaren
Describe the issue that you're seeing. Any Loom videos or screenshots usually help a lot!
The browser remembers what branch I am on even if it is deleted.

Figure: This is an old branch that's been deleted
I would expect Tina to notice the branch doesn't exist, and put me on main
Reproduction
https://github.com/SSWConsulting/SSW.Website-v3
Steps to reproduce

Go to editor
new branch
Make some changes
Delete branch
Close browser
Reopen
Go to Tina editor
You are on the deleted branch

System Info
System:
    OS: macOS 13.3.1
    CPU: (8) x64 Intel(R) Core(TM) i7-1068NG7 CPU @ 2.30GHz
    Memory: 51.56 MB / 16.00 GB
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 18.14.0 - ~/.nvm/versions/node/v18.14.0/bin/node
    Yarn: 3.5.1 - ~/Developer/SSW/SSW.Website-v3/node_modules/.bin/yarn
    npm: 9.3.1 - ~/.nvm/versions/node/v18.14.0/bin/npm
  Browsers:
    Brave Browser: <IP_ADDRESS>
    Chrome: 113.0.5672.126
    Firefox: 113.0.1
    Safari: 16.4
  npmPackages:
    @tinacms/cli: ^1.5.14 => 1.5.14
    tinacms: ^1.5.6 => 1.5.6

Validations

[X] Follow our Code of Conduct
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug.

Seems like this has been fixed
Thanks team 🙂


Disable prechecking by default

Until prechecking is implemented for Ithaca

@akoptelov anything pending for this one? or can it be merged already?


Attribute shorthand not working in Vue

Describe the bug
I fork an example on your website and also found a bug about Attribute shorthand bindings
In Vue, I pass loading to the prop but I get an empty string loading = ''  instead of a boolean. It is completely fine in React.
You can open Reproduced below and will see that.
I don't understand, Why is there such a difference between disabled: ButtonProps["disabled"]  and  expanded: boolean | undefined | null;

To Reproduce
https://stackblitz.com/edit/joe-bell-cva-qb58mc?file=src%2FApp.vue
Expected behavior
If I pass loading to prop, the component will receive loading = true

Sorry, please could you be more specific? I’m struggling to understand the problem here 🙏🏼

Right OK I see, thank you! Seems like very much a genuine bug with booleans. I'll do some digging and consider as part of this:
https://github.com/joe-bell/cva/discussions/79


Introduce meta-iris-thirdparty and meta-webserver

Maintainer Checklist

Is there a changelog entry? NO
Was a Jenkins run performed? NO


Refactor main into functions.

I wanted to work on #17, but that requires reusing parts of the code which wasn't possible with the existing implementation. This also reduces the scope of some variables and makes it easier to read.

Sweeet! Looks great, appreciate the refactor and the better error handling :D


[hotfix] fix tv-server build failed issue on arm and added CI for it

Problem

examples/tv-app/android build failed on the android arm
No CI for android arm ( only arm64)

Change overview

fix the build error
added ci

Testing

build_example.py for android

fast track: trivial change and build update only.


Popover does not close on PopoverButton click on Safari (macOS & iOS)

Describe the bug
When the parent element of a Popover component is focusable (e.g. div with tabindex), the Popover won't close if the PopoverButton is clicked.
Affected browser: Safari (macOS & iOS)
To Reproduce
REPL
Library version
1.0.2

How can I contribute in this repo?


KloudIdentity SCIM gateway added to partner driven integrations.

KloudIdentity SCIM gateway added under the partner driven integrations.

@microsoft-github-policy-service agree company="Kloudynet Technologies Sdn Bhd"

From: microsoft-github-policy-service[bot] @.>
Sent: Wednesday, March 6, 2024 1:57 PM
To: MicrosoftDocs/entra-docs @.>
Cc: Asela Samarakoon @.>; Mention @.>
Subject: Re: [MicrosoftDocs/entra-docs] KloudIdentity SCIM gateway added to partner driven integrations. (PR #476)
You don't often get email from @.*** Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification
@AselaSamarakoonhttps://github.com/AselaSamarakoon please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
@microsoft-github-policy-service agree company="Kloudynet Technologies Sdn Bhd"
Options:

(default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

(when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"
Contributor License Agreement
Contribution License Agreement
This Contribution License Agreement (“Agreement”) is agreed to by the party signing below (“You”),
and conveys certain license rights to Microsoft Corporation and its affiliates (“Microsoft”) for Your
contributions to Microsoft open source projects. This Agreement is effective as of the latest signature
date below.

Definitions.
“Code” means the computer software code, whether in human-readable or machine-executable form,
that is delivered by You to Microsoft under this Agreement.
“Project” means any of the projects owned or managed by Microsoft and offered under a license
approved by the Open Source Initiative (www.opensource.orghttp://www.opensource.org/).
“Submit” is the act of uploading, submitting, transmitting, or distributing code or other content to any
Project, including but not limited to communication on electronic mailing lists, source code control
systems, and issue tracking systems that are managed by, or on behalf of, the Project for the purpose of
discussing and improving that Project, but excluding communication that is conspicuously marked or
otherwise designated in writing by You as “Not a Submission.”
“Submission” means the Code and any other copyrightable material Submitted by You, including any
associated comments and documentation.
Your Submission. You must agree to the terms of this Agreement before making a Submission to any
Project. This Agreement covers any and all Submissions that You, now or in the future (except as
described in Section 4 below), Submit to any Project.
Originality of Work. You represent that each of Your Submissions is entirely Your original work.
Should You wish to Submit materials that are not Your original work, You may Submit them separately
to the Project if You (a) retain all copyright and license information that was in the materials as You
received them, (b) in the description accompanying Your Submission, include the phrase “Submission
containing materials of a third party:” followed by the names of the third party and any licenses or other
restrictions of which You are aware, and (c) follow any other instructions in the Project’s written
guidelines concerning Submissions.
Your Employer. References to “employer” in this Agreement include Your employer or anyone else
for whom You are acting in making Your Submission, e.g. as a contractor, vendor, or agent. If Your
Submission is made in the course of Your work for an employer or Your employer has intellectual
property rights in Your Submission by contract or applicable law, You must secure permission from Your
employer to make the Submission before signing this Agreement. In that case, the term “You” in this
Agreement will refer to You and the employer collectively. If You change employers in the future and
desire to Submit additional Submissions for the new employer, then You agree to sign a new Agreement
and secure permission from the new employer before Submitting those Submissions.
Licenses.

Copyright License. You grant Microsoft, and those who receive the Submission directly or
indirectly from Microsoft, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license in the
Submission to reproduce, prepare derivative works of, publicly display, publicly perform, and distribute
the Submission and such derivative works, and to sublicense any or all of the foregoing rights to third
parties.
Patent License. You grant Microsoft, and those who receive the Submission directly or
indirectly from Microsoft, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license under
Your patent claims that are necessarily infringed by the Submission or the combination of the
Submission with the Project to which it was Submitted to make, have made, use, offer to sell, sell and
import or otherwise dispose of the Submission alone or with the Project.
Other Rights Reserved. Each party reserves all rights not expressly granted in this Agreement.
No additional licenses or rights whatsoever (including, without limitation, any implied licenses) are
granted by implication, exhaustion, estoppel or otherwise.

Representations and Warranties. You represent that You are legally entitled to grant the above
licenses. You represent that each of Your Submissions is entirely Your original work (except as You may
have disclosed under Section 3). You represent that You have secured permission from Your employer to
make the Submission in cases where Your Submission is made in the course of Your work for Your
employer or Your employer has intellectual property rights in Your Submission by contract or applicable
law. If You are signing this Agreement on behalf of Your employer, You represent and warrant that You
have the necessary authority to bind the listed employer to the obligations contained in this Agreement.
You are not expected to provide support for Your Submission, unless You choose to do so. UNLESS
REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING, AND EXCEPT FOR THE WARRANTIES
EXPRESSLY STATED IN SECTIONS 3, 4, AND 6, THE SUBMISSION PROVIDED UNDER THIS AGREEMENT IS
PROVIDED WITHOUT WARRANTY OF ANY KIND, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTY OF
NONINFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
Notice to Microsoft. You agree to notify Microsoft in writing of any facts or circumstances of which
You later become aware that would make Your representations in this Agreement inaccurate in any
respect.
Information about Submissions. You agree that contributions to Projects and information about
contributions may be maintained indefinitely and disclosed publicly, including Your name and other
information that You submit with Your Submission.
Governing Law/Jurisdiction. This Agreement is governed by the laws of the State of Washington, and
the parties consent to exclusive jurisdiction and venue in the federal courts sitting in King County,
Washington, unless no federal subject matter jurisdiction exists, in which case the parties consent to
exclusive jurisdiction and venue in the Superior Court of King County, Washington. The parties waive all
defenses of lack of personal jurisdiction and forum non-conveniens.
Entire Agreement/Assignment. This Agreement is the entire agreement between the parties, and
supersedes any and all prior agreements, understandings or communications, written or oral, between
the parties relating to the subject matter hereof. This Agreement may be assigned by Microsoft.

—
Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/entra-docs/pull/476#issuecomment-1980138978, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AZH6W3H3IIBNT7XSUVBTWNDYW2V4ZAVCNFSM6AAAAABEINCHY2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBQGEZTQOJXHA.
You are receiving this because you were mentioned.

@billmath

Can you review this PR?
IMPORTANT: When this content is ready to merge, you must add #sign-off in a comment or the approval may get overlooked.

#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

#sign-off

#sign-off


mac os not working

it gives always this error whatever the file and the path are correct
anass@MacBook-Pro-de-anass-90 ~ % python3 theme_builder_setup.py -i /Users/anass -p ctk_theme_builder-develop.zip
Checking Python interpreter version...
Python version, 3.11.8, is a supported version.
Starting CTk Theme Builder deployment.
Installation base location: /Users/anass
Creating application home: /Users/anass/ctk_theme_builder
======================== KEY LOCATION MAPPINGS ==========================
Package Location: /Users/anass/ctk_theme_builder-develop.zip
App Home = /Users/anass/ctk_theme_builder
Assets Location = /Users/anass/ctk_theme_builder/assets
User Theme Location = /Users/anass/ctk_theme_builder/user_themes
Log Location = /Users/anass/ctk_theme_builder/log
Creating application assets location: /Users/anass/ctk_theme_builder/assets
Creating application assets location: /Users/anass/ctk_theme_builder/log
Creating application temp location: /Users/anass/ctk_theme_builder/tmp
Creating application data location: /Users/anass/ctk_theme_builder/assets/data
Creating user themes location: /Users/anass/ctk_theme_builder/user_themes
Checking for repository: /Users/anass/ctk_theme_builder/assets/data/ctk_theme_builder.db
Greenfield installation - creating a new repository.
Unpacking package: /Users/anass/ctk_theme_builder-develop.zip to: /Users/anass
[unpack_package] WARNING: Artefact file, ctk_theme_builder-develop.zip, appears to be an invalid ZIP file.
Traceback (most recent call last):
File "/Users/anass/theme_builder_setup.py", line 567, in
unpack_package(zip_pathname=package_path, install_location=install_location)
File "/Users/anass/theme_builder_setup.py", line 427, in unpack_package
with ZipFile(zip_pathname, 'r') as archive:
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/zipfile.py", line 1286, in init
self.fp = io.open(file, filemode)
^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'ctk_theme_builder-develop.zip'
anass@MacBook-Pro-de-anass-90 ~ %

Hi @rennoun . You need to put the full pathname to the ZIP file. Also the ZIP file name is unexpected. Did you rename it?
Typically:
From a terminal window:
brew install python
brew install python-tk

Download https://github.com/avalon60/ctk_theme_builder/releases/download/v3.0.1/ctk_theme_builder-3.0.1.zip
Unpack the theme_builder_setup.py file and run it. Assuming you have unpacked it and placed it in your Downloads directory:
cd $HOME/Downloads

python3 theme_builder_setup.py -i /Users/anass -p $HOME/Downloads/ctk_theme_builder-3.0.1.zip

PS: If you are installing to your current login home directory, you can skip the -i option:
python3 theme_builder_setup.py -p $HOME/Downloads/ctk_theme_builder-3.0.1.zip

same problem :
anass@MacBook-Pro-de-anass-90 ctk_theme_builder % python3 theme_builder_setup.py -i /Users/anass -p /Downloads/ctk_theme_builder-3.0.1.zip
Checking Python interpreter version...
Python version, 3.11.8, is a supported version.
ERROR: Cannot locate the specified package ZIP file: /Downloads/ctk_theme_builder-3.0.1.zip
Traceback (most recent call last):
File "/Users/anass/Downloads/ctk_theme_builder/theme_builder_setup.py", line 493, in
lprint(f'ERROR: Cannot locate the specified package ZIP file: {package_path}')
File "/Users/anass/Downloads/ctk_theme_builder/theme_builder_setup.py", line 50, in lprint
LOG_FILE.write(f'{output_text}\n')
^^^^^^^^
NameError: name 'LOG_FILE' is not defined
anass@MacBook-Pro-de-anass-90 ctk_theme_builder %

Hi @rennoun. You appear to have entered a bad pathname for the Downloads location:
python3 theme_builder_setup.py -i /Users/anass -p /Downloads/ctk_theme_builder-3.0.1.zip  # wrong
It should be:
python3 theme_builder_setup.py -i /Users/anass -p $HOME/Downloads/ctk_theme_builder-3.0.1.zip

thank you so much that works very well


Blazor Components Lifecycle updates

Fixes #16099
Moved from #15433.
Steve, you were requested to :eyes: by DR on this one. He signed off on #15433.

I see that @guardrex merged updated language for this a few weeks ago with #15630...

OnParametersSetAsync and OnParametersSet are called when a component has received parameters from its parent and the values are assigned to properties. These methods are executed after component initialization and each time the parent component is rendered:

...which seems more correct than the proposed wording in this request.  Mainly it's important to point out that these methods are called when the parent is rendered, not necessarily when the component is rendered.  For example, if a component has a button and handles it's click event to change some component-local state (state that the parent doesn't know about) then the OnParameterSet[Async] methods do not get called.
Also, I think the proposed change incorrectly implies that these methods are only called when parameters have changed, but the behavior is more nuanced than that.  The methods are always called after initialization (even if there are no parameters).  Then, before subsequent renders of the parent, if the RenderTreeDiffBuilder determines there's a change to a parameter or can't determine if there was a change then the methods are called again.  The RenderTreeDiffBuilder can determine when simple parameters change but not complex type parameters.  If one of the parameters has a complex type then the methods are called before every render, even if the parameters haven't changed.  See RenderTreeDiffBuilder code and comment: https://github.com/aspnet/AspNetCore/blob/0a79d34f62a2e153d7104a88b487b75ef8860bb7/src/Components/Components/src/RenderTree/RenderTreeDiffBuilder.cs#L512
Sorry for the wordy post for a simple change request, but I spent a lot of time trying to understand what was happening when implementing custom controls in my own Blazor app, and I'd love to see much better docs for all the lifecycle events.

Item 1

these methods are called when the parent is rendered

For ...

* After component initialization.

We could go with something like ...

* After component initialization when its parent component is rendered.

Item 2

these methods are only called when parameters have changed, but the behavior is more nuanced than that

For ...

* When a component has received changed parameters from its parent and the values are assigned to properties.

We could go with something like ...

* When a component has assigned values to properties from changes to simple parameters by its parent or when complex type parameters are used.

... which could alternatively compose as a 2nd level list (probably better) ...

* When a component has assigned values to properties:
  * From changes to simple parameters by its parent.
  * When complex type parameters are used.

@SteveSandersonMS Need your sign off ... everything is on the diff, and it's short (quick review!).

This isn't working out ...
It doesn't make sense to me to create a condition of the methods being called with "a possibly-changed set of parameters" followed by conditions that don't say clearly when the methods are called.
I'll float another draft ... a hybrid ... shortly.

Ok ... that seems to maintain the further explanation and keep it focused on conditions.


Unclear when to specify @ in space name

When attempting to send from one wallet to another where the target wallet own a space, docs are unclear that you need to preface space name with '@'.  Docs example have it correct.  https://docs.spacesprotocol.org/getting-started/resolver but spaces-cli send --help could be a bit clearer.
$ spaces send 5000 -f 3 --to phaedrus
{
"code": -1,
"message": "recipient must be a valid space name or an address"
}
$ spaces send 5000 -f 3 --to @phaedrus
{
"sent": [
{
"txid": "f2a7663d9b00aa4e5ea0f67a04a5fd329c2f24154d93cc7162c3bfee29516a34",
"tags": [
"transfers"
]
}
]
}

agreed perhaps this should be "recipient must be a valid space name prefixed with @ or an address"


Not approved for airdrop

My keybase account username is bbqlord
I have 2 phones and 1 laptop connected
I have a verified number from new Zealand
I have a beautiful smile
I still cannot recieve the airdrop... :( Pls help

#21055


Update bug_report.yaml for 1.4.0 stable

Description

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

/azp run


Change name of Curator.sol to ERC721Curator.sol?

This idea just came up while working on the readme. I think this could be really helpful for making the nft architecture of everything more obvious to devs specifically. One hold up is that it might inadvertently make ppl think u can only curate ERC721s. What say you @iainnash ??

It's a little confusing imo because the curation contract can curate individual NFTs, 1155s, or even wallets. It itself being 721 compliant is a bonus perhaps not a core part of its functionality?

good points -- I agree lets leave as is


[Question] Is it possible that Ninja can export compile_commands.json?

Ninja works quite well in my daily use, so thanks for this fabulous project.
I have a question now, is it possible that Ninja can export compile_commands.json now? If not, any solution exists?
Thanks very much.

Yes, ninja supports ouputting compilation database.
https://ninja-build.org/manual.html#_extra_tools
https://sarcasm.github.io/notes/dev/compilation-database.html#ninja
2019年5月2日(木) 14:30 Winterreise<EMAIL_ADDRESS>
Ninja works quite well in my daily use, so thanks for this fabulous
project.
I have a question now, is it possible that Ninja can export
compile_commands.json now? If not, any solution exists?
Thanks very much.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ninja-build/ninja/issues/1569, or mute the thread
https://github.com/notifications/unsubscribe-auth/AASMO4RVM6AGEVNYQJJKKMTPTJ36VANCNFSM4HJ3P4IA
.

Thanks very much!

Yes, ninja supports ouputting compilation database. https://ninja-build.org/manual.html#_extra_tools https://sarcasm.github.io/notes/dev/compilation-database.html#ninja 2019年5月2日(木) 14:30 Winterreise<EMAIL_ADDRESS>…
Ninja works quite well in my daily use, so thanks for this fabulous project. I have a question now, is it possible that Ninja can export compile_commands.json now? If not, any solution exists? Thanks very much. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1569>, or mute the thread https://github.com/notifications/unsubscribe-auth/AASMO4RVM6AGEVNYQJJKKMTPTJ36VANCNFSM4HJ3P4IA .

Thanks very much! Hope you have a good day.


Reduce Extension Size

Remove the artifacts from the extension bundle. Instead, pull the installer artifacts dynamically from bintray and extract on the build agent during execution.
This will add a few seconds to process, however eliminate tons of people asking questions about how to increase the extension file limit  - thanks MS :(
Invoke-WebRequest "https://dl.bintray.com/jeremy-long/owasp/dependency-check-5.3.2-release.zip" -OutFile "dependency-check-5.3.2-release.zip"

On the fly install of the tools has the file size down to 520KB. Good to go in PR #30


Tree View for Nuke Gizmo

https://github.com/SpinVFX/Cryptomatte/tree/crypto_tree_view
Hey Guys.
I started working on a tree view for the cryptomatte gizmo in Nuke.
It opened up a few points:

first I did a round of pylinting on the code, and it lit up with a few possible errors in the current code, mostly in the tests (some function calls with invalid arguments)
https://github.com/SpinVFX/Cryptomatte/commit/adaf4d7eb7143945f85ea989686ef8b16e04248c?diff=split
Possible errors at line 64, 68. The rest was just pep8 formatting recommendations.

I then added the tree view. So far it's QT based, though I would like to try the built in nuke SceneView knob to see if it performs any better. The challenge there is getting the cryptomatte data for the whole frame range so that the tree view stays consistent.
So far I rely on an external function that makes a set out of multiple sidecar manifests into a new JSON file and use that as my tree. Could not find a quick way to extract it out of EXRs, though Ben Dickson mentions he's done something similar in another thread.
This is my main issue so far, getting good data to put in the tree. Once it's in there, it works quite well.
One thing I was thinking about was to implement some sort of disk caching for cryptomatte manifests of the whole scene. It's easy enough to implement, the tricky part being re-caching it if the frames get overwritten.
Other challenge is some manifests represent paths as /path/to/object, while some others use path.to.object.

Once I had my tree working, I found the methods used in the gizmo to add or remove from the matteList to be really not optimized for running in a loop, trying to bulk add a few thousand objects would get extremely slow. I have modified the functions so that the current matteList gets converted to a set of floats, and the list being added or subtracted being also converted to a set of floats, do a set union or a set subtraction, then convert back to stings/ based on the manifest data. That didn't seem to affect performance much while picking one by one (it may even have improved it, to be confirmed) but it does a huge difference for large sets.

Just wanted to put this up there, as I'd love some feedback from you guys. Our Lighters didn't want to use cryptomatte because it's too manual to pick a few hundred objects manually, and I think we should put our brains together to see how we can best implement that as part of cryptomatte.
Cheers
Erwan

Sweet!
I'll take a look into it since it's something I was writing my self.
Congrats!
Cheers,
Diogo
On Thu, Jun 21, 2018 at 1:27 PM Erwan Leroy<EMAIL_ADDRESS>wrote:

https://github.com/SpinVFX/Cryptomatte/tree/crypto_tree_view
Hey Guys.
I started working on a tree view for the cryptomatte gizmo in Nuke.
It opened up a few points:

first I did a round of pylinting on the code, and it lit up with a few
possible errors in the current code, mostly in the tests (some function
calls with invalid arguments)
SpinVFX/Cryptomatte@adaf4d7?diff=split
https://github.com/SpinVFX/Cryptomatte/commit/adaf4d7eb7143945f85ea989686ef8b16e04248c?diff=split
Possible errors at line 64, 68. The rest was just pep8 formatting
recommendations.
I then added the tree view. So far it's QT based, though I would like
to try the built in nuke SceneView knob to see if it performs any better.
The challenge there is getting the cryptomatte data for the whole frame
range so that the tree view stays consistent.
So far I rely on an external function that makes a set out of multiple
sidecar manifests into a new JSON file and use that as my tree. Could not
find a quick way to extract it out of EXRs, though Ben Dickson mentions
he's done something similar in another thread.
This is my main issue so far, getting good data to put in the tree.
Once it's in there, it works quite well.
One thing I was thinking about was to implement some sort of disk
caching for cryptomatte manifests of the whole scene. It's easy enough to
implement, the tricky part being re-caching it if the frames get
overwritten.
Other challenge is some manifests represent paths as /path/to/object,
while some others use path.to.object.
Once I had my tree working, I found the methods used in the gizmo to
add or remove from the matteList to be really not optimized for running in
a loop, trying to bulk add a few thousand objects would get extremely slow.
I have modified the functions so that the current matteList gets converted
to a set of floats, and the list being added or subtracted being also
converted to a set of floats, do a set union or a set subtraction, then
convert back to stings/ based on the manifest data. That didn't seem to
affect performance much while picking one by one (it may even have improved
it, to be confirmed) but it does a huge difference for large sets.
Just wanted to put this up there, as I'd love some feedback from you guys.
Our Lighters didn't want to use cryptomatte because it's too manual to pick
a few hundred objects manually, and I think we should put our brains
together to see how we can best implement that as part of cryptomatte.
Cheers
Erwan
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/Psyop/Cryptomatte/issues/79, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAfs4ifWrekjJVlZ-V8fc8fGL5zCf_Vlks5t-8lzgaJpZM4UyZDr
.

Hi Erwan (@herronelou) , thanks for bringing us this. I really appreciate the contributions and want to encourage this kind of thing. I’m making a point of saying this because I’m in the position of having to be a stickler for certain things for the sake of the integrity of the project, so please don’t take any of the criticism as lack of interest in your work - encouraging contributions is important to us.
Tree view:
I’m definitely into the idea of a tree view. Loading the manifest on every frame is not great and a faster way to do this would be nice. It could also be loaded on demand, when the tab is accessed.
In the Cryptomatte spec, these are not meant to be paths, so there is no guarantee that the names actually represent a hierarchy or that one can be extracted from them (and in fact it usually can’t be). However in certain cases, such as Katana-like paths, a hierarchy could be extracted, and you could do it automatically in these cases. Also worth noting - there is also no standard for what the path delimiter is, in the case that the identifier is a path.
Set manipulations:
I’m definitely open to accepting changes like that. (Subject to everything continuing to work properly- see below). There’s a wrinkle here which makes it not totally straightforward - there may be things selected by names which are not in the manifest. That drove the complexity of the existing code. However, it could absolutely be improved, it just needs to be done in a way that doesn’t introduce subtle bugs. (See the part about tests below).
One word of caution about picking hundreds of objects. Currently we use an expression node for extraction, so there are potential performance issues here with large selections. We find the most commonly used Cryptomattes are the material ones, since those are already conglomerations of items and you don’t need to select hundreds of things, and this maps well to what people actually use mattes for most of the time, changing material properties. One thing we could use is a C++ node for performing the extraction more efficiently than an expression node for such a use case.
PyLinting:
There are some issues here.
The pylinting broke the tests. It removed @classmethod decorators, which caused most of the tests not to work. I’m not sure why it thought that was a safe change to make.
Second, the linting making tons of formatting changes, which is an issue if I want to merge your changes, or if you want to merge updates from mine. The issue is that your branch now contains small changes to hundreds of lines, which will be merge conflicts. I think it’s best to do large scale reformatting when introducing major version changes that are likely to cause merge conflicts anyway.
As for the errors that it found, thanks for that and we’ll look into them.
Failed tests:
Once the tests are repaired, there are 5 failed tests, once of which I think is an actual new bug. It is no longer possible to remove objects which are selected by name but not in the manifest. The tests should be fixed to accommodate the changes.
Tests aren’t just a way for me to nit-pick other people’s work, they protect you from introducing subtle bugs in your own branch as well. So I highly recommend using them!
Failed: CSVParsingNuke.test_big_csv_through_gizmo
AssertionError: Round trip failed: Came out as: <-1.82577004544e+11>, <-2.74633926288e+29>, <-5.90858871874e-06>, <1.34596486673e-20>, <1.92957949132e+34>, <123.449996948>, <16866966528.0>, <2.59943444547e-08>, <2.82940902496e-31>, <5.10401209831e-23>
---------
Failed: CSVParsingNuke.test_csv_through_gizmo
AssertionError: Round trip failed: Came out as: <-2.54616133732e-10>

I suspect these two are a red herring, based on a way in which you’ve reorganized the code.
---------
Failed: CryptomatteNukeTests.test_matte_list_numeric
AssertionError: Removal of name by number failed.. ("<7.36562399642e+18>, heroflower" vs "<2.07262543558e+26>, <7.36562399642e+18>")

This is a new bug. This means that if you have something selected by name, but that name is not in the manifest, you can’t remove the item.
---------
Failed: CryptomatteNukeTests.test_output_preview
AssertionError: Preview image did not light up: (red) 0.0199350118637 vs 1.0

---------
Failed: CryptomatteNukeTests.test_output_preview_multi
AssertionError: Bunny pixels are wrong.: (red) 0.0199350118637 vs 1.0
---------
TESTING FAILED: 5 failed, 0 errors. (47 test cases.)

These two are also only failing because you changed the preview color from yellow to green. (Not a big deal of course, just a good idea to update the tests when you change a behavior).

Hey Jonah.
Thanks for the detailed feedback.
I'll try to do an answer as detailed.
The current Branch was targeted for our in house use, and the idea to re-share it with the wold came later, and I had to get approval from the company to let the work out... more on this later.
Tree View.
By looking at multiple manifests, I did notice that there was no standards, and that is wasn't necessarily a tree structure. Even for those, however, it turned out to be quite helpful to have a list of objects sorted by name. (I have run the tree view on the rabbit test image, and it works pretty well, the issues come when I pick one of the few objects that do have children, as they have about 10k children, my processor goes crazy for a minute, though it does success eventually. I think it might be a dirty implementation of my recursive selection though.)
Set manipulation:
I'll cover the possible bug later on. About the performance issues when picking hundreds of objects, I did notice, though even at a few hundreds it processed really fast (my test asset in house is a spaceship with 152 visible pieces in the test frame) and that renders pretty much instantaneously at 2k.
I tried on the Rabbit test image from you guys (with 32k objects). I selected all, and that did take like a good minute to render a frame.
Pylinting:
I'll be honest there, I didn't actually run the tests. I knew I should, but I got lazy and didn't feel like trying to figure out how to run them.
I removed the @classmesthod decorators because they seemed wrong to me. Normally a classmethod must have the first parameter be 'cls', and not have self use. The classes you had here had self use and no cls argument, so I assumed they were regular methods. Now looking at it more, it made sense to have classmethods for unittests, and it runs in your case because you replaces cls by self both in the method definition and in the code. A proper fix from me would have been to replace all the 'self' by 'cls'.
About all the small format changes, I did that before thinking about trying to push back. I realized it after I made it, but I still think it's a good idea to respect formatting standards. If you want to wait for a major version to do them that's cool. I tend to do them early because I use pycharm and it lights up all over the place when there are badly formatted pieces of code.
Failed Tests:
I think it should still be possible to remove an object by name even if it's not in the manifest. The difference probably comes from the fact that I regenerate the list on each picking.
For example if you had 2 objects in the matteList as names, but not loaded in the manifest, on picking (let's say remove) they would be reconverted to floats (I do all my sets on floats), the one that matches the picked color would get removed, and the other one would make it back to the list, but because it's not in the manifest anymore, it would make it as a  string instead of the original name.
Seems like this is what is happening there: "<7.36562399642e+18>, heroflower" vs "<2.07262543558e+26>, <7.36562399642e+18>", heroflower is most likely not in the manifest anymore, so it got converted back to a float. It might not be ideal but it shouldn't affect functionality. The good news is that reloading the manifest and repicking anything would turn it back into heroflower.
I was hoping to find a good way to load/cache sequence manifests and that might be able to be used as the default manifest for a sequence, so that the manifest is always there, and every object is there on every frame.
For the color test, I did change the colors, though I mean to invert the two I picked so that the default color doesn't change from the real cryptomatte.
If you're able to push your version of my branch with the fixed unittests I can go from there and fix these issues.

Sorry for the slow response, a lot’s been happening and this fell off my radar. Anyway, I’ve tried it out with the sidecar manifest example, and the tree view is cool for sure. I like that the preview lights up one color when you select things, and then have +, =, - buttons, and then lights up another color when you’ve added them (though I would switch which color is which to not introduce arbitrary changes). For deriving the “tree”, I think it might be useful to use any non-alphanumeric character as a possible delimiter, so you’d see “flower” as the parent of “flower.petal”.
It also might be nice to have an indication of what's already in the matte list, perhaps as italicized entries.
Really though, I’m hung up on something here, which also applies to wild card / regex keyers: the way it is right now, selecting a few hundred or thousand items generates an expression which is unacceptably slow to evaluate. As a result I think giving users a way to do that easily or accidentally is asking for trouble. I think until there’s a C++ plugin which can do this with acceptable performance, this is probably a pretty fraught proposition. I think a way of performing the extraction with better performance should be a priority here. Maybe this would be a whole replacement of the gizmo, or maybe it should be a minimal node which lives inside the gizmo, only replacing the expression node.
Coding stardards, linting, etc:
I think it’s probably best to mostly talk about the tree view and not get hung up on this, but I will add some notes about this. I’d like to reiterate that I think reformatting all the code is a mistake even if it's going to be a private branch, because I have changes in a dev branch that you'll find impossible to rebase your changes onto without a lot of manual work. I use an auto-formatter myself sometimes, but I only run it on code I'm actively working on because I know there are people like you who are maintaining their own branches with modifications in them, who will have a lot of work to do if I reformat everything.
For running the tests, there are instructions in the readme.


Remove side effect from accumulate_more

In mrview's updater we regulate the size of processed docs batch either by number of items or by memory it allocates. We are comparing threshold memory to updater process memory, but it is not exclusively defined by size of the batch and as a result we can get different value for data_size on different nodes for the same view shards.
This patch compares memory threshold to memory allocated just by accumulator, removing side effect from the function.

@davisp Done.

A note just for PR's history sake: values reported by erts_debug:size/1 and ? term_size/1 quite different, because first one reports memory allocated by a term and second number of bytes for serialized term. But since both functions return consistent value for the same Acc they work for propose of this fix just fine.

+1