s.refresh() functionality?

Can someone explain the functionality of the refresh() method? Is the source code for refresh() public?

Hello @UsmanAyyaz. The refresh method redraws the graph. It is necessarily called when you want to update the graph's visual and is as such used when you move the camera around, for instance.
The source code is of course public. This is open source, after all. The method is defined here. But you might find difficult to understand everything it does without digging in some other files.

Thanks!
I am trying to implement time evolution of a graph and my initial idea was to keep growing the graph within the same sigma instance and refresh it at intervals. But since the refresh method returns the sigma instance when called, I cant update it again.
Do you have any suggestion on how to tackle this problem?

I don't understand why the fact that refresh returns your instance prevents you from calling refresh each time you need. The only reason why the method returns the instance is to enable you to chain methods.
Could you show me some of your code here so I can assess what's holding you back?

existing_nodes = []
          for(j=0; j<100; j++){
            console.log("j: ", j)
            source = temp_edges[j].source
            target = temp_edges[j].target

            var source_node = findNodes(temp_nodes, source)
            var target_node = findNodes(temp_nodes, target)

            if(!contains(existing_nodes,source)){
              s.graph.addNode(source_node) //add node
              existing_nodes.push(source)
            }
            if(!contains(existing_nodes,target)){
              s.graph.addNode(target_node) //add node
              existing_nodes.push(target)
            }
            s.graph.addEdge(temp_edges[j]) //add edge
          }

          sigma.plugins.relativeSize(s, 10)
          s.refresh();
          s.sleep(2000)

I have the above code in a for loop. It keeps adding nodes and edges in batches of 100 to the graph. At the end of each iteration I am putting the thread to sleep (to give the impression of dynamic building).
After more testing, I realized that s.refresh() is working but it only displays the final graph and not the in between building stages.

How does your sleep function work? In JavaScript, you cannot synchronously sleep without dirty tricks involving the Date object and, should you do so you might freeze the current frame and then your UI would not render until the last iteration.
You should probably use a setTimeout in your case.

function sleep(milliseconds) {
var start = new Date().getTime();
for (var i = 0; i < 1e7; i++) {
if ((new Date().getTime() - start) > milliseconds){
break;
}
}
}

Yes this is showing the exact trick you should probably never use in JavaScript code. Is there somewhere I can see your code with the setTimeout? Or can you copy the code here please?

        function build(a,b){

            for(j=a; j<b; j++){
              console.log("j: ", j)
              source = temp_edges[j].source
              target = temp_edges[j].target
              var source_node = findNodes(temp_nodes, source)
              var target_node = findNodes(temp_nodes, target)

              if(!contains(existing_nodes,source)){
                s.graph.addNode(source_node) //add node
                existing_nodes.push(source)
              }
              if(!contains(existing_nodes,target)){
                s.graph.addNode(target_node) //add node
                existing_nodes.push(target)
              }
              s.graph.addEdge(temp_edges[j]) //add edge
              }

              sigma.plugins.relativeSize(s, 10)
              s.refresh();
            }

            setTimeout(build(0,50),5000)
            console.log("BUILDING")
            setTimeout(build(50,200),5000)

First thing: you are not using setTimeout correctly. The first argument you must provide to setTimeout is a function and here you are actually passing the result of calling build(0, 50) and not a function. You should do the following:
setTimeout(function() {
  build(0, 50);
}, 5000);

// or using setTimeout's polymorphism
setTimeout(build, 5000, 0, 50);

// or by currying the build function
setTimeout(build.bind(this, 0, 50));

Thanks a lot! It worked. The visualization is running fine.

Glad I could help you!

Hi,
I think I have a similar problem with refresh not working for me when adding edges to my graph.
I mainly have a set of edges and want to dynamically draw edges between them. To do is I am using a single instance of sigma and the graph, and change the edges attributes of the graph at runtime. I can see the edges being properly created but then they are not displayed on the graph. Here is my code, what am I doing wrong?
`
var g = {
nodes: [],
edges: []
}
var s;
function clickedButton() {
UIElements.generateEdges();
UIElements.displayGraph();
}
function displayGraph() {
if(s == null) {
s = new sigma({
graph: g,
container: 'graph-container'
});
s.settings({
immutable: false,
clone: false
});
}
this.animation(s);
s.refresh();
}
}
function generateEdges() {
g.edges = [];//this.proceed();
console.log("Techniques edges called");
for(var i=0; i<languages.length; i++) {
for(var j=i; j<languages.length; j++) {
if(i != j && languages[i].technique == languages[j].technique) {
g.edges.push({
id: 'e' + languages[i].language + "-" + languages[j].language,
source: languages[i].language,
target: languages[j].language,
});
}
}
}
}
`

I figured it out, dont worry about it.
The thing to do is to add the edges to the graph directly, so instead of doing g.edges.push you should use s.graph.addEdge

Yes indeed @ncardozo. Sorry I was not able to help you sooner.

Hi,
I have a similar problem with refresh method. I have done my code to include the nodes and the edges with a for loop. Each iteration, I push my edge on the graph, and then I used the refresh method over sigma instance.
The problem is that when I run my HTML code, I don't see the refresh, but when I execute my code adding a breakpoint , I can see how the graph draw each edge at each iteration.
My number of nodes is 3669 and my number of edges is 3961.
I don't think that the amount of data damage the execution.
If any can help me... Thank you so much!!
My code is this:
var sigInstance = new sigma({
  container: 'graph-container',
});

setTimeout(function() {
		empezar(0,50);
	}, 5000);

setTimeout(function() {
	empezar(50,1000);
	}, 5000);

function empezar(a,b)
{
	for(var i=a; i<b;i+=1)
	{
		edgeAhora = edges[i];
		source = edgeAhora.source;
		target = edgeAhora.target;
		var sourceNode = findNodes(nodes,source);
		var targetNode = findNodes(nodes,target);
		build(sourceNode,targetNode);
	}
}

function build(sourceNode,targetNode)
{
	var esCiertoSource = contains(existingNodes,sourceNode);
	var esCiertoTarget = contains(existingNodes,targetNode);
		if(esCiertoSource == false){
			sigInstance.graph.addNode(sourceNode);
			existingNodes.push(sourceNode);
		}
		if(esCiertoTarget == false){
			sigInstance.graph.addNode(targetNode);
			existingNodes.push(targetNode);
		}
		sigInstance.graph.addEdge(edgeAhora);
		sigInstance.refresh();
}

function contains(existingNodes,nodeCont)
{
	var esCierto;
	if(existingNodes.length == 0)
	{
		esCierto = false;
		existingNodes.push(nodeCont.id);
	}
	else
	{
		for(var h=0;h<existingNodes.length;h+=1)
		{
			if(existingNodes[h].id == nodeCont.id)
			{
				esCierto = true;
				break;
			}
			else
				esCierto = false;
		}
		if(esCierto == true)
		{
		}
		else
		{
			esCierto = false;
		}
	}
	return esCierto;
}

function findNodes(nodes,idNodeAct)
{
	var nodeAct;
	for(var j=0; j<nodes.length;j+=1)
	{
		if(idNodeAct == nodes[j].id)
		{
			nodeAct = nodes[j];
		}
	}
	return nodeAct;
}


Override 'copy_local' in 'paket.dependencies' with values from 'paket.references' in projects

Helo.
We often build all projects of a solution to a common output directory. So we would like not to copy the nuget references with every project again and again.
So we deactivate 'Copy Local' by default in our projects. Only the project that has the major use of the package, set 'Copy Local' to true (in most cases the the adapter/proxy project).
I tried to rebuild this behaviour with paket, but without success.
In paket.dependecies is copy_local set to false. In the adapter project one of the nuget references in paket.references is set with copy_local to true. In all projects 'Copy Local' is set to false.
paket.dependencies
source http://localhost:8081/nexus/service/local/nuget/NugetPublic/  // Nexus

framework: net45
copy_local: false

# Libraries
nuget Prism ~> 5.0
nuget Prism.PubSubEvents ~> 1.0
nuget Prism.UnityExtensions ~> 5.0

paket.references
Prism.PubSubEvents copy_local: true

please try 1.5.1 - I still have to cleanup the mess I did with the commit, but I want to get feedback if it works.

Helo.
I tested with the version 1.5.6 today. It doesn't work for me. As soon as I declare copy_local:false in paket.dependecies the flag is FALSE in all projects, no matter if the reference in paket.references is set to TRUE.

Ok will check again
On May 26, 2015 7:10 AM, "Eugen Richter"<EMAIL_ADDRESS>wrote:

Helo.
I tested with the version 1.5.6 today. It doesn't work for me. As soon as
I declare copy_local:false in paket.dependecies the flag is FALSE in all
projects, no matter if the reference in paket.references is set to TRUE.
—
Reply to this email directly or view it on GitHub
https://github.com/fsprojects/Paket/issues/836#issuecomment-105396077.

Oups that helped me to discover a bigger issue. Thanks for reporting.
It works for me now with 1.6 when I do "paket install"
On May 26, 2015 7:24 AM, "Steffen Forkmann"<EMAIL_ADDRESS>wrote:

Ok will check again
On May 26, 2015 7:10 AM, "Eugen Richter"<EMAIL_ADDRESS>wrote:

Helo.
I tested with the version 1.5.6 today. It doesn't work for me. As soon as
I declare copy_local:false in paket.dependecies the flag is FALSE in all
projects, no matter if the reference in paket.references is set to TRUE.
—
Reply to this email directly or view it on GitHub
https://github.com/fsprojects/Paket/issues/836#issuecomment-105396077.

Now it works! Tested today with 1.6.2.
Thanks a lot.


Improve GetConnectorReturnTypeAsync to recursively expand return type

Add GetConnectorReturnTypeAsync with max number of network calls limit (default is 1)
Add support of OptionSets/EnumSymbols

✅ No public API change.

✅ No public API change.

✅ No public API change.

Public API changes: ❌CP0002 M:Microsoft.PowerFx.Connectors.ConnectorType.get_OptionSet


Adding catalan language

I would like to contribute by adding support for the catalan language to gruut (and gruut-ipa / ipa2kaldi) but I am not sure about the g2p model.
I have a phonetisaurus g2p model which outputs CMU phonemes and the corresponding dictionary, would that suffice or should the model output IPA phonemes? I could maybe manually map the CMU phonemes to IPA and retrain the model.
I have also seen you have extracted g2p models from espeak-ng, how could I do so? Or have you converted a lexicon to its IPA phonetic representation with espeak and then trained a g2p model based on that?

Hi @ccoreilly, thanks for offering to volunteer!
When adding a new language, my first step is to add the phonemes to gruut-ipa. These should be IPA, and I usually just use a Wikipedia page.
If you can manually map the CMU phonemes to IPA, that would be great. If you follow the convention here for English, it will be possible for gruut-ipa to convert between the CMU and IPA phonemes automatically.

I have also seen you have extracted g2p models from espeak-ng, how could I do so?

I created a small script for this. I start by creating a list of words, usually just the words from my lexicon plus a list of frequent words in the language (I have one for Catalan). Make sure to lower-case and de-duplicate the words. Then I create the espeak-ng lexicon like this:
./espeak_word.sh < words.txt > lexicon.espeak.txt

After that, converting it to a database is straightforward:
python3 -m gruut.lexicon2db --casing lower --lexicon lexicon.espeak.txt --database espeak/lexicon.db

I train separate g2p models for IPA and espeak-ng phonemes. See below for instructions on that, and let me know if you have any questions :slightly_smiling_face:
G2P
Recent versions of gruut aren't using Phonetisaurus at runtime anymore to reduce the runtime dependencies. I'm hoping to add support for reading the g2p FSTs in pure Python, but for now I'm using a different framework.
Training still needs Phonetisuarus, however, for initial alignment of the corpus. If you're using my phonetisaurus Python package, you can get this when you train a model:
phonetisaurus train --corpus g2p.corpus --model g2p.fst lexicon.txt

The g2p.corpus file contains the alignments for all words in the lexicon. You use this to train a model in my new framework like this:
python3 -m gruut.g2p train --corpus g2p.corpus --output g2p/model.crf

Thanks for the thorough response Michael! I have been a bit busy lately but will make time to contribute.

Hi Michael,
i'm trying to add new language and created model.fst and model.corpus with phonetisaurus.
Howver, when i try to run the below command to get "model.crt" with :
python3 -m gruut.g2p train --corpus g2p.corpus --output g2p/model.crf
i'm getting error as
zsh: killed python3 -m gruut.g2p train --corpus g2p.corpus --output g2p/model.crf
that's it. Any idea or troubleshooting steps to get rid of this or any other way to get model.crt ?

 Is it eating up all of your memory?

Thanks for the reply. The corpus file is of 23M size
Is it too big to train? what would be the ideal size?

@mlrober are you working on Catalan or another language? (I haven't had the time so it'd be great if your questions were specific to the catalan language :)

Hi Michael,
I'm working on another language however i put a comment on the catalin
language query.
I reduced the file size and it is done.
However, i found the "loss:" parameter contains higher no i mean i have
around 1,78,000 words and is howling all the nos in loss?
Is it intended?
Appreciate for your response.
On Fri, Nov 5, 2021 at 1:02 PM ccoreilly @.***> wrote:

@mlrober https://github.com/mlrober are you working on Catalan or
another language? (I haven't had the time so it'd be great if your
questions were specific to the catalan language :)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rhasspy/gruut/issues/8#issuecomment-961681333, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AVPVACB5YQKDZI6C2OIS7BTUKOCCBANCNFSM464HG6KA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

I guess we can consider this thread as "adding a new language" more generally :slightly_smiling_face:
@mlrober, can you clarify what "howling all the nos in loss" means? Sorry, I can't quite interpret it :confused:

Hi Michael,
Sure. I was saying that after completing model training l, I got some
results stating that scores variable is empty and loss variable is having
all no of words.
Here  I'm bit confused is the model trained properly or not?
Also, what are the steps we need to follow to train glow TTS model and how
many hours of data required?
Sorry if it goes out of context l.
Kindly let me know
Thanks,
On Sat, Nov 6, 2021, 02:28 Michael Hansen @.***> wrote:

I guess we can consider this thread as "adding a new language" more
generally 🙂
@mlrober https://github.com/mlrober, can you clarify what "howling all
the nos in loss" means? Sorry, I can't quite interpret it 😕
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rhasspy/gruut/issues/8#issuecomment-962215635, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AVPVACAL2J4J6SCIQJ5HLBLUKRHOTANCNFSM464HG6KA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.


MM-35689 Fix error caused by trying to render non-string attachment field

As far as I can tell, field values are the only thing in a message attachment that's allowed to be a string or a number, and we failed to check for that on the client before sending them into the Markdown component. I tried swapping other fields for numbers, but it seems like the server sanitizes those already even though I couldn't find exactly where that happens.
Ticket Link
https://mattermost.atlassian.net/browse/MM-35689
Release Note
Fixed error caused by a post created with a non-string attachment field.

/check-cla


Update README.md

Hey, I just made a Pull Request!
Added alpha and beta release info to techdocs roadmap. Purpose is to complete the roadmap and elevate alpha to beta migration info.
:heavy_check_mark: Checklist

[ ] A changeset describing the change and affected packages. (more info)
[X] Added or updated documentation
[ ] Tests for new functionality and regression tests for bug fixes
[ ] Screenshots attached (for UI changes)
[ ] All your commits have a Signed-off-by line in the message. (more info)

just a sign off needed from you @bodilb, then its good to go!


Remove ros .launch in CMakeList and update tutorial2.py in order to use gepetto-viewer instead of RVIZ

I made a mistake when I merged the files before my last pull request, this is the real modifications.
Sorry for the inconvenience.

Merged in branch master


Send many many push notifications

According to Apple Docs, they want you to send as many push notifications using the one connection. The documentation states that you can actually send the data to each push notification next to each other instead one at a time.
In fact the docs state that if you send many push notifications separately, it may interpret it as a DoS attack.
Is there any way to do this using your package?

I guess what I'm trying to say is how cam I satisfy this recommendation
You may establish multiple connections to the same gateway or to multiple gateway instances. If you need to send a large number of remote notifications, spread them out over connections to several different gateways. This improves performance compared to using a single connection: it lets you send the remote notifications faster, and it lets APNs deliver them faster.
Keep your connections with APNs open across multiple notifications; don’t repeatedly open and close connections. APNs treats rapid connection and disconnection as a denial-of-service attack. You should leave a connection open unless you know it will be idle for an extended period of time—for example, if you only send notifications to your users once a day it is ok to use a new connection each day.

And also this https://developer.apple.com/library/ios/documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/Chapters/CommunicatingWIthAPS.html#//apple_ref/doc/uid/TP40008194-CH101-SW3
The binary interface employs a plain TCP socket for binary content that is streaming in nature. For optimum performance, batch multiple notifications in a single transmission over the interface, either explicitly or using a TCP/IP Nagle algorithm. The format of notifications is shown in Figure 5-1.

+1

+1 for sending many push notification in one connection.

how one connection sending many?


Add functions that checks the log level desired

The functions will return true if log level is enabled I.e:
log.IsDebug() // returns true for trace and debug levels
log.IsTrace() // returns true for trace level
log.IsInfo() // returns true for trace, debug and info levels
// may a enum could be added to simplify those checks
this will help to avoid doing unnecessary, and maybe expensive, operations, for example, marshalling an object, or doing a reflection to check some condition or something else in debug mode.

I totally agree.


Iterative Doc Improvement: Auto-Matchmaking Tutorial Edition

This is a relatively small one for once. 😅
All the links and information in the tutorial itself seemed fine and correct as far as I can tell, so I just added links to the related classes/functions at the top. I also decided that going forward I would alphabetize these references at the top of the tutorials, so I made sure to also alphabetize them at the top of the previous tutorials I added them to.
And finally, I also checked the documentation for those functions and made minor corrections or clarifications to them.

Haha, all good changes! Surely you do this professionally somewhere? Great job!


Provided Bedfile is not hg19 coordinate

Hi,
It seems that the provided bed file is b37 coordinate (without "chr" before chromosome numbers), and when liftover the bed file to hg19, the program exits with:
Traceback (most recent call last):
  File "/home/user/miniconda2/envs/nanosv/bin/NanoSV", line 11, in <module>
    sys.exit(main())
  File "/home/user/miniconda2/envs/nanosv/lib/python3.6/site-packages/nanosv/NanoSV.py", line 76, in main
    utils.parse_svs.parse_svs()
  File "/home/user/miniconda2/envs/nanosv/lib/python3.6/site-packages/nanosv/utils/../utils/parse_svs.py", line 55, in parse_svs
    breakpoint.structural_variants[sv_id_2].setArguments(NanoSV.opts_depth_support)
  File "/home/user/miniconda2/envs/nanosv/lib/python3.6/site-packages/nanosv/utils/../classes/sv.py", line 159, in setArguments
    avg_dupdel_cov = self.getDupDelcoverage()
  File "/home/user/miniconda2/envs/nanosv/lib/python3.6/site-packages/nanosv/utils/../classes/sv.py", line 225, in getDupDelcoverage
    return float(sum(dupdel_coverages)) / len(dupdel_coverages)
ZeroDivisionError: float division by zero
Command exited with non-zero status 1

It looks like that the dupdel_coverages list is empty. This means there are no positions between the start and end of an sv, or the depth function returns an error. Can you try the following command, to check the depth function:
$SAMBAMBA depth base --min-coverage=0 $BAM -L $CHR:$START-$END

or is you use samtools:
$SAMTOOLS depth $BAM -r $CHR:$START-$END

I tested with the liftover bed file, sambamba output an error:
core.exception.AssertError@/gnu/store/sp0qvwcw5yhv4mi32sabzpi8jid47gh7-ldc-1.1.1/include/d/std/parallelism.d(1825): null this

The problem is then with sambamba. You can try to contact the authors of sambamba and report there this issue. Normally it should work with sambamba and/or samtools. Maybe you can try to run it with samtools?

Can you add the next line to the classes/sv.py just above line 225
print(cmd)

This will print the sambamba command (just above the error) that it try to execute and then try to execute this on the command line.
Or maybe you can shared the bam file if it's not too big?

Hi,
I added the code and it outputs a lot of commands, the last one it output before the "division by zero" crash is:
sambamba depth base --min-coverage=0 sort.bam -L chr10:72942148-72942182 --nthreads=16 2> /dev/null | awk '{if (NR!=1) print $3}'

which prints nothing, while others prints a list of numbers. Maybe there's no coverage at that region, is there a way to prevent the crash?

Hmmm strange, if there's no coverage at that region the output should be like:
REF	POS	COV	A	C	G	T	DEL	REFSKIP	SAMPLE
10	72942147	0	0	0	0	0	0	0	A
10	72942148	0	0	0	0	0	0	0	A
10	72942149	0	0	0	0	0	0	0	A
10	72942150	0	0	0	0	0	0	0	A

Can you try to run the command without catching the stderr, like:
sambamba depth base --min-coverage=0 sort.bam -L chr10:72942148-72942182 --nthreads=16

This could indeed the problem. Maybe you can try to run it with samtools.

Hi,
It turns out that Samtools updated it's BAM specification to deal with long CIGAR strings in version 1.7 (https://github.com/samtools/samtools/releases/tag/1.7), and sambamba is not compatible with the bam files samtools generated. That's what is causing the problem. After switching to samtools the program runs fine.


Translate to Chinese

Hi,
Your plan looks awesome. Would like to translate this to Chinese so more people will be able to use your plan. I created another branch and I will try to get everything translated slowly. I would also take this opportunity for me learn the Chinese words of these technical terms & words because I learned all of these in English and I would like to know each way too.

https://github.com/MachineLearning100/100-Days-Of-ML-Code

Hello @Bluesn0w  Please refer to the Chinese version of the repository and add contributions there.


Add data constants to parts

example rec:
https://lux.collections.yale.edu/data/text/614f2504-a240-4720-8e2a-d87d9bf7eb5f
blocks https://github.com/project-lux/lux-frontend/issues/309

This example is still missing AAT data constants. Debug this.

not the code, at least not this code. closing.
(YUL vending their own records here, not AATs, so data constants code can't find anything).

Re-opening to continue investigating.
YUL is vending AATs now:
https://linked-art.library.yale.edu/node/008df9b6-860d-4135-aac9-7780ed048094
But pipeline is not adding them:
https://lux-front-sbx.collections.yale.edu/data/text/614f2504-a240-4720-8e2a-d87d9bf7eb5f
This is clearly working on other records:
https://lux-front-sbx.collections.yale.edu/data/object/6a76f826-54e7-4b94-8fee-f043b29bb2b8
Is the issue related to there being multiple parts? Is there something wrong in the YUL record I am not seeing?

closing again, data/ships passing in the night


Bump repo config to v0.6.2

Can you reapprove this pls? The commit wasn't properly signed. Also can you update the branch proteciton rules here as well, don't have access.

Looks like merging a few dependabot PRs are causing a conflict here. Needs a rebase.


investigate cuda graphs + dynamic scaling leading to memory fragmentations

TODO add more context / a repro, for now just capturing the request from chat

https://github.com/pytorch/ao/issues/567


1st stage of LEX Manage Household UI

This 1st stage includes:

a visualforce page which just hosts a lightning component
3 lightning components: HH_Container, HH_Canvas, and HH_ContactCard
the visualforce page can be bound to a action/button just like the current Manage Household page
the page correctly loads, displays all contacts in Cards, and displays a set of fields on the Household
the page supports drag/drop to re-order the Cards and update the Household Name and greetings
Contact exclusion checkboxes are supported
Household auto-naming checkboxes are supported
correctly saves all changes to both contacts and the household
untested against Household objects at this point, only Household accounts
there is a separate work item to figure out using custom labels

@dospromptman @njjc Here is the first pass at the new LEX Manage Household UI Page.  this pull request is into a REVIEWED branch (not Dev)!  Not sure if you both would like to review this to learn more about Lightning Components, or just one of you.  I'm happy to give a code walk through so you aren't looking at it blind.  Let me know how you two want to proceed.  thanks!


Block Count

Hi,
I am currently using your library with PPD file format. I have noticed that when I write the document, there is no additional records consisting 9 values being added.
Block Count: Total number of records in the file (include all headers and trailer) divided by 10 (This number
must be evenly divisible by 10. If not, additional records consisting of all 9’s are added to the file
after the initial ‘9’ record to fill out the block 10.)

@wcarswell you know, I will change it right now

Thanks @afrunt for the prompt response.

@wcarswell implemented in 0.2.5 version, deployment is in progress right now

Hi Anrii,
Thank you. I can test as well and let you know


T265: A lot of USB problems on macOS

Required Info

Camera Model
T265

Firmware Version
<IP_ADDRESS>9

Operating System & Version
macOS Catalina

Kernel Version (Linux Only)

Platform

SDK Version
<IP_ADDRESS>

Language

Segment
others

Issue Description
Well, I know, T265 on macOS is not officially supported, but maybe one has a clue. I was able (finally) to compile under macOS Catalina. Tried the rs-pose sample first. It just runs very randomly.
Most the time I'm getting errors like these on attempt to start (sometimes it also breaks in the middle of a run):
~/Documents/librealsense/build/examples/pose $ ./rs-pose
17:53:46.782 [0] [E] Device-E800: FW crashed - got error in interrupt endpoint thread function: status = -1 (LIBUSB_ERROR_IO), actual = 0
17:53:47.021 [0] [E] Device-E800: State [ACTIVE_STATE] got event [ON_ERROR] ==> [ERROR_STATE]
17:53:47.021 [0] [E] Device-E800: Entered state [ERROR_STATE]
^C

~/Documents/librealsense/build/examples/pose $ ./rs-pose
17:54:01.217 [0] [E] Device-0000: MessageID 0x1006 (SLAM_6DOF_CONTROL) failed with status 0x8
17:54:01.217 [0] [E] Device-0000: Error Transferring SLAM_6DOF_CONTROL
17:54:01.217 [0] [E] Device-0000: Error: Failed setting 6dof Control (0x6)
RealSense error calling rs2_pipeline_start_with_config(pipe:0x7fe48470cbc0, config:0x7fe48470a980):
    Failed to start TM2 camera

Anybody having a clue what to do in order to make USB work better?

@bfulkers-i  I can confirm your PR https://github.com/IntelRealSense/librealsense/pull/5213 works great on macOS Catalina. I hope it will get merged soon into development and or master.
Thanks

@bfulkers-i I was applying the PR to master and I'm running into the same issues as described in #5276. Catch 22 now: Either have a working USB or a working python. Choose wisely...

@neilyoung I believe I found the issue with the development branch and python and fixed them in #5336. For convenience I also currently have that patch on t265_direct #5213


Add joshua's ppp

Codecov Report

Merging #246 (7160c5c) into master (f21100f) will increase coverage by 0.13%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master     #246      +/-   ##
============================================
+ Coverage     73.80%   73.93%   +0.13%
- Complexity      669      680      +11
============================================
  Files            96       96
  Lines          2111     2122      +11
  Branches        260      264       +4
============================================
+ Hits           1558     1569      +11
  Misses          473      473
  Partials         80       80

Impacted Files
Coverage Δ
Complexity Δ

src/main/java/quickcache/ui/MainWindow.java
0.00% <0.00%> (ø)
0.00% <0.00%> (ø%)

.../java/quickcache/logic/commands/CommandResult.java
97.56% <0.00%> (+0.50%)
28.00% <0.00%> (+10.00%)

...quickcache/model/flashcard/FlashcardPredicate.java
93.33% <0.00%> (+2.42%)
8.00% <0.00%> (+1.00%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f21100f...7160c5c. Read the comment docs.


Remove unneeded fallback logic.

It was only temporarily needed to be backwards
compatible with the old one-arg notifier interface
that the other repos were using.

Seems the specs where using the fallback logic :)

Kicked the travis build now that rspec/rspec-mocks#962 is merged.

LGTM


[TC-SU-2.2] - SDK needs to implement the features in query image response

IN [TC-SU-2.2] , This test case not verify all steps mentioned in testplan, step no- 2,3 and 6 needs to be update in SDK part.
Test step needs to be implement - QueryStatus is set to Busy,NotAvailable,ImageURI

2, 3 Fixed here: https://github.com/project-chip/connectedhomeip/pull/15286
6 out of scope for V1.0 (Already mentioned in test plan)


Layer deletion operation can be lost in case of a pageserver crash/restart

Steps to reproduce

Schedule layer deletion
Upload index_file.json change
Crash before actual deletion is done

Expected result
Layer should be deleted
Actual result
Layer wont be deleted. It is no longer in the index file and there is nothing that detects those. One option to solve it would be to persist delete intention in the index file first, then by looking at the index file after restart deletions can be retried.
Environment
Logs, links

https://neondb.slack.com/archives/C04PSBP2SAF/p1684918340070659

The same can happen during upload, if we uploaded the layer successfully but index file wasnt updated. Likely after restart there upload operation will be retried, so in this case there is no actual leakage.


Draw score because of fifty move rule when it's mate

fen: 8/8/2p5/3b1K1k/4p3/4PpR1/5P2/8 w - - 99 113
info depth 42 score cp 0 nodes 2378760 nps 2147718 time 1107 pv g3g7
info depth 43 score cp 0 nodes 1102889 nps 2175235 time 507 pv g3g7
info depth 44 score cp 0 nodes 1191410 nps 2155794 time 552 pv g3g7
info depth 45 score cp 0 nodes 214258 nps 2660628 time 80 pv g3g7

The position is mate in 1 (Rh3#) and not a draw, this happens because the mating move also allows draw by fifty move rule, however checkmate overwrites the fifty move rule.

Hi, thanks for bringing this issue up. I'm working on a bug fix and it should be pushed within a couple of hours.


Remove convey and use testify assert

Remove tests with goconvey and use stdlib test with testify assert.

LGTM

LGTM

Nice, now I can continue to push more tests that use assert. 👍


Importing @ts-rest/express in a monorepo causes module not found error

Using @ts-rest/express 3.12.1.
Node 14.19.3
React 17.0.1
Using Craco 7.0.0 to override CRA Webpack configs
I have a Lerna monorepo. I have a React (CRA) app in one workspace called Navigator, which imports a TS file from another workspace (Arachne). That TS file in Arachne imports @ts-rest/express. However, when I load the React app, I get the following error:
Failed to compile.

Module not found: Error: Package path . is not exported from package /Users/seanmcgowan/Dev/CancerIQ/umbrella2/node_modules/@ts-rest/express (see exports field in<EMAIL_ADDRESS>ERROR in ../arachne/src/middleware/ts-rest.ts 6:0-46
Module not found: Error: Package path . is not exported from package /Users/seanmcgowan/Dev/CancerIQ/umbrella2/node_modules/@ts-rest/express (see exports field in<EMAIL_ADDRESS>
I've tried updating my Webpack configuration to accommodate the module mapping, e.g.
  resolve: {
    alias: {
      "@ts-rest/express": path.resolve(
        __dirname,
        "node_modules/@ts-rest/express/src"
      ),
    },
  },

But it hasn't worked. Is there a problem with @ts-rest/express not supporting Monorepos, or am I doing something wrong? Any help would be greatly appreciated.

hey! before we try and fix the issue here, to clarify, you're trying to import @ts-rest/express from a shared library? 🧐
just checking because normally i'd expect you'd import the contract, built using @ts-rest/core from the shared module not the backend packages!

Hey! Thanks for the prompt reply. So from your question, I realised my mistake. I was placing my server/express code in the same file as my API contract, and trying to import that file into the React app. Once I split out the server code, it worked. Thanks for your help 🙏

Amazing! Great to know. If having the contract alongside your implementation is something you'd like, and if we can't offer that right now please open a new feature request issue :)


Add "topic" parameter to the "Teleop" plugin

Desired behavior

The Teleop GUI plugin does not accept any parameter.
Setting the topic by SDF would be really useful.
Alternatives considered

N/A
Implementation suggestion

The idea is to have something like this:
<plugin filename="Teleop" name="robot_name Teleop">
  <ignition-gui>
     <property type="string" key="state">docked</property>
  </ignition-gui>
  <topic>/robot_name/cmd_vel</topic>
</plugin>

Additional context

N/A

Hi@Myzhar ,
Teleop already support this. Check this line https://github.com/gazebosim/gz-gui/blob/ign-gui6/src/plugins/teleop/Teleop.cc#L139-L141.
        <plugin filename="Teleop">
          <ignition-gui>
            <title>Teleop!</title>
           </ignition-gui>
          <topic>your topic name here</topic>
        </plugin>


Update Footer.js

Related Issue
Fixes #[IssueNumber]
Closes #565
Description
[Please include a brief description of the changes or features added]
Here's a description you can use for your pull request:

PR Title: Update Footer to Include Dynamic Copyright Year
Description:
This PR enhances the footer section of the website by adding the current copyright year, which is fetched dynamically. This ensures that the copyright year will always be up-to-date without requiring manual updates each year.
Changes:

Footer HTML: Updated the footer to include a placeholder for the copyright year.
JavaScript: Added a script to fetch the current year dynamically and insert it into the footer.
CSS: Ensured that the styling of the footer remains consistent with the new dynamic content.

Benefits:

Automation: The copyright year updates automatically, reducing maintenance overhead.
Accuracy: Ensures the displayed year is always current, improving the website's professionalism.

Type of PR

[X] Bug fix
[ ] Feature enhancement
[ ] Documentation update
[ ] Other (specify): _______________

Screenshots / videos (if applicable)
[Attach any relevant screenshots or videos demonstrating the changes]
Checklist:

[X] I have performed a self-review of my code
[X] I have read and followed the Contribution Guidelines.
[X] I have tested the changes thoroughly before submitting this pull request.
[X] I have provided relevant issue numbers, screenshots, and videos after making the changes.
[X] I have commented my code, particularly in hard-to-understand areas.

Additional context:
[Include any additional information or context that might be helpful for reviewers.]

@SUGAM-ARORA done


[bug] City Supplies mission

Version: <IP_ADDRESS>
Mods: CBA, TFAR, ACE (no-medical)
Environment: MP dedi.
.rpt attatched? Partially
have you edited the missionfile?: Yes (TFAR system)
Issue:

Complete 'City Supplies' mission.
Dead then respawn.
Try to take mission again - nothing happens.

21:05:11 "antistasi-altis/log: ERROR: [BIS_fnc_taskType] Task 'LOG' does not exists."

Sidemissions have a cooldown, after some time you should be able to ask again. Petros should tell it.


Add starboard support

(part of the Carl ⇒ Ferris migration)
Implement starboard support in Ferris. When a message gets a certain number of :star: reactions, it should be sent to a specific channel (specified by env var with ID). This should support the following commands:

(mod only) ?starboard setlimit <n>: sets the required number of stars for the starboard to <n>

Open questions:

Do we want the emoji to be configurable?
Do we want to have user stats like Carl does?
Should messages which go below the limit (7 ⇒ 6 if 7 is the limit) be de-starboarded? I don't think so, because that just gives people extra notifications if it's removed and then added a few minutes later.
How do we prevent a message showing up in the starboard twice because of reactions being removed and re-added? Possible solutions:

Maintain some database table which keeps track of which messages have been starboarded. This would be necessary if we want user stats anyway.
If we don't, we could have Ferris just react to the same message with a :heavy_check_mark: or some other emoji. If it detects that from itself on that message, then it'll ignore it.

How do we deal with messages that have been starboarded by Carl previously?

One thing that's missing is the ability to exclude certain channels from starboarding. That's not super relevant for RPLCS, but it would be e.g. to ROUS if they want to use Ferris as well.

#offtopic was just blacklisted completely from the starboard. I suggested the possibility of still having a starboard-like experience in #memes-club by pinning messages that would normally be on starboard to the #memes-club thread instead.
This raises a bunch of questions which don't really belong in this issue, a new one would have to be created if this is considered for implementation.
Another approach is to have multiple starboard channels, and the possibility to redirect starred messages from specific channels to such a starboard channel.
The latter is probably the more sound option, which is why I'm commenting on this issue first before creating a new one. I'd be happy to hear your feedback.
I'm also putting myself out there as I'd be willing to implement the former suggestion. (It might take me some time)


Fix broken search code

I get "unexpected token '', JSONError" whenever I run the search courses test

Fixed! Turns out I didn't account for when query was nil


Add extended-regexp flag to grep

Your development orb has been published. It will expire in 30 days.
You can preview what this will look like on the CircleCI Orb Registry at the following link:
https://circleci.com/developer/orbs/orb/membraneframework/elixir?version=dev:5ab5fea4cc0acfe17a00c3423639e2a98e36520c

Your development orb has been published. It will expire in 30 days.
You can preview what this will look like on the CircleCI Orb Registry at the following link:
https://circleci.com/developer/orbs/orb/membraneframework/elixir?version=dev:5ab5fea4cc0acfe17a00c3423639e2a98e36520c

Your orb has been published to the CircleCI Orb Registry.
You can view your published orb on the CircleCI Orb Registry at the following link:
https://circleci.com/developer/orbs/orb/membraneframework/elixir?version=1.4.5


WIP: Add benchmark for tfidf

Add a vanilla Rust program for calculating tfidf scores and calculating the top 10 similar documents
We use a cosine similarity for calculating relevancy of documents for a given query.
@ashvardanian , thanks for your time !
Wanted to check with you on the approach to see if it makes sense to you as a valid benchmark
Use the leipzig1m and the § XL Sum datasetas corpuses.  I assumes that each 10000 lines in theleipzig1m` dataset is a single document and then calculate the tf-idf scores for each (term, document) in the corpus.

Given a query, calculate the tf-idf score of a given query based on the same corpus (and assume the query to be a separate document)

The next step is to calculate the cosine score. I couldn't fine a good source for where I can calculate the score as a vector for a query or a document (Claude gave me a fn that I could possibly use)

Once we have a cosine similarity score, sort and fetch top 10.

For benchmarking SImSIMD, I assume the cosine part is where I can use methods from SimSIMD and benchmark it against the vanilla implementation ?
And for the query, in memchr vs stringzilla you basically pick a random set of tokens and then benchmark searching from left and right using memchr and stringzilla . I was thinking of doing something similar by picking terms at random from the corpus and constructing random queries to benchmark.

Hi @GoWind! The biggest SimSIMD improvement should come from the Sparse kernels. Have you managed to use them in your TFIDF implementation?

@ashvardanian , not yet, I am trying to figure out what could be the ideal benchmark (and also learning how to use TF-IDF)
In scikitlearn, the TfIdfVectorizer creates an 2d array for the documents where each row is the document, and there is a column for each "token" in the corpus and array[row][column] = frequency
Similarly, we can tokenize the query and run intersect between the query (as a row vector) and each document in our corpus to get the intersection size.
the intersection would be if the frequency of every unique token in the query matches the frequency of the term in the document (something like using simsimd_intersect ). Is that what you had in mind using sparse kernels ?

@GoWind, the intersect function may not be the only one you need. Also look into: spdot_counts and spdot_weights 🤗

will do. Thanks for the pointers ! Also reading through the scikit implementation to see how I can possibly do this :)

Hi @ashvardanian , making progress on the tfidf based similarity calculator.
I noticed some discrepancy when calculating the cosine similarity for vec of f64s via the Rust bindings
The plain rust cosine calculations match the values I get both from numpy and from the simsimd bindings via Python
For the implementation I tried to compute a vector of tfidf_values per query and then compute a cosine similarity between the query and each document , based on the answers on the SO question
Here is how i prepared the script (added a few hacks to test quickly, will update the scripts to a be [[bench]] in the subsequent commits)
head -n 10000 leipzig1m.txt > leipzig10000.txt
cargo run --bin tfidf leipzig10000.txt

I took the first 10k lines and batched them into 10 document of 1k lines each.
The query (hardcoded) in the script is transformed into a vector representation and I compute the cosine similarities
Similarity for document via simsimd 0: Some(0.5928754241421308)
Similarity for document via plain cosine similarity 0: Some(0.40712457585786876)
Similarity for document via simsimd 1: Some(0.5993249839897541)
Similarity for document via plain cosine similarity 1: Some(0.4006750160102468)
Similarity for document via simsimd 2: Some(0.5914559242162761)
Similarity for document via plain cosine similarity 2: Some(0.408544075783724)
Similarity for document via simsimd 3: Some(0.5998267820476098)
Similarity for document via plain cosine similarity 3: Some(0.40017321795239075)
Similarity for document via simsimd 4: Some(0.5906444006555799)
Similarity for document via plain cosine similarity 4: Some(0.40935559934442023)
Similarity for document via simsimd 5: Some(0.5902192553116478)
Similarity for document via plain cosine similarity 5: Some(0.4097807446883521)
Similarity for document via simsimd 6: Some(0.5943923707602529)
Similarity for document via plain cosine similarity 6: Some(0.4056076292397477)
Similarity for document via simsimd 7: Some(0.6028015678055032)
Similarity for document via plain cosine similarity 7: Some(0.3971984321944968)
Similarity for document via simsimd 8: Some(0.5957380843868555)
Similarity for document via plain cosine similarity 8: Some(0.4042619156131435)
Similarity for document via simsimd 9: Some(0.5913356879962984)
Similarity for document via plain cosine similarity 9: Some(0.4086643120037022)

Not sure if I am doing something wrong, but could there be some sort of discrepancy here ?

Seems like one is x and the other is 1-x. One is a similarity score and the other is a distance.

Ah, I see. now it makes sense

SIMSIMD_INTERNAL simsimd_distance_t _simsimd_cos_normalize_f64_neon(simsimd_f64_t ab, simsimd_f64_t a2,
                                                                    simsimd_f64_t b2) {
    if (a2 == 0 && b2 == 0) return 0;
    if (ab == 0) return 1;
    simsimd_f64_t squares_arr[2] = {a2, b2};
    float64x2_t squares = vld1q_f64(squares_arr);
    ......
    rsqrts = vmulq_f64(rsqrts, vrsqrtsq_f64(vmulq_f64(squares, rsqrts), rsqrts));
    rsqrts = vmulq_f64(rsqrts, vrsqrtsq_f64(vmulq_f64(squares, rsqrts), rsqrts));
    vst1q_f64(squares_arr, rsqrts);
    simsimd_distance_t result = 1 - ab * squares_arr[0] * squares_arr[1];
    return result > 0 ? result : 0;
}

I was able to get a benchmark of running the cosine search across a database against a query and the SimSIMD version run upto 5x faster than the plain Rust version.
I am still not sure how to use spdot here instead of a cosine similarity. Can you give me some pointers on how I can employ spdotand benchmark it (now I have most of the foundation for a TF-IDF implementation, reckon it should be easier)
Benchmarks done on a M2 Pro with 32GB of Ram
For the dataset I used the first 10k lines from the Leipzig dataset head -n 10000 leipzig1m.txt > leipzig10000.txt
warning: `simsimd` (bench "tfidf") generated 1 warning (run `cargo fix --bench "tfidf"` to apply 1 suggestion)
    Finished bench [optimized] target(s) in 0.08s
     Running scripts/bench_tfidf.rs (target/release/deps/tfidf-601c525378540ae8)
Gnuplot not found, using plotters backend
Benchmarking TF-IDF Similarity/SimSIMD Cosine Similarity: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 6.7s or enable flat sampling.
TF-IDF Similarity/SimSIMD Cosine Similarity
                        time:   [120.97 ms 121.69 ms 122.57 ms]
                        change: [-10.060% -6.1234% -2.7896%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high mild
Benchmarking TF-IDF Similarity/Rust plain Cosine similarity: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 5.4s.
TF-IDF Similarity/Rust plain Cosine similarity
                        time:   [534.30 ms 535.29 ms 536.42 ms]
                        change: [-8.7229% -5.3702% -2.4358%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high mild

Thanks for sharing these benchmarks—great to see SimSIMD achieving a 5x speedup over the Rust implementation! To use spdot, you can directly compute the weighted dot product of TF-IDF vectors without normalizing them, as required for cosine similarity. This is both faster and aligns naturally with the sparse representation of TF-IDF.
For benchmarking, precompute the TF-IDF vectors as sparse arrays (indices for term IDs, weights for TF-IDF values) and replace the cosine similarity calculation with spdot. This avoids unnecessary operations like normalization and takes full advantage of sparsity. Let me know how the updated results compare—excited to see the numbers!

you can directly compute the weighted dot product of TF-IDF vectors without normalizing them, as required for cosine similarity. This is both faster and aligns naturally with the sparse representation of TF-IDF.
precompute the TF-IDF vectors as sparse arrays (indices for term IDs, weights for TF-IDF values)

Got it. thanks for the pointers. I will setup the benchmarks and share numbers again. This does look promising !

Hey @GoWind! I've added a placeholder file for this benchmark on the main-dev branch of this repo. Let's move it there 🤗
Thanks!

Sure, will move the benchmark code to the repo linked !

Hi @ashvardanian , I noticed that there isn't  a NEON implementation for simsimd_spdot_weights_u16 (or simsimd_spdot_counts_u16 either). I think it might make sense to add it as part of the benchmark as I am running this on a Mac (and other ARM devices might benefit anyway)
Also, I wrote the tfidf benchmark using f64 and realized that there isn't a spdot version for u16 with f32 or f64 weights either. Does it make sense to add them as well ?
Also , there seems to be simsimd native bf16 type and a half bf16 types and both atleast at the type level are not compatible. Do you think it might be safe to use the half::bf16 types and then cast them to simsimd bf16s ?


False positives that are already fixed upstream

I just found this, it's really cool, thanks for making this!
However it seems like it's not up to date with the VS Code plugin, since some syntax is incorrectly detected as wrong even though it's working in the VS Code.  An example of this is
gamerule playersSleepingPercentage 33

You can see the failed run here: https://github.com/Olexorus/PandamiumDatapack/actions/runs/1201405430
(Some of those issues are also wrongly detected in VS Code)

We have just released v1.3.4. Can you please check if the problem still occurs using it?

Updating to 1.3.4 fixed it, thank you!


[Feature]: Default widget visibility when using JS to determine visibility

Is there an existing issue for this?

[X] I have searched the existing issues

Summary
When JS is switched on for the Visible property of a widget we should get an additional Default Visibility toggle.
Why should this be worked on?
I have some widgets that are conditionally shown based on the result of a long running query. Until this query loads the widgets are in a visible state. If the should be shown they get the proper value, and if not they are hidden. I'd like them to default to hidden instead and only shown if the condition is met.
 Front conversations

@LagunaElectric this is possible using JS by defaulting the value to false right?

@Nikhil-Nandagopal The user said they tried that and it didn't resolve their issue. Instead it kept items hidden that shouldn't have been when they used || to set the default value to false.


Comments are not exported when exporting to .odt, .PDF or markdown files

I suppose figuring out a way to represent comments in markdown could be tricky, but leaving them out when exporting to rich formats like OpenDocument (ODT) and PDF is some pretty serious data loss that can undermine users' trust in the system; they may not notice until it is too late when they reopen the exported document after the etherpad instance is gone/expired...

I agree re odt, we love our libre office brothers and sisters.  But pdf....   Do Adobe even have an open spec for this?!
Does soffice.bin have support for comments in html to odt?  If so, point me at the docs and I will make sure the html we pass to soffice is compliant.
Fwiw we pass html from etherpad to soffice which is a libre office tool for converting files :)

I don't know the PDF ISO spec, but I suspect comments and annotations are very standard in there. Otherwise we wouldn't have annotation comments & highlighting in Evince with the poppler library... I hope someone smarter than me will point out where exactly this is in the spec.
As for LibreOffice being used as a converter and whether this supports comments, maybe @mmeeks would know or would be able to bring this question to the attention of someone who knows?


feat(storagetransfer): add quickstart for storage transfer service

Adds a quickstart for storage transfer service, along with a test
I have never written Go code before (ever), so apologies for any mistakes!

Addressed your comments, however running go get cloud.google.com/go/storagetransfer didn't seem to do anything

Seems like this is just pending https://github.com/GoogleCloudPlatform/golang-samples/issues/2300

Yeah, seems to be consistently failing. I made #2308 to fix.


[Remote Store] Add support to separate segment infos snapshot from metadata

Description

Currently, in remote backed index, we upload segment metadata to remote store which also contains SegmentInfosSnapshot.
This was done as part of https://github.com/opensearch-project/OpenSearch/pull/7777 to keep metadata and SegmentInfos consistent with each other.
By combining metadata and SegmentInfosSnapshot, we are combining data and metadata and this coupling will create issue s where we try to use separate remote stores for data and metadata (https://github.com/opensearch-project/OpenSearch/pull/12993)
The inconsistency problem existed as we used to overwrite metadata and SegmentInfosSnapshot files for a given segment generation.
In https://github.com/opensearch-project/OpenSearch/pull/8363, we made metadata file immutable.
Using the same concept of immutability, we can separate out SegmentInfosSnapshot and avoid the inconsistency problem.
In this PR, we provide a setting that controls if the SegmentInfosSnapshot can be combined with the metadata or not.

Related Issues
Resolves #13129
Check List

[x] New functionality includes testing.

[x] All tests pass

[ ] ~New functionality has been documented.~

[ ] ~New functionality has javadoc added~

[ ] ~Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)~
[x] Commits are signed per the DCO using --signoff
[x] Commit changes are listed out in CHANGELOG.md file (See: Changelog)
[ ] ~Public documentation issue/PR created~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@sachinpkale can we please tag this to an opensearch release?

[Storage Triage - attendees 1 2 3 4 5 6 7 8 9 10 11 12]
@sachinpkale Lets resolve the conflicts and update the new release target with this PR


NH-62591 Upgrade APM Python to Otel 1.21.0/0.42b0

WIP

Closing this, as dependabot will make PRs for these upgrades now


drop  from dependency-check-maven plugin configuration

Dear Jeremy,
warm thanks for your brilliant work in the last years, this plugin is a must-have.
Our Goal
We try to avoid any connection to untrusted (external) resources in our CI process.
For some few exceptions we allow accessing external servers by a proxy. The proxy must be declared for each use case separately, so we are aware of the risc and know which component is using the proxy for what kind of resource.
Current Solution for the dependency-check-maven plugin
The plugin uses one of the  from the Maven settings.xml.
The targeted proxy must be set  as you correctly described in https://groups.google.com/g/dependency-check/c/8p_0j4oMcAA/m/otagX2LRAwAJ.
The Problem
There are some other Maven plugins that look in the Maven settings.xml for active proxies and use them automatically, e.g. the https://github.com/eirslett/frontend-maven-plugin/ (but there may be more).
If we configure active proxies, we loose control and may silently access untrusted sources.
Describe the solution you'd like
The proxy definition should be part of the dependency-check-maven configuration and not used for anything else.
Describe alternatives you've considered
The dependency-check-maven plugin might pick up a named inactive proxy definition from the Maven settings.xml, but other plugins could scan for those proxy definitions too (the frontend-maven-plugin doesn't).

Looked in the source code and found https://github.com/jeremylong/DependencyCheck/blob/969892a677e49e8c1d71cc6d65eac8a9400de169/maven/src/main/java/org/owasp/dependencycheck/maven/BaseDependencyCheckMojo.java#L2343
I'm using Maven 3.8.6 and tried many times: the plugin picks up the proxy with the matching  only if it's set  too - strange.

I'm a little confused on the ask. This should be possible by setting the maven settings proxy id?

Damn, I should improve my english.
No matter if you configure the <mavenSettingsProxyId> or not: in Maven 3.5 and 3.8 org.apache.maven.settings.Settings.getProxies() will return only proxies set to <active>true</active>. Therefore org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.getMavenProxy() may not pick up an inactive proxy.
--> We have to provide 2 active proxies like suggested in https://groups.google.com/g/dependency-check/c/8p_0j4oMcAA/m/otagX2LRAwAJ
--> the https://github.com/eirslett/frontend-maven-plugin/ will automatically use this proxies
--> we would open a hole when doing this

TBH it sounds like things are working as designed for Maven, but don't match your very specific case.
Have you considered using profiles to switch/activate proxies or having an internal mirror of the nvd so you don't need any proxy at all?....

The proxy determined in org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.getMavenProxy() is used in more components, for example in the CentralAnalyzer doing a SHA search (that is not applicable to a mirror).
Yes: I understand the reason behind hiding inactive proxies in Maven.
But: I don't think my use case is special:
Having limited internet access is caused by security concerns. And I want to allow exceptionally internet access to trusted components.
I want to provide access for the trusted dependency-check-maven plugin.
I don't trust every other plugin and therefore don't want to provide a globally available proxy definition.
I don't see how I may activate a Maven profile for a specific plugin only.

Only run the plugin when the profile is active? Honestly - if you have a plugin you don't trust why is it in your build at all?

I don't talk about "my build" but hundreds of pipelines from different projects and teams. I want them to manually activate the proxy for selected plugins.
Providing the active proxy only when a specific Maven profile is active, will result in tons of pipelines where the profile is always active - developers tend to take the shortest path.

If you don't trust a plugin don't use it. If it was really malicious the plugin could search for proxies (inspecting the settings.xml, even proxies that are not active, etc.). In its current state - the ODC plugin is working as expected, following Maven standards.
While I understand the concern raised - solving the issue on a plugin by plugin basis is the incorrect solution. I would suggest working with the Maven team to see if they are willing to re-architect how plugins and proxies work so the concern can be solved at an ecosystem level as opposed to one off solutions (which would be trivial to bypass for a malicious plugin).


loading local GLTF data at runtime

Hello, thanks for the wonderful work!
I have a bit of volumetric data saved as GLTF files along with JSON file i.e. x,y,lod to the corresponding GLTF File, I would like to load them on runtime directly from disk. I have handwritten logic that can tell me which GLTF files to load based upon location. Can someone help me how I can integrate it with FloatingPawn location & fetch & show GLTF files on runtime. At this moment, I am mostly interested in loading these files at runtime. Would appreciate if someone can help.

@preetishkakkar It might make sense to convert your data to 3D Tiles rather than using your custom LOD format.
Cesium for Unreal currently does not allow glTFs to be loaded independently of 3D Tiles, but it would be relative easy to add. I wrote #390 to cover that. I'm going to close this issue, and I'd suggest taking it to the Community Forum if you're interested in discussing the use of 3D Tiles for your use-case.


GORM + NR: is not possible to see the executed SQL query?

Hi Smacker; thanks for your work :)
We're enhancing our NR integration and we found out that it seems that is not possible to log the executed SQL query (SELECT... ) in NR; can you confirm this?
From what we understood it's a GORM limitation, right?
Thanks

Hi @runa !
The query itself should be logged but not the values in the query as it will cause security issues by leaking sensitive information such as passwords, credit cards numbers and so on.

Hi @smacker.
Did you prove it in the new versions of GORM? I ask for this statement by one of the creators.
https://github.com/jinzhu/gorm/issues/1302#issuecomment-294737745
Thanks

The lib runs tests on gorm v1.9.12 which is the latest available version of gorm.
The issue you mentioned is from 2017, I believe the code has changed since then.


Docker compose for running all services

hmm what version docker are you running? I got this

invalid interpolation format for services.pinning_service-api.environment.REGEN_CLI_COMMAND: "docker run --rm -i --network host -v $(pwd)/localnet/keys:/regen-keys ghcr.io/cambiumteam/regen-cli:latest". You may need to escape any $ with another $.

with
$ docker version
Client: Docker Engine - Community
 Version:           20.10.18
 API version:       1.41

Easy fix adding the additional $ but want to make sure that doesn't break your environment

Add network_mode: host to docker-compose services
Configure regen binary via regen config ... : https://github.com/regen-network/regen-ledger/commit/3ee3aa3d8e9d831876199546341e90e81a9ae994#diff-8ff22b3bf42a9b5085fea655c3a67f883d67838332f7051b86b12fbefbf3891bR37-R42
Add service key in entrypoint command


Transaction author is not applied to the context for some functions

Fix application of transactionAuthor to repository context in all editing functions
Update tests to verify transactionAuthor is present in history when used

Codecov Report
Patch coverage: 96.55% and project coverage change: +0.44 :tada:

Comparison is base (5b4681b) 85.45% compared to head (60d25c9) 85.89%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #17      +/-   ##
==========================================
+ Coverage   85.45%   85.89%   +0.44%
==========================================
  Files          20       20
  Lines        1588     1659      +71
==========================================
+ Hits         1357     1425      +68
- Misses        231      234       +3

Impacted Files
Coverage Δ

.../CoreDataRepository/CoreDataRepository+Batch.swift
84.21% <75.00%> (-1.25%)
:arrow_down:

...s/CoreDataRepository/CoreDataRepository+CRUD.swift
92.30% <100.00%> (+0.48%)
:arrow_up:

...CoreDataRepositoryTests/BatchRepositoryTests.swift
99.23% <100.00%> (+0.10%)
:arrow_up:

.../CoreDataRepositoryTests/CRUDRepositoryTests.swift
93.00% <100.00%> (+0.39%)
:arrow_up:

Tests/CoreDataRepositoryTests/CoreDataStack.swift
98.33% <100.00%> (+0.02%)
:arrow_up:

...s/CoreDataRepositoryTests/CoreDataXCTestCase.swift
84.00% <100.00%> (+2.75%)
:arrow_up:

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


[SPARK-9833] [yarn] Add options to disable delegation token retrieval.

This allows skipping the code that tries to talk to Hive and HBase to
fetch delegation tokens, in case that somehow conflicts with the application
being run.

Merged build triggered.

Merged build started.

Test build #40654 has started for   PR 8134 at commit 3c4c683.

Test build #40654 timed out     for PR 8134 at commit 3c4c683     after a configured wait of 175m.

Merged build finished. Test FAILed.

retest this please

Merged build triggered.

Merged build started.

Test build #40685 has started for   PR 8134 at commit 3c4c683.

Test build #40685 timed out     for PR 8134 at commit 3c4c683     after a configured wait of 175m.

Merged build finished. Test FAILed.

retest this please

Merged build triggered.

Merged build started.

Merged build finished. Test FAILed.

retest this please

Merged build triggered.

Merged build started.

Test build #40808 has started for   PR 8134 at commit 3c4c683.

Test build #40808 timed out     for PR 8134 at commit 3c4c683     after a configured wait of 175m.

Merged build finished. Test FAILed.

sigh. retest this please

Merged build triggered.

Merged build started.

Test build #40834 has started for   PR 8134 at commit 3c4c683.

Test build #40834 timed out     for PR 8134 at commit 3c4c683     after a configured wait of 175m.

Merged build finished. Test FAILed.

retest this please

Merged build triggered.

Merged build started.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40908/
Test FAILed.

I'm not against this just wondering what the use case is here? Did you actually run into an issue?

Did you actually run into an issue?

Yes actually. Hive-on-Spark uses Spark to execute jobs, but not for metastore access. The inherited configuration sometimes has references to classes that are not in the Spark classpath (HS2 has access to those classes), and this code will try to load those classes and fail.
You could add these classes or filter the configuration, but the fact is that this application has no need for this feature, and there's no explicit way to disable it right now.

retest this please

Merged build triggered.

Merged build started.

Test build #40918 has started for   PR 8134 at commit 3c4c683.

Test build #40918 has finished for   PR 8134 at commit 3c4c683.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40918/
Test PASSed.

changes lgtm. Only thing would be if we want to make some common config name spark.yarn.access.* or other for everything access/token/security related.
@sryza  thoughts

Merged build triggered.

Merged build started.

Test build #41143 has started for   PR 8134 at commit 119d060.

Test build #41143 has finished for   PR 8134 at commit 119d060.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41143/
Test FAILed.

@tgravescs I chose a slightly different name than you suggested, how does that sound?

+1, looks good. go ahead and commit.


why net_worth charts are different between posted article and github notebook file?

hi,
I appreciate the TensorTrade_Tutorial.ipynb file got fixed, and I may run it myself. However, I have a question about the content. The networth chart, what we really care about as the final result, is different between the one in the Medium article (title: "Trade and Invest Smarter -- The Reinforcement Learning Way) and the one (actually two) in TensorTrade_Tutorial.ipynb file.
The networth chart in Medium article, rise slowly and smoothly. Refer to MediumArticle.png
And the two networth charts in TensorTrade_Tutorial.ipynb, just fall all the way (1st one) or rise shortly then fall. Refer to github-chart1.png and github-chart2
So what went wrong? What is different? It is not easy for us to choose a trading strategy to all the way lose, unless the tradebot completely trade OPPOSITE TO THE OPTIMIZED STRATEGY. Do you mind to take a look?

.png

One of the primary differences is in the episodes that the neural network
agent was able to train on. The example in the source is set to train for
10 episodes so that it can finish within a demo period. The agent you see
represented in the final image was tuned over many thousand episodes. This
of course takes time that a demo would not have (weeks or more depending on
how you train it).
The framework has also undergone somewhat of a surge of development
recently as people have joined the platform. We are still working out the
kinks as we race towards Alpha. The 2.0rc1 that is on the master branch now
is substantially different than the 0.16 version that existed 2 weeks ago.
Thanks,
Dallas
Professional leader of highly performant engineering teams since 1997
512.923.4426 | Github  https://github.com/codeninja| Twitter
https://twitter.com/dallaspool| Linkedin
https://linkedin.com/in/dallaspool
On Fri, Nov 1, 2019 at 10:56 PM Thomas Mao<EMAIL_ADDRESS>wrote:

hi,
I appreciate the TensorTrade_Tutorial.ipynb file got fixed, and I may run
it myself. However, I have a question about the content. The networth
chart, what we really care about as the final result, is different between
the one in the Medium article (title: "Trade and Invest Smarter -- The
Reinforcement Learning Way) and the one (actually two) in
TensorTrade_Tutorial.ipynb file.
The networth chart in Medium article, rise slowly and smoothly. Refer to
MediumArticle.png
And the two networth charts in TensorTrade_Tutorial.ipynb, just fall all
the way (1st one) or rise shortly then fall. Refer to github-chart1.png and
github-chart2
So what went wrong? What is different? It is not easy for us to choose a
trading strategy to all the way lose, unless the tradebot completely trade
OPPOSITE TO THE OPTIMIZED STRATEGY. Do you mind to take a look?
[image: github-chart2]
https://user-images.githubusercontent.com/6857537/68065653-51f54c80-fd67-11e9-855a-132b96aa1ad8.png
[image: github-chart1]
https://user-images.githubusercontent.com/6857537/68065654-51f54c80-fd67-11e9-8811-95bef77d3840.png
[image: MediumArticle]
https://user-images.githubusercontent.com/6857537/68065655-51f54c80-fd67-11e9-8af6-a7139684c03d.png
.png
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/notadamking/tensortrade/issues/86?email_source=notifications&email_token=AAADUQQBIESB4SYPQCPTFZDQRT3ATA5CNFSM4JICP4D2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HWISIIA,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAADUQROOPGOBBZ4BZFCQZDQRT3ATANCNFSM4JICP4DQ
.

Hi Dallas:
I just updated everything, from github tensortrade, including the TensorTrade_Tutorial.ipynb file. I changed steps to episodes(=50), but still get similar result, as attached.

Daily result just come out. It is a bit better, but eventually still
.....

50 episodes is not nearly enough training for a production system. Train
for 1000 episodes and your results should improve to production performance
levels.
As with any framework, the outcome will be dependent on how you implement
your network, your rewards, and your strategy. Also, you must test your
network on data which it has never seen before to ensure it's performance
is not overfit to the training environment. (FBMExchange will do this
automatically every iteration.)
Thanks,
Dallas
Professional leader of highly performant engineering teams since 1997
512.923.4426 | Github  https://github.com/codeninja| Twitter
https://twitter.com/dallaspool| Linkedin
https://linkedin.com/in/dallaspool
On Sat, Nov 2, 2019 at 10:28 AM Thomas Mao<EMAIL_ADDRESS>wrote:

Daily result just come out. It is a bit better, but eventually still
.....
[image: github-chart2episodes50]
https://user-images.githubusercontent.com/6857537/68073213-79c8cc80-fdc8-11e9-909d-c6d49f9a73db.png
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/notadamking/tensortrade/issues/86?email_source=notifications&email_token=AAADUQTWV5F6QJOA36TZ4TDQRWMC7A5CNFSM4JICP4D2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC46MJQ#issuecomment-549053990,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAADUQQFIP3ZGBNRTPPCYVLQRWMC7ANCNFSM4JICP4DQ
.

I get it. However, the difference between the article and Jupiter notebook file is misleading and confusing. Anyway, I look forward to the framework to become available in the near future. Thank you for your hard work and contribution.


new command to run tests for pysparse

This fixes the pysparse tests, ready for review and merge.

will re-roll this one

@eric-wieser do you happen to know if this PR is good?

This line does seem to indicate so:
https://github.com/pydata/sparse/blob/master/ci/azure-steps.yml#L13

@hameerabbasi is the person to ask here

You need pip install -e .[tests] in addition.

Ah, I see that you already have pip install -e .[all]. This should be good.

@eric-wieser @hameerabbasi thanks for your feedback!

Thanks for the fixes/checking etc. Looks good.

@stuartarchibald thanks for merging!


ESP32 Uart uart_esp32_irq_tx_ready

The modbus serial uart_cb_handler callback function handles all uart interrupt events. In recv state (rx irq enable, tx irq disable), cb_handler_rx obtains the data, and then judges uart_irq_tx_ready (esp32 uart is realized as judgment tx fifo len> 0). RS485 feature, at this time fifo len is 0, the data just received will be It was sent immediately, causing it to fail to receive normally.
https://github.com/zephyrproject-rtos/zephyr/blob/c9a3dae6519fc788d33233fc1c4d1364ba129034/subsys/modbus/modbus_serial.c#L422
https://github.com/zephyrproject-rtos/zephyr/blob/c9a3dae6519fc788d33233fc1c4d1364ba129034/drivers/serial/uart_esp32.c#L394
The uart_esp32_irq_tx_ready in the esp32 uart driver should be modified to:
static int uart_esp32_irq_tx_ready(const struct device *dev)
{
    return (uart_hal_get_txfifo_len(&DEV_CFG(dev)->hal) > 0 && uart_hal_get_intr_ena_status(&DEV_CFG(dev)->hal) & UART_TXFIFO_EMPTY_INT_ENA);
}

@huarunlin, correct. I think that UART_TXFIFO_EMPTY_INT_ENA should be replaced to UART_INTR_TXFIFO_EMPTY. I am submitting this.

@sylvioalves I tried to profile UART_TXFIFO_EMPTY_INT_RAW before and found that UART_TXFIFO_EMPTY_INT_RAW is still flagged when UART_TXFIFO_EMPTY_INT_ENA is disabled. Just now I re-read the manual and esp-idf, and found that the correct way should be to evaluate the UART_TXFIFO_EMPTY_INT_ST flag. If you use uart_hal_get_intsts_mask() to evaluate UART_INTR_TXFIFO_EMPTY, that should be correct.

@huarunlin, correct. I think that UART_TXFIFO_EMPTY_INT_ENA should be replaced to UART_INTR_TXFIFO_EMPTY. I am submitting this.

@sylvioalves I tried to profile UART_TXFIFO_EMPTY_INT_RAW before and found that UART_TXFIFO_EMPTY_INT_RAW is still flagged when UART_TXFIFO_EMPTY_INT_ENA is disabled. Just now I re-read the manual and esp-idf, and found that the correct way should be to evaluate the UART_TXFIFO_EMPTY_INT_ST flag. If you use uart_hal_get_intsts_mask() to evaluate UART_INTR_TXFIFO_EMPTY, that should be correct.

@huarunlin, correct. I think that UART_TXFIFO_EMPTY_INT_ENA should be replaced to UART_INTR_TXFIFO_EMPTY. I am submitting this.

@sylvioalves I tried to profile UART_TXFIFO_EMPTY_INT_RAW before and found that UART_TXFIFO_EMPTY_INT_RAW is still flagged when UART_TXFIFO_EMPTY_INT_ENA is disabled. Just now I re-read the manual and esp-idf, and found that the correct way should be to evaluate the UART_TXFIFO_EMPTY_INT_ST flag. If you use uart_hal_get_intsts_mask() to evaluate UART_INTR_TXFIFO_EMPTY, that should be correct.

@dkalowsk can you check this? #41892


Unable to load image for FEED

I'm facing an issue sending a share intent to Instagram on Android, from the last update from 3 days ago. This is on an Ionic app
I can reproduce the error in the following conditions:
Android >= 8
Last Instagram Version (<IP_ADDRESS>.120)
Only sharing to Feed, Stories read the image as expected
I will not add the code I'm using to share because seems to be another thing, some change that Instagram introduced in the last update, maybe permissions or something else.
I discard the image path (read/write) because for Stories it's working fine.
Any ideas? Thanks.

Same here.
Android >=8
Instagram Version (<IP_ADDRESS>.120)
Sharing to feeds, direct fails.  Debugging the issue it looks like the error message returned is ShareCancelled

This all happened when Instagram updated to their latest version

Same issue here, was working fine before Instagram version <IP_ADDRESS>.120.  Instagram shows message "Unable to load image"  then kicks out. Same error as @hdeshmukh, any ideas how to troubleshoot this one?

Hello guys. I'm trying to fix this issue but not lucky yet. I could say that tried with different images, sizes, dimensions, and formats, and all fails the same. So the problem should be related to anything else, maybe permissions.
This is weird because Stories work like a charm :/

Weird thing is I cannot reproduce the issue on Android 6.  Everything seems to be working fine on an older device.

Yes, meanwhile email to<EMAIL_ADDRESS>maybe we get their attention

I emailed them as well, if I get any useful feedback will post back to this thread.

@souvenirpixels @hdeshmukh hello guys! IG just release a new version for Android, could you check if is working now for you or still have the issue? Thanks

Just checked, still not working with the new version (<IP_ADDRESS>.121), also haven't gotten any response from Instagram yet.

Hello guys @souvenirpixels @hdeshmukh, there is a ticket on Facebook bugs section: https://developers.facebook.com/support/bugs/1326888287510350/
Can you leave a message there?
Thanks

Thanks, I left a message and following the thread.  There is also a slackoverflow thread that suggests a work around may be writing to the media store then posting from there will work.
https://stackoverflow.com/questions/60786976/share-intent-issue-in-instagram-feed
Haven't been able to confirm if this is a possible fix.

Thanks! I will try to do the same workaround in Java, but i'm a frontend developer há please add information here if you have

Workaround solution to fix temporary this IG bug (I think is a bug from the Instagram app)
https://stackoverflow.com/questions/60776730/instagram-android-share-intent-unable-to-load-image

Can confirm this problem.
This explains the increase in installs and uninstalls of my app... (probably all apps are broken)
Instagram version: <IP_ADDRESS>.1
Android 9

It was a prolem with instagram app, other applications as whatsapp also had the same problem. After some weeks, i tested and share function is working.

Yeah, its working again.


Can someone explain to me about hint in level "Branch Spaghetti"?

The hint says  "Make sure to do everything in the proper order! Branch one first, then two, then three"
and the answer is
git checkout one;
git cherry-pick C4 C3 C2;
git checkout two;
git cherry-pick C5 C4 C3 C2;
git branch -f three C2;

But doesn't it matter the order?
ex.
git checkout two;
git cherry-pick C5 C4 C3 C2;
git checkout one;
git cherry-pick C4 C3 C2;
git branch -f three C2;

order is important,
look at the show goal,  HEAD must be point to branch two.


Create more meaningful issue templates

Description
Currently only one issue template exists and it is very ISSUE focused.
Steps to reproduce
Click New Issue button on GitHub repo.
Current result
A single issue template is used based on ISSUE_TEMPLATE.md.
Desired result
The user should be able to choose from several issues types (e.g., Bug Report, Feature Request, Help & Other Questions)
Affected version

[x] 9.2.1
[x] 9.2
[x] 9.1.1
[x] 9.1
[x] 9.0

Affected browser

[x] Chrome
[x] Firefox
[x] Safari
[x] Internet Explorer
[x] Edge

We will be submitting a PR shortly for this.


New provider: DigitalOcean.

To use this provider, one additional environment variable is required:

DO_PAT: DigitalOcean personal access token, required to update DNS.

The root domain name needs to be defined as well, of course.
Adding this provider required adding some additional packages to Godeps:

github.com/digitalocean/godo
github.com/tent/http-link-go
golang.com/x/oauth2

One line in main.go to enable the new provider, and the provider itself.
I think I understand what's going on with Docker with this package now,
but I could be wrong so let me know what I need to change!

@janeczku I understand you're the person to ping -- if that's not the case, please let me know who! :-)

Thanks a lot for the PR @mathuin. Looks like you committed the vendored libraries as submodules. Could you fix that and also update Godeps.json with the libraries?
godep save ./... should do that for you.

When I tried godep save ./..., godep said the package github.com/pkg/errors was not included.  I ran go get github.com/pkg/errors and that retrieved the package and stuck it in Godeps/_workspace.  I don't know which if any packages use "pkg/errors".
If you could make sure I didn't screw something up with godep I'd appreciate it, as this is my first time with this tool.

@mathuin Looks like the godepstuff went fine. I have added two minor comments. Thanks!

@janeczku I looked for your comments and couldn't find them.  If they were in the code making it easier to understand, then that's awesome -- but if they were on GitHub to review the pull request, I don't know where they are.  Let me know if there's anything specific you need me to do to make this PR acceptable!

@mathuin sorry my fault i forgot to submit the review. Comments should be visible to you now.

I have addressed both your comments in the recent commit.
When testing this change, I noticed a number of TXT records that I hadn't seen when originally developing the plugin.  external-dns-adminproject is the domain name and the value appears to be a list of domain names for which external-dns is maintaining records.  The TTL in the container logs is 300 which is the default from the environment.  From this information I was able to identify the section of code where these records were being created: metadata/metadata.go has a method on the MetadataClient object which gets DNS records for containers.  At the end of that method, it creates StateRecord entries and calls a local function addToDnsEntries which converts those to DnsRecord entries.
The reason I noticed this is because my domain now has dozens of these TXT records.  Only three are retrieved by dig, but the DO interface is absolutely spammed with them.  It is likely a bug in their interface, and I will ask them about it. :-)

I decided to remove the TTL from the client and use config.TTL instead, now that I know it's being used elsewhere.  I'm removing 425 of these TXT records.  When it's done, I'll test my change.  If it works, then I'll update the branch.

@janeczku this is now ready for review.  Thank you for your patience!

@mathuin thanks for your work on this! As for the state TXT records unfortunately we have to create separate records for each service domain because of the 255 chars limit for a single TXT record.
With Route53 you can easily filter those records in the UI. Maybe you can get DO to implement that functionality as well 😄


Add explicit queue flag to the agent

It's currently hidden under tags, but it's a very important part of the agent lifecycle, and deserves its own flag. At current, it just gets munged into the agent's tags in the register request as it would have previously, but having --queue makes it more explicit
Description
which queue the agent listens for work on is an important part of the agent's lifecycle, especially with queues being made more explicit as part of our work to make clusters the default in buildkite. however, getting the agent to listen on a particular queue is a bit of an odd experience, it's a super special option in the --tags flag
buildkite-agent start /* ... */ --tags "queue=deploy"

this PR brings queue up to being a first-class citizen in the agent's CLI, giving it its own flag, so now it's buildkite-agent start --queue deploy. much nicer!
behind the scenes, we're still munging the queue into the tags that the agent sends to the buildkite mothership when it registers, as that's what the backend expects, but the CLI flag makes things nicer to use.
Testing

[x] Tests have run locally (with go test ./...). Buildkite employees may check this if the pipeline has run automatically.
[x] Code is formatted (with go fmt ./...)

What should people who are used to setting multiple queues do?
If someone passes both --queue and queue tags, this will override the first queue= tag, but I think any further queue tags will not be overridden?

yeah, i wondered about that when writing it... maybe we should just exit if both --queue and --tags "queue=..." are set?

What should people who are used to setting multiple queues do?

That whole doc page is highly misleading when it comes to cluster queues.
Things like:

If you don't configure a queue for your agent by setting an agent tag of queue=my_example_queue it will accept jobs from the default queue as if you had set queue=default.

are flat out not true for cluster queues.
IIRC we plan to deprecate it soon. Should we until then to merge this?

Should we [wait] until then to merge this?

i think this is probably fine to merge now, and then later on down the track we can crack down on multiple queues. i think it'd probably have to be a major version, and we'd have to have everybody on clusters, so i'm keen to bias for action here


[Suggestion] Add permission to follow-emerald-blocks (by villagers)

Add permission to follow-emerald-blocks (by villagers)

Something like this is best implemented as a plugin. Permission checks can be heavy, especially on a pathfinder goal that can potentially trigger multiple times per tick. Villagers are already laggy enough as-is, so I will not include this into the server itself.
For anyone wanting this feature here is a small plugin to handle it (note that it has the potential to add more villager lag).
Use permission node villager.follow.emeralds
public final class FollowEmeraldPermission extends JavaPlugin {
    @Override
    public void onEnable() {
        getServer().getPluginManager().registerEvents(new Listener() {
            @EventHandler
            public void onVillagerFollowPlayer(EntityTargetLivingEntityEvent event) {
                if (event.getEntityType() != EntityType.VILLAGER) {
                    return;
                }

                LivingEntity target = event.getTarget();
                if (target.getType() != EntityType.PLAYER) {
                    return;
                }

                Player player = (Player) target;
                ItemStack hand = player.getInventory().getItemInMainHand();
                if (hand.getType() != Material.EMERALD_BLOCK) {
                    hand = player.getInventory().getItemInOffHand();
                    if (hand.getType() != Material.EMERALD_BLOCK) {
                        return;
                    }
                }

                if (!player.hasPermission("villager.follow.emeralds")) {
                    event.setCancelled(true);
                }
            }
        }, this);
    }
}

FollowEmeraldPermission.zip


Generator converging to wrong output

Hey Tao,
I am trying to implement your rationale model in pytorch right now and I keep running into the problem, that after a couple iterations z becomes all one's. This obviously makes the encoder quite strong but does not do what I want.
The generator's loss function is cost(x,y,z) * logp(z|x). While the first term is large, logpz becomes all zeros (since the model learns to predict all ones with 100% prob and log(1) = 0). Therefore, the overall loss (and derivative) for the generator becomes zero, leading to this all one's phenomenon.
How did you address this in your code?

Hi @sebastianGehrmann
The issue is probably due to one of the following reasons:

The cost function includes an "sparsity" regularization that tries to suppress the selection too many one's. In my current implementation, the weight of this regularization (--sparsity) has to be carefully tuned, so neither the model selects all one's nor it selects all zero's.

Since the learning procedure samples the gradient via REINFORCE, the variance of gradient is high and the model sometimes suddenly "jumps" to the bad optima of selecting all one's or zero's. I saw this more often for the dependent-selection version. To alleviate this, I used a larger batch size of 256 and a smaller initial learning rate. The code also monitors the cost value on train and dev sets. If the cost jumps after one epoch, I redo the parameter changes of this epoch and halve the learning rate. See this.
For general REINFORCE and reinforcement learning, there are more principled ways of reducing the gradient variance. One is called "baseline trick". See Jiwei's follow-up paper (page 8).

Hope this can help!


implicit_clone does not play nice with indexmap 2.26

Problem
If yew's indexmap dependency is upgraded to 2.2.6 the following error can be seen when building:
error[E0277]: the trait bound `implicit_clone::unsync::IMap<_, _>: From<IndexMap<K, V>>` is not satisfied
   --> packages/yew/src/html/conversion/into_prop_value.rs:277:9
    |
277 |         IMap::from(self)
    |         ^^^^ the trait `From<IndexMap<K, V>>` is not implemented for `implicit_clone::unsync::IMap<_, _>`
    |
    = help: the following other types implement trait `From<T>`:
              <implicit_clone::unsync::IMap<K, V> as From<indexmap::map::IndexMap<K, V>>>
              <implicit_clone::unsync::IMap<K, V> as From<std::rc::Rc<indexmap::map::IndexMap<K, V>>>>
              <implicit_clone::unsync::IMap<K, V> as From<&'static [(K, V)]>>
              <implicit_clone::unsync::IMap<K, V> as From<&implicit_clone::unsync::IMap<K, V>>>

I see that From<IndexMap<K, V>> is implemented here so I'm definitely misunderstanding something. Any pointers appreciated!
Steps To Reproduce

Update packages/yew/Cargo.toml to:

diff --git a/packages/yew/Cargo.toml b/packages/yew/Cargo.toml
index 27c6c545..4dc03941 100644
--- a/packages/yew/Cargo.toml
+++ b/packages/yew/Cargo.toml
@@ -19,7 +19,7 @@ rust-version = "1.64.0"
 [dependencies]
 console_error_panic_hook = "0.1"
 gloo = "0.10"
-indexmap = { version = "2", features = ["std"] }
+indexmap = { version = "2.2.6", features = ["std"] }
 js-sys = "0.3"
 slab = "0.4"
 wasm-bindgen = "0.2"

Run cargo build

Expected behavior
cargo build is successful
Environment:

Yew version: master
Rust version: 1.77.0

Questionnaire

[x] I'm interested in fixing this myself but don't know where to start
[ ] I would like to fix and I have a solution
[ ] I don't have time to fix this right now, but maybe later

I get that problem from time to time,  executing a  cargo update will fix it. No idea why.

If cargo update fixes it then the reason for this error is that implicit_clone's dependency in the lockfile isn't updated when you modify the Cargo.toml. You have to tell cargo to update indexmap everywhere

Thanks for the suggestions, that fixed it!


Torch 1.0.1

You have put in some great work! :) We recently updated the package for PyTorch v1.3, if there are certain features that you think may be useful to all users of deepmatcher, it'll be great if you could create more modular pull requests for specific components, I'll be happy to merge them to master.

You have put in some great work! :) We recently updated the package for PyTorch v1.3, if there are certain features that you think may be useful to all users of deepmatcher, it'll be great if you could create more modular pull requests for specific components, I'll be happy to merge them to master.

Hi, I really think that a big improvement would be to allow users to load a dataset directly as a Pandas.DataFrame instead of loading it from csv files. I've already done some work but it's still incomplete and it's a kind of hard fix


feat: add question to provider

Related issue: https://github.com/odpf/guardian/issues/268
This is the PR to cover second requirement

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 74.656%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
6024

Relevant Lines:
8069

💛  - Coveralls

@haveiss we'll need to validate the appeal if it passes the required parameters. This should be done in providerService.ValidateAppeal


Why OnPaintSurface method is called so may times ?

on SKCanvas I am drawing multiple bitmaps and let user scale, drag and rotate that images.
Everything works fine but OnPaintSurface is called 2 times for 1 bitmap. Because of this, my performance gets slow after some time.
These things are decided by OS from device or are managed in plugin itself ?
Please help !

I think that I may need a bit more information. The paint event will get fired each time something invalidates the underlying surface.
This can include view transformations (animations), view manipulations (resizing) or just by calling InvalidateSurface() on some platforms.
How are you doing the transformations? And, have you ensured that you are not attaching multiple event handlers to the event (or the same one multiple times)?

Sorry, I am little bit late. Actually when I was searching for this issue, I first tried this example. but at that time bitmap scaling translating operations were not implemented. It was just simple skiasharp demo. After that I found this example. I tried that and implemented using renderer. So now, my current code includes both the code.  Actually I am not at my working place at the time of writing. So here is the description what I have done.
So I have TouchCanvasView Where you defined only OnTouch I have defined all the other methods OnScale,OnTranslate and others. Now in renderer in OnTouch I calculate distance, scale or rotate and return events defined in TouchCanvasView. and catch it in MainPage.xaml.cs. And I implemented rotation after SkiaSharpFormsDemos got updated. because there was matrix related issue.
I know its crazy but after trying for so many days, its working !!
And I am not calling event multiple times.
One more question when I Invalidate from MainPage.xaml.cs, OnPaint is not called. why ??
Thank you !!

Great to hear that you are making progress.
I am not too sure what you mean by Invalidate... Do you mean InvalidateSurface? This works for mean - how are you testing that it is not being called? It might be the case that either the width or height is zero, so in this case, the paint is skipped.