Roborazzi AI-Powered Image Verification prototype

Background
Advancements in AI: AI technologies are becoming increasingly sophisticated and affordable.
Manual Screenshot Testing: Traditional methods require manual verification, limiting scalability.
Problem
Scalability Limitations: Manual checks hinder large-scale screenshot testing.
High Maintenance Costs: Human intervention increases effort and costs.
Proposal
Integrate AI for Image Comparison: Use AI to automate screenshot verification in tests.
Enhance Roborazzi: Add AiOptions to enable AI-based image evaluation.
Current code
@RunWith(AndroidJUnit4::class)
@GraphicsMode(GraphicsMode.Mode.NATIVE)
@Config(
  sdk = [30],
  qualifiers = RobolectricDeviceQualifiers.NexusOne
)
class AiTest {
  @get:Rule
  val composeTestRule = createAndroidComposeRule<MainActivity>()

  @get:Rule
  val roborazziRule = RoborazziRule(
    options = RoborazziRule.Options(
      roborazziOptions = RoborazziOptions(
        compareOptions = RoborazziOptions.CompareOptions(
          aiOptions = AiOptions(
            aiModel = AiOptions.AiModel.Gemini(
              apiKey = System.getenv("gemini_api_key")!!
            ),
          )
        )
      )
    )
  )

  @Test
  fun captureWithAi() {
    ROBORAZZI_DEBUG = true
    onView(ViewMatchers.isRoot())
      .captureRoboImage(
        roborazziOptions = provideRoborazziContext().options.addedCompareAiAssertions(
          AiOptions.AiAssertion(
            assertPrompt = "it should have PREVIOUS button",
            requiredFulfillmentPercent = 90,
          ),
          AiOptions.AiAssertion(
            assertPrompt = "it should show First Fragment",
            requiredFulfillmentPercent = 90,
          )
        )
      )
  }
}

Changes for Existing Projects
We won't introduce any new dependencies for existing projects. I think I'll add the roborazzi-ai module to manage dependencies like Gemini or others, so you don't have to worry about disrupting existing dependencies.
Discussion Points
When We Execute This
The current AI implementation runs only when:

AiOptions are set

Because we can't run without prompts and API keys

Comparing or verifying tasks
The image is changed with ImageComparator

Because I think we can save a lot of money if we check only when images are changed.

And the test fails only when:

All above conditions are met
requiredFulfillmentPercent is set
It doesn't fulfill
(These tests fail even if it's a compare task.)

How We Set Prompts
Currently, it is a little difficult to set a prompt because we need to change RoborazziOptions.CompareOptions.AiOptions.prompt.
Things to Do

Get feedback from someone
Decide on the name of this feature
Decide if it is okay to have prompt in RoborazziOptions.CompareOptions. If we create a new function for this, like captureRoboImageWithAI, it will require bigger changes. Alternatively, provide a way to change these options easily for RoborazziOptions.

TODO:

Add a test using AiModel.Manual
Separate roborazzi-ai into roborazzi-ai and roborazzi-ai-gemini


Bellman equation with description of variables

I have review the document and made the necessary changes to the .tex file.
I upload the .tex and the .pdf to my repository, but I'm not sure how to send it to yours.
I can meet any day to review this.
Now I'm working on the SAS program, picking up were we left of last week.


Reconstruction-code crashes

Hi!
Thank you so much for this very interesting project!
I'm having issues when it comes to running the reconstruction code(the binary crashes) on the breast models you have provided and so I can't produce images.
In order for you to understand exactly what's going on, I've added a detailed description of the steps I've taken in the following gist:
a detailed description

Hi,
We are happy to hear of your interest in using our tools.  My first suggestion would be for you to generate the flatfield image which is used by the reconstruction code.
To generate flatfield image using MCGPU, you will have to move the phantom out of the detector boundary.  In the MC-GPU_v1.5b_sample_mammo_and_DBT_simulation.in file for MCGPU, make the following change and run - this will give you the flatfield corrected image.
999.0    0.0    0.0              # OFFSET OF THE VOXEL GEOMETRY (DEFAULT ORIGIN AT LOWER BACK CORNER) [cm]
Hope this helps!
Best,
Diksha Sharma

Thank you for your super-fast response!
The tools you guys have created may help save lives so thx for that.
I've tried your suggestion, and it made some progress, but the reconstruction looks very different from  TomoSynthesis images I've seen in hospitals eg that's how it looks like on my end: https://www.youtube.com/watch?v=A6zAqupFgro
Please see this gist which describes what I've done differently this time.
It's available here: https://gist.github.com/yoniker/e581ec4467a0e399d7b08775e792cafb

Here is my suggestion: Based on the provided youtube images, the problem is that you did not specify the reconstruction FOV properly. You need to make the reconstruction FOV to be consistent with the object FOV according to the DBT geometry you simulated.

Thanks for your answer,
Following up on your suggestion, I redid the entire process from scratch and created a gist for your convenience,you can follow the exact steps I took right here:
https://gist.github.com/yoniker/80994c67ae94567e8f83e22ad9585cf6
The only difference between the flatfield raw images and the raw images containing the phantom is the offset from the origin as you have suggested previously, and yet the reconstructed tomo image looks like this: https://youtu.be/qLFHc1eqzHc
How do you suggest for us to get a more realistic looking reconstructed tomo volume?
What would you change in the steps I've taken?

From the images you showed, the reconstruction field of view was out of the object region. That's probably why the reconstructed images looked out of focus. To solve this problem, you can play with the parameter "offset_xyz" and others that define the volume positions to be reconstructed to make the images right. Hope this helps.

Thanks  for your quick response,
I've followed the parameters' values you have provided within the input configuration file you have uploaded using your breast model, as well as your suggestions.
Each iteration (of creating a tomo image) does take a while (a few hours) so that playing with the parameters might take me a very very long time- remember that I'm not as familiar with your software as you.
I've imaged the scattered density breast model which you provided, with its corresponding input configuration file (MC-GPU_v1.5b_scattered_phantom_mammo+DBT.in) which contains the values of the parameters.
It will be awesome if you can tell me- What exact changes do I need to make to which parameters in that file in order to get a proper output from the imaging system?

Hi,
It looks like you are combining all the DBT projections with the "cat" command. But note that the output files contain two separate images: primary+scatter (the "real" image) and primary only. If you did not remove the primary only images from the input to the reconstruction, it will be using these repeated images instead of the actual projections at the required angles. It looks like this was not documented too well...
I uploaded the utility (extract_projections.c) that I use to extract the correct images and combine them into a single file before DBT recon (you will need to do the same for the flatfields). Here is the link and some explanations: VICTRE_MCGPU/example_simulations.
By the way, you can save yourself some hours in DBT reconstructions by editing the Makefile and change the debugging option CFLAGS := -g -O0 -w for CFLAGS     := -g -O3 -w

Thanks Andreu!
So I've used your utility as suggested to concatenate the raw images files and now the reconstruction program gives me as an output a set of black screens as the different tomo slices (https://youtu.be/oIF1RRtEMtk). I've created a gist so that you can let me know what I should correct right here:
https://gist.github.com/yoniker/e3d5581bac386c6190b3ebe597dbab1a
It will be great if you can provide a single end-to-end example,starting with an existing breast model with (or without) a lesion, and show how to produce a reasonable looking Tomo and/or a Mammogram image from the existing model.
Up to the point of imaging everything went smoothly-ish for me, but actually imaging a model and running the reconstruction stage is...let's say,not as user friendly as possible :)

Hi Yoni,
I followed the process you describe and I was able to get a valid reconstruction.
The output you got from the reconstruction code is a completely black image (all zeros) or noise?
Did you open the output file as 64-bit real (double) values, Little-Endian byte order, size 1321x1024x57?
(I know it is confusing that the projections are in single precision but the recon in doubles; it doesn't really make sense).
I could share my simulated projections for the example if you have a way to receive a gigabyte of data.
Have a nice weekend!
  Andreu

Hi Andreu!
First of all,Thank you so much for going over my previous efforts!
In the last couple of days I've tried to debug it myself (tried different machines as well) but got the same result.
I've opened the output file as a 64-bit real,Little endian of size 1421x1024x57, imagej (and numpy) is showing the pixel values as 0.0.
I've decided to upload all of the output files (like in the gist,every output file went to the results directory under MC-GPU/example_simulations/results) since you are in a better position than me to determine the reason for the difference between my output files and yours:
https://drive.google.com/drive/folders/1PiXBgtALL9hrTQqcMJ_lPgRAlPLU7GJd
So let me know if I'm somehow misusing any of the tools-I did put in my best efforts to use those,since we might be able to literally save lives using those tools! (if you are interested I can elaborate more in person).

Hi, I am also getting similar images as Yoni's. I am able to get same dimensions of the reconstructed volume as the phantom used in simulation. But the images look off focus.

Hi Harshit,
The image in your post looks ok for the first or last slice of a DBT recon. There are always massive cone-beam artifacts at the extremes. Could you send the image at the center of the volume?
From the image it looks like your volume has size 894x505x715. This does not seem to match any of our phantoms or reconstruction sizes. We reconstructed our phantoms using 0.085x0.085x1.0 mm voxels.
For example, for a scattered phantom the original size was 1740x2415x1140, 0.050 mm voxels. The corresponding reconstructed volume had size: 174050/85 x 241550/85 x 1140*50/1000 =  1023.53 x 1420.59 x 57.00 ~ 1024x 1421x57
It is possible also that the recon software inverts the meaning of X and Y.
Let me know if you can see a DBT volume or not. At least your image is not all zeros as yoniker`s!

By the way, there is a small bug in the recon code. It has no effect in my computer but it might be problematic in other architectures.
In file FBP_DBTrecon.c, line 939, it says:
"fread(projdbt_un, sizeof(double), ns_oldnt_oldnadowndown, fid);"
The input projections are actually float, not double. Try correcting the code to:
"fread(projdbt_un, sizeof(float), ns_oldnt_oldnadowndown, fid);"
Let me know if this changes anything in your reconstructions.

Dear Yoniker,
Please edit line 905 in your "FBP_DBTrecon.c" file to point to the actual DBT projections. You are using the same file name for the flat fields and the projections!
By the way, since it is never a good idea to normalize an image with a noisy image, I recommend running the fast flat field simulations at least 10 times, and inputting the average flat fields to the recon algorithm (this is actually what people do with real imaging systems).
I hope the reconstruction works now!

Hey Andreu!
Thank you for your answer.
Sorry for the mistake - I got it right the first time and somehow i got things mixed up over time...
The reconstruction certainly looks way way better- (here it is : https://youtu.be/6oGU5iQcqo4) and that's from running the ff simulation only once.
That being said, the reconstruction output still doesn't look like something I would expect when it comes to a Tomo(an example for a real Tomo can be viewed here: https://www.youtube.com/watch?v=YLaIlLxTlVQ&feature=youtu.be&t=77).
How can I make the output look more realistic?

Hi Andreu, thanks for reply. The phantom size was such because I changed u and v values. But I again tried with default values and dense phantom for 0.050 mm voxels. The phantom size was 1791x1010x1434. The corresponding volume size was (1791x0.0050/0.0085) x (1010x0.0050/0.00850) x (1434x0.0050/0.100) = 1057x595x72.
But the images are similar to Yoni's and my question was also similiar. Can we get more realistic images?

The reconstruction video looks good to me.
Commercial DBT systems might post-process the reconstructed images to try to enhance some aspects of the images and make them look more appealing based on feedback from their customers (the same is true for the mammography simulations: our images are "for processing", not "for presentation" views). They might also attempt to reduce some of the artifacts. Nothing of this is done by our basic filtered back-projection algorithm (it is not clear that post-processing actually improves clinical performance anyway).
But you should definitely optimize the visualization gray scale. You could change the gray scale window/level value to set the contrast between adipose and glandular tissue to the level that you prefer. And you can also set to black anything outside the breast. Many artifacts are visible outside the breast and they are totally irrelevant for any diagnostic task.


Layer's Message network.

Now Layers interaction only use default TCP stream.
We want to refactor it to use chamomile's p2p network.
Layers connection use chamomile's peer_id as it's network address.
Layers' id is group_id, and this id should signed peer_id, so application can verify it. (we will define how to generate layer's id, and how to sign, how to verify. maybe will use tdn_did.)

Test in Esse.

Test in Esse.

over it.


Add '--branch' to command 'repository add'

Changes

:broom: Add support for named branches in func repository add command

Fixes #1197

Thanks for the review. Just back from national holidays.  working on a new version of PR now.

Hi @daisy-ycguo just checking in to see if you've had a chance to look into these test failures.

Hi @daisy-ycguo just checking in here - any progress?

@daisy-ycguo gentle ping


about install

Hi,when I run the pipline;I met the problem:
/CentIERv2.0/bin/gt/bin/gt: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /CentIERv2.0/bin/gt/bin/gt)
This may involve in the Linux update;
so ;could you upload on the Conda?
Thanks.

Thank you for your question. This error may be caused by gt version. I will update the CentIER version within a week, please be patient. CentIER utilizes the gt program, so in the new version, users will need to install gt on their own and place it in the environment variable.
At 2024-06-19 10:40:57, "frwjo" @.***> wrote:
Hi,when I run the pipline;I met the problem:
CentIERv2.0/bin/gt/bin/gt: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /pub4/wujie/softwore/CentIER/CentIERv2.0/bin/gt/bin/gt)
This may involve in the Linux update;
so ;could you upload on the Conda?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.Message ID: @.***>


Don't wrap direct response messages with Forward wrapper(s)

Fixes #198

Codecov Report

Merging #199 into master will increase coverage by 0.63%.
The diff coverage is 52%.

@@            Coverage Diff             @@
##           master     #199      +/-   ##
==========================================
+ Coverage   71.58%   72.22%   +0.63%
==========================================
  Files         218      218
  Lines        9982     9990       +8
==========================================
+ Hits         7146     7215      +69
+ Misses       2836     2775      -61


[mesheryctl] Docs: Remove apply, add import, promote from experimental

Signed-off-by: Lee Calcote<EMAIL_ADDRESS>Notes for Reviewers
This PR fixes promotes the mesheryctl filter command from experimental and replaces the apply command with the import command.
The markdown file for the apply subcommand might need to be manually deleted.
Signed commits

[x] Yes, I signed my commits.

🚀 Preview for commit 3546c441907646655f84bec469df8a5d7a2e16a0 at: https://64b086cc7d95646034f1ab3b--meshery-docs-preview.netlify.app


[Question] Can I strict the order of Closure annotation?

Hello
Thank you so much development.
I would like to strict the order of closure annotation? like this.
Do you have any options or roadmap as a feature request?
🙅♂ NG
let badClosure: @escaping @Sendable () -> Void

🙆♂ Good
let goodClosure: @Sendable @escaping () -> Void

Synced to Apple’s issue tracker as rdar://128436416


Stale documentation on filter_immune_slice_fields

Screenshot

Description
Superset documentation at https://superset.apache.org/docs/frequently-asked-questions mentions filter_immune_slice_fields as a top-level key in dashboard metadata JSON. However, the key was dropped in favour of filter_scopes. Using filter_immune_slice_fields to modify dashboard metadata results in 400 response code with filter_immune_slice_fields: unknown_field validation error.

Closing this issue here, and moving it to BugHerd, where we keep our docs requests. We could use help opening PRs to fix these things, if you have interest in this area.
CC @sfirke — https://www.bugherd.com/projects/334204/tasks/127


feat!: add Makefile rule for formatting

In order to have a consistent formatted source code, this PR adds a Makefile rule format calling ocamlformat.0.8 and provides a minimal .ocamlformat file (which is not meant to be definitive at all).

Note: learn-ocaml using dune.1.7, the stanza formatting can't be used. That's why I added the rule in the Makefile.

I think ocamlformat is fine for new projects, but in this case I wonder if it is worth it to break all of the git history (by which I mean git blame-related functions)... Apparently it's not so common but I check the git history of a given source line to understand why it was put there pretty often.

Agree with @AltGr; moreover, merging this ocamlformat change at some point would mean that all currently opened PRs will raise git conflicts everywhere… so if one decide to fully switch to ocamlformat, one should at least wait for having no open PR 😅
otherwise, ideally (but I don't know if this is possible), ocamlformat could only rewrite code lines that have been recently edited… (using git diff info or so)

I think ocamlformat is fine for new projects, but in this case I wonder if it is worth it to break all of the git history (by which I mean git blame-related functions)... Apparently it's not so common but I check the git history of a given source line to understand why it was put there pretty often.

Since git 2.23, using the --ignore-rev flag can specify a commit to be ignored by git blame. I added the commit hash in the git-blame-ignore-revs file.
So, you just need to run once:
git config blame.ignoreRevsFile .git-blame-ignore-revs

and git blame will ignore changes caused by formatting.

[...] merging this ocamlformat change at some point would mean that all currently opened PRs will raise git conflicts everywhere...

I think for this point, updating the PR branch from the master one and then run make format should be enough, no?

[...] merging this ocamlformat change at some point would mean that all currently opened PRs will raise git conflicts everywhere...

I think for this point, updating the PR branch

How should we do this?
IINM, either git fetch upstream && git merge upstream/master or git fetch upstream && git rebase upstream/master would create a ton of conflicts (one for each line that was both modified in the branch by the user / in master by ocamlformat)

You are certainly right... What about installing ocamlformat and formatting the source code manually in branches before merging master into them?


Crash when metaballs hits a bot

Hitting a bot, not its candle, causes a crash in debug and release.

Resolved


button-full does not work in ion-footer-bar

Buttons with the class button-full do not render properly when included in an ion-footer-bar. For instance:
<ion-footer-bar>

    <button class="button button-full">Button 1</button>

Title

    <button class="button">Button 2</button>

</ion-footer-bar>

@i'm facing the same issue when the app is opened in tablet. The button in footer doesn't come as required. The button is a very small

This is another alternative.
Other than this, you will have to write your own css for this.
http://codepen.io/mhartington/pen/bdGNGr

@mhartington the button works weel in all the devices with my css. just like it doesn't support in the tablet? Any idea about this ?

could you open a new issue for this.

yeah i'll

work for me with this css class tricks :

            A full link in bottom

div.footer-button a.button.button-full {
    width:100%;
    margin-top: 0
}

@mhartington that tab hack is cool. I just made a minor change of adding tab-item-active class .. and its exactly what I needed. Thanks!
<ion-footer-bar class="bar-positive tabs tabs-stable">

    Add Contact

</ion-footer-bar>

I added my button at the bottom like this:

Filter

You can even get rid of button tag and make your ion-footer-bar pretend to be a "button" by passing ng-click to it:

Filter

But it should be a better way to achieve this.

Mb, not good idea, u can use $rootScope
<ion-footer-bar>   <button class="bar bar-footer bar-positive" ng-click="closeModal()"></button> </ion-footer-bar>
 $rootScope.closeModal = function ( ) {  }


multimacd causes significant CPU load

Describe the bug
after a running period of day's ( one or two weeks ) the multimacd causes significant CPU load
Mem: 729056K used, 249664K free, 5564K shrd, 28344K buff, 187560K cached
CPU:  14% usr  15% sys   0% nic  69% idle   0% io   0% irq   1% sirq
Load average: 2.19 2.13 2.44 5/275 11513
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
  895     1 root     S<   12216   1%  24% /bin/multimacd -f /var/etc/multimacd.conf -l 4

This causes a huge count of service messages. And response times for CCU triggered actions become higher.
Not 100% sure, but seems only BidCoS-RF devices affected.
Steps to reproduce the behavior
Not clear, what the root cause is. No special activity, log enty in Log-File.
Expected behavior
Clear Log-Entry or better, the issue isn't happen.
Screenshots

System information:

RaspberryMatic Version: <IP_ADDRESS>10525
Used Hardware: Raspberry Pi 3
Used HomeMatic RF-Module: RPI-RF-MOD

Additional context
...

Sorry, but this can't be a general problem/issue because for me and several others we are not having any such high CPU load issue with multimacd. So please proceed according to the following statement and please come back if you have others replicating your issue or if you have more technical information pointing the finger at a specific/definite bug.
Sorry, but this is no support or discussion fora here. This is a bug and issue tracker. So please open a new discussion thread in the discussion fora of this GitHub project or use the external german speaking RaspberryMatic Discussion fora (https://homematic-forum.de/) for discussing your matter. This was also already mentioned in the ticket template!

Tut mir leid, aber das hier ist kein Supportforum, sondern ein Bug/Issuetracker, d.h. für Dinge die DEFINITIV ein Bug darstellen oder Funktionserweiterungen sind. Daher bitte entweder hier im Diskussusionsteil des GitHub Projektes einen neuen Beitrag eröffnen oder das externe RaspberryMatic Forum (https://homematic-forum.de/) dafür nutzen.  Dies stand übrigens auch so im Ticket Template!

Keep it in mind, it was only for your information.

Of course, but this is a bug tracker and no discussion fora. Thus, such information (which didn't immediately qualify as a clear bug) is better to be discussed first and then perhaps moved over to the bug/issue tracker if others have reproduced it or if there are clear technical indications for work to do.


fix: Fix pathType warnings on ingress resources

Closes #21
PR Checklist
Please check if your PR fulfills the following requirements:

[X] I am not introducing a breaking change (if you are, flag in conventional commit message with BREAKING CHANGE: describing the break)
[X] I have fully tested (add details below) this the new feature or bug fix (if not, why?)
[ ] I have opened a PR for the related docs change (if not, why?)

Testing Instructions
Install helm chart as usual

don't remember what version of Kubernetes we required, only version 1.18 or above has this pathType feature

https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/basic-configuration/
in section New Features Available in Kubernetes 1.18 and Above

don't remember what version of Kubernetes we required, only version 1.18 or above has this pathType feature

https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/basic-configuration/ in section New Features Available in Kubernetes 1.18 and Above

https://kubernetes.io/releases/
The Kubernetes project maintains release branches for the most recent three minor releases (1.28, 1.27, 1.26). Kubernetes 1.19 and newer receive approximately 1 year of patch support. Kubernetes 1.18 and older received approximately 9 months of patch support.
1.17 reached end of support on 25 Dec 2020. (https://endoflife.date/kubernetes)

don't remember what version of Kubernetes we required, only version 1.18 or above has this pathType feature

https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/basic-configuration/ in section New Features Available in Kubernetes 1.18 and Above

https://kubernetes.io/releases/
The Kubernetes project maintains release branches for the most recent three minor releases (1.28, 1.27, 1.26). Kubernetes 1.19 and newer receive approximately 1 year of patch support. Kubernetes 1.18 and older received approximately 9 months of patch support.
1.17 reached end of support on 25 Dec 2020. (https://endoflife.date/kubernetes)

ok, it is interesting in that ngnix docs, it says The ImplementationSpecific pathType is treated as equivilent to Prefix pathType, with the exception that when this pathType is configured, the path field in spec.rules[].http.paths[] is not mandatory. path defaults to / if not set but the pathType is set to ImplementationSpecific.

ok, it is interesting in that ngnix docs, it says The ImplementationSpecific pathType is treated as equivilent to Prefix pathType, with the exception that when this pathType is configured, the path field in spec.rules[].http.paths[] is not mandatory. path defaults to / if not set but the pathType is set to ImplementationSpecific.

I can't make sense of that, but the regexp is needed for the rewrite rule and when I use ImplementationSpecific the warning goes away :-)


[Grouping] treeRowHeaderAlwaysVisible not working as expected

My expectation with this option is that when set to false,  the "plus" column will be removed/added based on whether or not grouping is active. This does not appear to be working.

@PaulL1  I have been working a bit on this. The core problem is solved by resetting grid.treeBase.numberLevels to 0 at the beginning of the createTree method. But then we run into the problem that we are considering a single grouping to have 0 numberLevels, which is also the default without any grouping.
I am going to change this so numLevels becomes 1 if there is a single grouped column. Alternatively we could make it undefined if there are no grouped columns.

Sounds reasonable.  I don't think I did it deliberately, I recall noticing it had ended up zero based then just going with it.


etcd v3 fails to self heal on a 3 host Kubernetes setup

Rancher versions:
rancher/server:v1.6.8-rc1
Infrastructure Stack versions:
kubernetes (if applicable):v1.7.2
**Docker version: (docker version,docker info preferred)**1.12.6
**Operating system and kernel: (cat /etc/os-release, uname -r preferred)**Ubuntu
**Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)**DO
**Setup details: (single node rancher vs. HA rancher, internal DB vs. external DB)**Single node
**Environment Template: (Cattle/Kubernetes/Swarm/Mesos)**Kubernetes
Steps to Reproduce:

Start a 3 host Kubernetes setup. Create pods/rcs
Power down 2 hosts(h2 and h3). Power back 1 host(h3).

Ideally this should be a self healing scenario. But, the etcd on host3 failed to join the quorum and was stuck in initializing state.
Kubernetes stack was not active. Kubernetes dashboard was not active. etcd cluster was unhealthy.
bash-4.3# etcdctl cluster-health
cluster may be unhealthy: failed to list members
Error:  client: etcd cluster is unavailable or misconfigured
error #0: dial tcp <IP_ADDRESS>:4001: getsockopt: connection refused
error #1: client: endpoint http://<IP_ADDRESS>:2379 exceeded header timeout

etcd logs
8/11/2017 3:03:34 PM+++ grep <IP_ADDRESS>
8/11/2017 3:03:34 PM+++ etcdctl_one member list
8/11/2017 3:03:34 PM+++ target_ip=0
8/11/2017 3:03:34 PM++++ giddyup service containers
8/11/2017 3:03:34 PM+++ for container in '$(giddyup service containers)'
8/11/2017 3:03:34 PM++++ wget -q -O - http://<IP_ADDRESS>/2015-12-19/self/service/containers/kubernetes-etcd-1/primary_ip
8/11/2017 3:03:34 PM+++ primary_ip=<IP_ADDRESS>
8/11/2017 3:03:34 PM+++ giddyup probe tcp://<IP_ADDRESS>:2379
8/11/2017 3:03:34 PM+++ '[' 0 == 0 ']'
8/11/2017 3:03:34 PM+++ target_ip=<IP_ADDRESS>
8/11/2017 3:03:34 PM+++ break
8/11/2017 3:03:34 PM+++ '[' <IP_ADDRESS> == 0 ']'
8/11/2017 3:03:34 PM+++ etcdctl --endpoints http://<IP_ADDRESS>:2379 member list
8/11/2017 3:05:20 PMError:  read tcp <IP_ADDRESS>:50782-><IP_ADDRESS>:2379: read: connection reset by peer
8/11/2017 3:05:20 PM++ '[' '' ']'
8/11/2017 3:05:20 PM++ giddyup leader check
8/11/2017 3:05:20 PM++ echo Adding Node
8/11/2017 3:05:20 PM++ runtime_node
8/11/2017 3:05:20 PMAdding Node
8/11/2017 3:05:20 PM++ rm -rf '/pdata/data.current/*'
8/11/2017 3:05:20 PM+++ date -R
8/11/2017 3:05:20 PM++ timestamp='Fri, 11 Aug 2017 22:05:20 +0000'
8/11/2017 3:05:20 PM++ echo 'Fri, 11 Aug 2017 22:05:20 +0000 -> Scaling up. Deleted stale data'
8/11/2017 3:05:20 PM+++ giddyup service containers --exclude-self
8/11/2017 3:05:20 PM++ for container in '$(giddyup service containers --exclude-self)'
8/11/2017 3:05:20 PM++ echo Waiting for lower index nodes to all be active
8/11/2017 3:05:20 PMWaiting for lower index nodes to all be active
8/11/2017 3:05:20 PM+++ wget -q -O - http://<IP_ADDRESS>/2015-12-19/self/service/containers/kubernetes-etcd-1/create_index
8/11/2017 3:05:20 PM++ ctx_index=1
8/11/2017 3:05:20 PM+++ wget -q -O - http://<IP_ADDRESS>/2015-12-19/self/service/containers/kubernetes-etcd-1/primary_ip
8/11/2017 3:05:20 PM++ primary_ip=<IP_ADDRESS>
8/11/2017 3:05:20 PM++ '[' 1 -lt 1116 ']'
8/11/2017 3:05:20 PM++ giddyup probe http://<IP_ADDRESS>:2379/health --loop --min 1s --max 15s --backoff 1.2
8/11/2017 3:05:25 PMGet http://<IP_ADDRESS>:2379/health: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
8/11/2017 3:05:31 PMGet http://<IP_ADDRESS>:2379/health: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
8/11/2017 3:05:37 PMGet http://<IP_ADDRESS>:2379/health: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
8/11/2017 3:05:43 PMGet http://<IP_ADDRESS>:2379/health: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
8/11/2017 3:05:50 PMGet http://<IP_ADDRESS>:2379/health: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
8/11/2017 3:05:57 PMGet http://<IP_ADDRESS>:2379/health: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
8/11/2017 3:06:05 PMGet http://<IP_ADDRESS>:2379/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

etcd fails to self heal in the following case too:
1.Start a 3 host Kubernetes setup. Create pods/rcs
2.Power down 2 hosts(h2 and h3).
3. Power back both hosts(h2 and h3).
Ideally this should be a self healing scenario, but etcd cluster fails to self heal and is unhealthy.
bash-4.3# etcdctl cluster-health
member 609d837ceaf22e4a is unhealthy: got unhealthy result from http://10.42.218
.183:2379
failed to check the health of member 663e8e43312368f0 on http://<IP_ADDRESS>:237
9: Get http://<IP_ADDRESS>:2379/health: dial tcp <IP_ADDRESS>:2379: getsockopt:
connection refused
member 663e8e43312368f0 is unreachable: [http://<IP_ADDRESS>:2379] are all unrea
chable
failed to check the health of member c01cb86b178180e3 on http://<IP_ADDRESS>:23
79: Get http://<IP_ADDRESS>:2379/health: dial tcp <IP_ADDRESS>:2379: getsockop
t: connection refused
member c01cb86b178180e3 is unreachable: [http://<IP_ADDRESS>:2379] are all unre
achable
cluster is unhealthy

The etcds on the hosts that were powered ON are stuck in initializing state.
etcd-1 logs:
8/11/2017 5:03:30 PM+ CREATE_INDEX=523
8/11/2017 5:03:30 PM++ wget -q -O - http://<IP_ADDRESS>/2015-12-19/self/container/service_index
8/11/2017 5:03:30 PM+ SERVICE_INDEX=1
8/11/2017 5:03:30 PM++ wget -q -O - http://<IP_ADDRESS>/2015-12-19/self/host/uuid
8/11/2017 5:03:30 PM+ HOST_UUID=ddaeed3c-6226-4d8e-93b1-821b940ba0a1
8/11/2017 5:03:30 PM+ LEGACY_DATA_DIR=/data
8/11/2017 5:03:30 PM+ DATA_DIR=/pdata
8/11/2017 5:03:30 PM+ DR_FLAG=/pdata/DR
8/11/2017 5:03:30 PM+ export ETCD_DATA_DIR=/pdata/data.current
8/11/2017 5:03:30 PM+ ETCD_DATA_DIR=/pdata/data.current
8/11/2017 5:03:30 PM+ export ETCDCTL_ENDPOINT=http://etcd.kubernetes:2379
8/11/2017 5:03:30 PM+ ETCDCTL_ENDPOINT=http://etcd.kubernetes:2379
8/11/2017 5:03:30 PM+ export ETCDCTL_API=2
8/11/2017 5:03:30 PM+ ETCDCTL_API=2
8/11/2017 5:03:30 PM++ echo <IP_ADDRESS>
8/11/2017 5:03:30 PM++ tr . -
8/11/2017 5:03:30 PM+ NAME=10-42-61-255
8/11/2017 5:03:30 PM+ '[' 1 -eq 0 ']'
8/11/2017 5:03:30 PM+ eval node
8/11/2017 5:03:30 PM++ node
8/11/2017 5:03:30 PM++ mkdir -p /pdata/data.current
8/11/2017 5:03:30 PM++ '[' -d /data/member ']'
8/11/2017 5:03:30 PM++ '[' -d /data/data.current ']'
8/11/2017 5:03:30 PM++ '[' -f /pdata/DR ']'
8/11/2017 5:03:30 PM++ '[' -d /pdata/data.current/member ']'
8/11/2017 5:03:30 PM+++ grep <IP_ADDRESS>
8/11/2017 5:03:30 PM+++ etcdctl_one member list
8/11/2017 5:03:30 PM+++ target_ip=0
8/11/2017 5:03:30 PM++++ giddyup service containers
8/11/2017 5:03:30 PM+++ for container in '$(giddyup service containers)'
8/11/2017 5:03:30 PM++++ wget -q -O - http://<IP_ADDRESS>/2015-12-19/self/service/containers/kubernetes-etcd-3/primary_ip
8/11/2017 5:03:31 PM+++ primary_ip=<IP_ADDRESS>
8/11/2017 5:03:31 PM+++ giddyup probe tcp://<IP_ADDRESS>:2379
8/11/2017 5:03:31 PM+++ '[' 0 == 0 ']'
8/11/2017 5:03:31 PM+++ target_ip=<IP_ADDRESS>
8/11/2017 5:03:31 PM+++ break
8/11/2017 5:03:31 PM+++ '[' <IP_ADDRESS> == 0 ']'
8/11/2017 5:03:31 PM+++ etcdctl --endpoints http://<IP_ADDRESS>:2379 member list
8/11/2017 5:03:31 PMFailed to get leader:  client: etcd cluster is unavailable or misconfigured
8/11/2017 5:03:31 PM++ '[' '' ']'
8/11/2017 5:03:31 PM++ giddyup leader check
8/11/2017 5:03:31 PMAdding Node
8/11/2017 5:03:31 PM++ echo Adding Node
8/11/2017 5:03:31 PM++ runtime_node
8/11/2017 5:03:31 PM++ rm -rf '/pdata/data.current/*'
8/11/2017 5:03:31 PM+++ date -R
8/11/2017 5:03:31 PM++ timestamp='Sat, 12 Aug 2017 00:03:31 +0000'
8/11/2017 5:03:31 PM++ echo 'Sat, 12 Aug 2017 00:03:31 +0000 -> Scaling up. Deleted stale data'
8/11/2017 5:03:31 PM+++ giddyup service containers --exclude-self
8/11/2017 5:03:31 PM++ for container in '$(giddyup service containers --exclude-self)'
8/11/2017 5:03:31 PM++ echo Waiting for lower index nodes to all be active
8/11/2017 5:03:31 PMWaiting for lower index nodes to all be active
8/11/2017 5:03:31 PM+++ wget -q -O - http://<IP_ADDRESS>/2015-12-19/self/service/containers/kubernetes-etcd-3/create_index
8/11/2017 5:03:31 PM++ ctx_index=7
8/11/2017 5:03:31 PM+++ wget -q -O - http://<IP_ADDRESS>/2015-12-19/self/service/containers/kubernetes-etcd-3/primary_ip
8/11/2017 5:03:31 PM++ primary_ip=<IP_ADDRESS>
8/11/2017 5:03:31 PM++ '[' 7 -lt 523 ']'
8/11/2017 5:03:31 PM++ giddyup probe http://<IP_ADDRESS>:2379/health --loop --min 1s --max 15s --backoff 1.2
8/11/2017 5:03:31 PMHTTP 503
8/11/2017 5:03:31 PM
8/11/2017 5:03:32 PMHTTP 503
8/11/2017 5:03:32 PM
8/11/2017 5:03:33 PMHTTP 503
8/11/2017 5:03:33 PM
8/11/2017 5:03:34 PMHTTP 503
8/11/2017 5:03:34 PM
8/11/2017 5:03:36 PMHTTP 503
8/11/2017 5:03:36 PM
8/11/2017 5:03:38 PMHTTP 503
8/11/2017 5:03:38 PM
8/11/2017 5:03:41 PMHTTP 503
8/11/2017 5:03:41 PM
8/11/2017 5:03:44 PMHTTP 503
8/11/2017 5:03:44 PM
8/11/2017 5:03:47 PMHTTP 503
8/11/2017 5:03:47 PM
8/11/2017 5:03:52 PMHTTP 503
8/11/2017 5:03:52 PM
8/11/2017 5:03:57 PMHTTP 503

etcd-2 logs
 8/11/2017 5:06:30 PM+ META_URL=http://<IP_ADDRESS>/2015-12-19
8/11/2017 5:06:30 PM++ wget -q -O - http://<IP_ADDRESS>/2015-12-19/self/stack/name
8/11/2017 5:06:30 PM+ STACK_NAME=kubernetes
8/11/2017 5:06:30 PM+ '[' kubernetes == '' ']'
8/11/2017 5:06:30 PM++ giddyup service scale etcd
8/11/2017 5:06:30 PM+ SCALE=3
8/11/2017 5:06:30 PM++ echo
8/11/2017 5:06:30 PM++ grep -E '^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$'
8/11/2017 5:06:30 PM+ '[' '!' '' ']'
8/11/2017 5:06:30 PM+ sleep 1
8/11/2017 5:06:31 PM++ wget -q -O - http://<IP_ADDRESS>/2015-12-19/self/container/primary_ip
8/11/2017 5:06:31 PM+ IP=<IP_ADDRESS>
8/11/2017 5:06:31 PM++ echo <IP_ADDRESS>
8/11/2017 5:06:31 PM++ grep -E '^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$'
8/11/2017 5:06:31 PM+ '[' '!' <IP_ADDRESS> ']'
8/11/2017 5:06:31 PM++ wget -q -O - http://<IP_ADDRESS>/2015-12-19/self/container/create_index
8/11/2017 5:06:31 PM+ CREATE_INDEX=526
8/11/2017 5:06:31 PM++ wget -q -O - http://<IP_ADDRESS>/2015-12-19/self/container/service_index
8/11/2017 5:06:31 PM+ SERVICE_INDEX=2
8/11/2017 5:06:31 PM++ wget -q -O - http://<IP_ADDRESS>/2015-12-19/self/host/uuid
8/11/2017 5:06:31 PM+ HOST_UUID=a4372bc9-2200-4ab6-84b0-95b8727ce419
8/11/2017 5:06:31 PM+ LEGACY_DATA_DIR=/data
8/11/2017 5:06:31 PM+ DATA_DIR=/pdata
8/11/2017 5:06:31 PM+ DR_FLAG=/pdata/DR
8/11/2017 5:06:31 PM+ export ETCD_DATA_DIR=/pdata/data.current
8/11/2017 5:06:31 PM+ ETCD_DATA_DIR=/pdata/data.current
8/11/2017 5:06:31 PM+ export ETCDCTL_ENDPOINT=http://etcd.kubernetes:2379
8/11/2017 5:06:31 PM+ ETCDCTL_ENDPOINT=http://etcd.kubernetes:2379
8/11/2017 5:06:31 PM+ export ETCDCTL_API=2
8/11/2017 5:06:31 PM+ ETCDCTL_API=2
8/11/2017 5:06:31 PM++ echo <IP_ADDRESS>
8/11/2017 5:06:31 PM++ tr . -
8/11/2017 5:06:31 PM+ NAME=10-42-253-152
8/11/2017 5:06:31 PM+ '[' 1 -eq 0 ']'
8/11/2017 5:06:31 PM+ eval node
8/11/2017 5:06:31 PM++ node
8/11/2017 5:06:31 PM++ mkdir -p /pdata/data.current
8/11/2017 5:06:31 PM++ '[' -d /data/member ']'
8/11/2017 5:06:31 PM++ '[' -d /data/data.current ']'
8/11/2017 5:06:31 PM++ '[' -f /pdata/DR ']'
8/11/2017 5:06:31 PM++ '[' -d /pdata/data.current/member ']'
8/11/2017 5:06:31 PM+++ etcdctl_one member list
8/11/2017 5:06:31 PM+++ target_ip=0
8/11/2017 5:06:31 PM+++ grep <IP_ADDRESS>
8/11/2017 5:06:31 PM++++ giddyup service containers
8/11/2017 5:06:31 PM+++ for container in '$(giddyup service containers)'
8/11/2017 5:06:31 PM++++ wget -q -O - http://<IP_ADDRESS>/2015-12-19/self/service/containers/kubernetes-etcd-3/primary_ip
8/11/2017 5:06:31 PM+++ primary_ip=<IP_ADDRESS>
8/11/2017 5:06:31 PM+++ giddyup probe tcp://<IP_ADDRESS>:2379
8/11/2017 5:06:31 PM+++ '[' 0 == 0 ']'
8/11/2017 5:06:31 PM+++ target_ip=<IP_ADDRESS>
8/11/2017 5:06:31 PM+++ break
8/11/2017 5:06:31 PM+++ '[' <IP_ADDRESS> == 0 ']'
8/11/2017 5:06:31 PM+++ etcdctl --endpoints http://<IP_ADDRESS>:2379 member list
8/11/2017 5:06:31 PMFailed to get leader:  client: etcd cluster is unavailable or misconfigured
8/11/2017 5:06:31 PM++ '[' '' ']'
8/11/2017 5:06:31 PM++ giddyup leader check
8/11/2017 5:06:32 PM++ echo Adding Node
8/11/2017 5:06:32 PMAdding Node
8/11/2017 5:06:32 PM++ runtime_node
8/11/2017 5:06:32 PM++ rm -rf '/pdata/data.current/*'
8/11/2017 5:06:32 PM+++ date -R
8/11/2017 5:06:32 PM++ timestamp='Sat, 12 Aug 2017 00:06:32 +0000'
8/11/2017 5:06:32 PM++ echo 'Sat, 12 Aug 2017 00:06:32 +0000 -> Scaling up. Deleted stale data'
8/11/2017 5:06:32 PM+++ giddyup service containers --exclude-self
8/11/2017 5:06:32 PM++ for container in '$(giddyup service containers --exclude-self)'
8/11/2017 5:06:32 PM++ echo Waiting for lower index nodes to all be active
8/11/2017 5:06:32 PMWaiting for lower index nodes to all be active
8/11/2017 5:06:32 PM+++ wget -q -O - http://<IP_ADDRESS>/2015-12-19/self/service/containers/kubernetes-etcd-3/create_index
8/11/2017 5:06:32 PM++ ctx_index=7
8/11/2017 5:06:32 PM+++ wget -q -O - http://<IP_ADDRESS>/2015-12-19/self/service/containers/kubernetes-etcd-3/primary_ip
8/11/2017 5:06:32 PM++ primary_ip=<IP_ADDRESS>
8/11/2017 5:06:32 PM++ '[' 7 -lt 526 ']'
8/11/2017 5:06:32 PM++ giddyup probe http://<IP_ADDRESS>:2379/health --loop --min 1s --max 15s --backoff 1.2
8/11/2017 5:06:32 PMHTTP 503
8/11/2017 5:06:32 PM
8/11/2017 5:06:33 PMHTTP 503
8/11/2017 5:06:33 PM
8/11/2017 5:06:34 PMHTTP 503
8/11/2017 5:06:34 PM
8/11/2017 5:06:35 PMHTTP 503
8/11/2017 5:06:35 PM
8/11/2017 5:06:37 PMHTTP 503
8/11/2017 5:06:37 PM
8/11/2017 5:06:39 PMHTTP 503

Tested on master with Kubernetes 1.7.4-rancher2-etcdv3 on a 3 node setup.
1. Start a 3 host Kubernetes setup. Create pods/rcs
2. Power down 2 hosts(h1 and h3). Power back 1 host(h1).

Ideally this should be a self healing scenario. But, the etcd on host1 failed to join the quorum and was stuck in initializing state.
Kubernetes stack was not active. Kubernetes dashboard was not active. etcd cluster was unhealthy.
bash-4.3# etcdctl cluster-health
failed to check the health of member 481005694ded93cb on http://<IP_ADDRESS>:23
79: Get http://<IP_ADDRESS>:2379/health: dial tcp <IP_ADDRESS>:2379: getsockop
t: no route to host
member 481005694ded93cb is unreachable: [http://<IP_ADDRESS>:2379] are all unre
achable
failed to check the health of member 6486e7e62f09be13 on http://<IP_ADDRESS>:237
9: Get http://<IP_ADDRESS>:2379/health: dial tcp <IP_ADDRESS>:2379: getsockopt:
no route to host
member 6486e7e62f09be13 is unreachable: [http://<IP_ADDRESS>:2379] are all unrea
chable
member e36af8e035386332 is unhealthy: got unhealthy result from http://10.42.110
.221:2379
cluster is unhealthy

There is another problem even with retain_ip usage, the etcd container may land on different host so it will pick up the wrong IP and consider itself a new node, and the cluster will fail.

Tested it with the updated catalog provided on master, the issue above still exists.

The only solution with this is to restart the container on the original host first, if possible.

I think in this particular scenario, we just have to apply our DR solution.  This is not very common.


Panic occurred when handle Segmentation fault signal

What version of Go are you using (go version)?

$ go version
go version go1.11.13 linux/amd64

Does this issue reproduce with the latest release?
Don't konw, we did not tried.
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/data/jenkins/workspace/gohere"
GOPROXY=""
GORACE=""
GOROOT="/root/go"
GOTMPDIR=""
GOTOOLDIR="/root/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build208195420=/tmp/go-build -gno-record-gcc-switches"

What did you do?

This is our online service, and we use cgo to invoke our C++ code, recently we find our service crashed something. So,

We setted GOTRACEBACK=crash
# ulimit -c unlimited
Started the service again to expect another crash

What did you expect to see?
We want a core file and use gdb to see what happend.
What did you see instead?
We seen this with gdb and the core file:
Core was generated by `xxxxx'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000000000603e3e in runtime.(*sigctxt).preparePanic (c=0xc001389258, gp=<optimized out>, sig=<optimized out>) at /root/go/src/runtime/signal_linux_amd64.go:20
20              return (*sigcontext)(unsafe.Pointer(&(*ucontext)(c.ctxt).uc_mcontext))
Loading Go Runtime support.

I want to know is this normal?

Yes, this is the result of using unsafe in your code. Unless you are sure there are no bugs, your code can crash.
If you think there's a bug in Go rather than in your code, please post a short reproducer containing the code that triggers this. But given the fact that there is cgo and unsafe involved, it is unlikely that there is any bug here.
If you use Go 1.14 with -race, you can get the benefits of the -d=checkptr flag which adds instrumentation to ensure that unsafe rules are followed properly. https://tip.golang.org/doc/go1.14#compiler.

Thanks for your reply, but I want to know how to get the real reason of the crash?

It's a bug in your code, due to which a segmentation fault occurs. That's the real reason. I'm afraid I cannot explain any better than this.

OK, I expect an crash detail of my code in core dump, but currently I only got a signal handle panic.


WIP : Attempt to constrain drag to adjacent points

This currently does not work
Unless I missed something, I do not understand why the points leave the plot area
With respect to https://github.com/ScottPlot/ScottPlot/issues/1422#issuecomment-1015981553 this implementation should be able to handle the cases where Xs and Ys are totally random, at the cost of two initial sort.

I do not understand why the points leave the plot area

Don't feel too bad about this - I gave this a quick try and couldn't figure it out either!
I don't know if this will result in less effort/time or more... but what about modifying ScatterPlotDraggable to inherit ScatterPlotList instead of ScatterPlot? Maybe there's some funniness going on with arrays being referenced vs. copied, and switching to one that uses a List instead may help.

@swharden I think I'm gonna close this for the moment. I really do not see how I can improve this more at the moment.


Fix some number ranges were wrongly extracted as datetime ranges

Fix some number ranges were wrongly extracted as datetime ranges, like "one may", "twenty-one", "two to four". This issue is due to CenturyRegex has little constraints.
In order to filter out those cases, a new regex "CenturyPhraseEndRegex" was added. The mechanism behind this is if a simple number ( eg. twenty) in the middle of sentences, we consider it is possible to be a datetime range.
However, when it at the end of a phrase, it must be followed by another number (eg. thousand, hundred, etc) and cannot be recognized as a datetime range by itself.

I see you've pushed a new version as #1044. I'll close this one then.
In future PRs, please push changes/fixes to the same PR.
Thanks.


[5.6] Bump SwiftArgumentParser checkout to 1.0.3

This changes the checked-out version of swift-argument-parser from 1.0.1 to 1.0.3 in update-checkout-config.json. This pulls in a bug fix needed by SwiftPM (rdar://88097258).
Resolves rdar://88097258.
(cherry picked from commit 182e194b7eebc0dfd3929f68718ab86aea59af0f)

@swift-ci test


Breaking error in 1.11.0 - Asked for scope that doesn't exist on the resource

I use remix-auth, remix-auth-microsoft, and remix-auth-oauth2 to authenticate using Office 365 in my tenant.
After updating from 1.10.0 to 1.11.0, authentication fails with the following error:
GET /auth/microsoft/callback?error=invalid_client&error_description=AADSTS650053%3a+The+application+%27test-app-microsoft%27+asked+for+scope+%27openid%2cprofile%2cemail%27+that+doesn%27t+exist+on+the+resource+%2700000003-0000-0000-c000-000000000000%27.+Contact+the+app+vendor.

I believe this is due to the latest scope change. I noticed the same issue with https://github.com/pbteja1998/remix-auth-google. In that strat, the scope is added to the URL search params in authorizationParams as a string separated by a space. So something like openid email profile. In the constructor however, the scope is stored as an array of values. Since the latest change sets the scope after authorizationParams is called, it overwrites the scope url param by setting it to an array of values. It looks like when you pass an array to URLSearchParams that you get a comma separated list and because of that you end up with a scope that is openid,email,profile.
Long story short, I think the way to fix this is to just check if the params returned from authorizationParams has the scope parameter already.
 let params = new URLSearchParams(
      this.authorizationParams(new URL(request.url).searchParams),
    );
    params.set("response_type", this.responseType);
    params.set("client_id", this.clientID);
    params.set("redirect_uri", this.getCallbackURL(request).toString());
    params.set("state", state);
    if (this.scope) {
      params.set("scope", this.scope);
    }

becomes:
 let params = new URLSearchParams(
      this.authorizationParams(new URL(request.url).searchParams),
    );
    params.set("response_type", this.responseType);
    params.set("client_id", this.clientID);
    params.set("redirect_uri", this.getCallbackURL(request).toString());
    params.set("state", state);
    if (!params.has("scope") && this.scope) {
      params.set("scope", this.scope);
    }

Also for a temp fix that worked for me, just install the previous version of remix-auth-oauth2. "remix-auth-oauth2": "1.10.0"

Wasted a ton of time on this error too. Great.

Thank you @gregermendle. It took me all weekend to figure out how to address this issue.

@sergiodxa Can we get this fixed?


Question about query with date that includes "+"

Hello!
I tried to use Search.Repositories function.
I see it almost works well, but I tried to use query string created: that includes "+" doesn't work well.
Could you please give me some advice to use query in such situation.
Here's snippet I tried:
// c is *github.Client
result, resp, err := c.Search.Repositories(
            context.Background(),                                                                                | 18             context.Background(),
            "user:pankona created:2019-01-18T17:35:34+09:00..2019-02-18T17:35:34+09:00 fork:true",                                      | 19             "user:pankona created:2019-01-18T17:35:34+09:00..2019-02-18T17:35:34+09:00 fork:true",
            &github.SearchOptions{                                                                                | 20             &github.SearchOptions{
                ListOptions: github.ListOptions{                                                                                | 21                 ListOptions: github.ListOptions{
                    PerPage: 100,                                                                                | 22                     PerPage: 100,
                    Page:    page,                                                                                | 23                     Page:    page,
                }})

According to result of httpuitl.DumpRequestOut, the escaped query seems as follows:
GET /search/repositories?q=user:pankona+created:2019-01-18T17:35:34+09:00..2019-02-18T17:35:34+09:00+fork:true&page=1&per_page=100 HTTP/1.1
I expects 1 results by this search but it returns 0.
(I think "+" included in query's date part is not escaped correctly. I guess it should be %2B. Is my assuming correct?)

Hi @pankona - thank you for the question.
Here https://github.com/google/go-github/blob/master/github/search.go#L230
we have: query := "q=" + url.PathEscape(q) and I'm thinking that this instead needs to be
query := "q=" + url.QueryEscape(q)
Could you please try that out and report back if it solves the problem you are seeing?

Hi @gmlewis , thank you for your reply!
I tried to use url.QueryEscsape then I got escaped url as follows:
GET /search/repositories?q=user%3Apankona%2Bcreated%3A2019-01-18T17%3A35%3A34%2B09%3A00..2019-02-18T17%3A35%3A34%2B09%3A00%2Bfork%3Atrue&page=1&per_page=100 HTTP/1.1
Unfortunately I got failed from github API. Returned message is:
{"message":"Validation Failed","errors":[{"message":"The listed users and repositories cannot be searched either because the resources do not exist or you do not have permission to view them.","resource":"Search","field":"q","code":"invalid"}],"documentation_url":"https://developer.github.com/v3/search/"}
I think,

url.QueryEscape escapes "+" correctly ("+" are replaced to %2B)
But all "+" should not be escaped since some "+" may mean white space

search.go#L226 replaces white speces to "+"

Here's snippet around here:
https://github.com/google/go-github/blob/4e1a504644bfd34564773b2f8c4c2ba91c8e21ec/github/search.go#L226-L230
I'm not sure but is this expected URL and query?

"+" that means white space are remained
"+" in date part are escaped to %2B
 GET /search/repositories?q=user:pankona+created:2019-01-18T17:35:34%2B09:00..2019-02-18T17:35:34%2B09:00+fork:true&page=1&per_page=100 HTTP/1.1

I tried this URL and query, I got expected result from GitHub API.

Thank you, @pankona for investigating!
It sounds like we need a more robust escape mechanism.
If anyone would like to tackle this issue, that would be fantastic.
This would need a wide range of (table-driven) tests in the Go style to accompany the PR.
Please check out our CONTRIBUTING.md guide to get started.
Thank you!

@gmlewis
Thank you so much for summarizing. I will try to create patch for this issue!

@gmlewis
I found simple solution. Could you please review #1125 ?

Thank you, @pankona - please see my comment about #966 and #963.

@gauntface @gmlewis
As mentioned on issue, I supplied comments on search() to highlight usage between " " vs "+".
Could you please take a look?


Refactor WordPress to use ProviderDataIngester

Fixes

Fixes #599 by @stacimc
Description

This PR refactors the WordPress Photo Directory provider script to use the ProviderDataIngester class. Since this was one of our more recent provider additions, it was already fairly closely aligned with our general layout now! I removed all of the boilerplate functions the class now handles, and most of the remaining functions did not need significant updates.
Testing Instructions

just test
Run the workflow locally and ensure that data gets inserted into the catalog.

Checklist

[x] My pull request has a descriptive title (not a vague title like Update index.md).
[x] My pull request targets the default branch of the repository (main) or a parent feature branch.
[x] My commit messages follow best practices.
[x] My code follows the established code style of the repository.
[x] I added or updated tests for the changes I made (if applicable).
[ ] I added or updated documentation (if applicable).
[x] I tried running the project locally and verified that there are no visible errors.

Developer Certificate of Origin

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.

Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

@obulat @stacimc removed the filetype extraction and addressed other feedback, this should be good to go!


Adding esp_lcd component to build only serves the purpose of adding an include... (IDFGH-11715)

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.
5.0.2
Operating System used.
Linux
How did you build your project?
Command line with Make
If you are using Windows, please specify command line type.
None
What is the expected behavior?
when a component is added to IDF_COMPONENTS  what takes place when that is done? It appears that with the esp_lcd component the only thing that gets done is the include path to the header files gets added to the build. The source files are already added to the list of files to be compiled.
I would have thought that the source files only get added when a component gets added to IDF_COMPONENTS. why compile something that doesn't get used at all?
If a component has not been added to IDF_COMPONENTS and I add the include path to that component I would expect the build to fail. There should be linking errors because the source files have not been compiled.
What is the actual behavior?
I am able to successfully build having a dependency on the esp_lcd component without having to add the component to IDF_COMPONENTS. I am able to do this by adding the include path to the esp_lcd headers to the compiler options. I do not have to add any of the source files so they get compiled.
Steps to reproduce.
have code that uses some part of the esp_lcd component. don't add esp_lcd to IDF_COMPONENTS. In the cmake file add ${IDF_PATH}/components/esp_lcd/include/ to the includes.
You should get linker errors but that doesn't happen
Build or installation Logs.
No response
More Information.
This has been tested on 2 different computers both with clean clones of the IDF version 5.0.2

How did you build your project?
Command line with Make

IDF has dropped the Make build system support since 5.0: https://docs.espressif.com/projects/esp-idf/en/latest/esp32/migration-guides/release-5.x/5.0/build-system.html#migrating-from-gnu-make-build-system
If your project doesn't use esp_lcd component, and you don't want to see that component get included to the build, one quick solution is: https://github.com/espressif/esp-idf/blob/master/components/esp_lcd/test_apps/spi_lcd/CMakeLists.txt#L5 and the specify the dependency for your main component manually: https://github.com/espressif/esp-idf/blob/master/components/esp_lcd/test_apps/spi_lcd/main/CMakeLists.txt#L7

I'm compiling for MicroPython and that is the build system they use for the main entry point. The only MCU that MicroPython supports that uses CMAKE is the ESP32. The make build system sets everything up and then it runs cmake to do the build. I kind of goofed when I stated make it should have been cmake because that is what is really being used.
My question then becomes what is the purpose of IDF_COMPONENTS. If all of the source files are added by default than there is no reason to even bother with IDF_COMPONENTS may as well just add the include paths since the source files are already being compiled anyhow.
I do really believe this is a bug because the purpose of IDF_COMPONENTS is to add components to be built. If all components are already added then they shouldn't be because that is the purpose of IDF_COMPONENTS, hence that would be a bug.

adding set(COMPONENTS main)  doesn't work. The esp_lcd component still gets added to the build. I know this is happening because the only thing I have to do is add an include to the header files and my code compiles without any errors. That means the esp_lcd source files have been added to the build when I have not instructed the IDF to add them.

My question then becomes what is the purpose of IDF_COMPONENTS

This variable looks like something defined in Micropython build system, over here: https://github.com/micropython/micropython/blob/3270d856fda58585d44dd05aefa7a95551fa76cc/ports/esp32/esp32_common.cmake#L103. It is not something that is intrinsic to the IDF build system, so it's a bit hard to comment about the purpose from ESP-IDF project perspective.

adding set(COMPONENTS main) doesn't work.

@kdschlosser Could you please attach the log (preferably as a text file) of the build process, when doing a clean build?
To help us you troubleshoot this, you can also enable dependency graph generation in the project CMakeLists.txt file, like it is done here:
https://github.com/espressif/esp-idf/blob/692c1fcc52b9b936c73dead4ef0c2ea1fbdfb602/tools/test_apps/system/g1_components/CMakeLists.txt#L13
This should result in a component dependency graph being generated during the build, you should find it in build/component_deps.dot. Please attach this file to the issue as well. It should help find, which of the components brings the dependency on esp_lcd.


Edb dependent on .NET Regional Settings

If the decimal delimiter in Windows OS is set to comma ",", then some EDB methods and properties are corrupted. I found this in
class EdbPadstacks(object): in the padstack.py module.  The property rotation in the
    def create_padstack()

method cannot be parsed if the .NET delimiter is set to comma.

Hi @dcrawforAtAnsys,
I think this is not only concerning the this command and has to be addressed at higher level in Pyaedt architecture. We will see how to handle this

we will address this in pyedb. This limitation has been communicated to EDB Api dev team


Changed file on disk updates in Brackets with no confirmation (if no unsaved changes)

Replication steps:

Open file
Make changes
Save
Open console
git reset --hard
Brackets changes content of file with no confirmation and no history to retrieve changes

I'd expect to be asked for any open files if I want to update from disk. I've just lost a whole bunch of work (reset on file system to clean out some other stuff I wanted to lose).
Very annoying, can we get a fix for this issue?

Was there a recent change to this behavior? Now my brackets doesn't seem to be reloading files from disk when they're changed, and also does not show a prompt. (Running: Release 1.2 build 1.2.0-15697 (release 8f82e2a97) )

@andyburke No changes went into the file system area for 1.2 release. Which version of Brackets were you on before updating to 1.2? And also it would be very helpful if you can answer the following questions

Which OS are you in?
Are you using any folder from network share?
Are you using any third part file managers?

Here's my two cents on the issue. Brackets is currently configured the way I like my editors to work. I bounce between editors on occasion, mostly because Vim doesn't do everything perfectly, as much as everyone wants to believe that it does, and Brackets is one of the editors that I keep on my computer for times when Vim isn't the right tool. That being said, if a file changes on disk, it's probably because I did something that I want to keep, and I want my editor to just load the change without a fuss.
Other editors I have and their behavior:

Lighttable: Warns if change is made on disk to a file that has unsaved changes in Lighttable, otherwise the file is reloaded automatically.
Textwrangler: Same as Lighttable.
My Vim: Files that have changed on disk are reloaded without a fuss before Vim tries to write its changes to disk. If the file has changed both in Vim and on disk, I'm warned.
Default Vim: The default Vim configuration is to prompt the user on save. There are two prompts, and my config removes the first one. The first prompt is something like, "This file has changed on disk and in Vim, are you sure you want to save it?" If you answer no, nothing happens, and you are prompted again next time you save.
Default Emacs: You are prompted to load a file from disk when you try to edit the file again.

@nethip Unfortunately I don't recall the last version I was using. It was whatever prompted me to upgrade before the current release did.

OSX (10.10.2 (14C109))
No
No? Not sure what you mean. I am editing in folders that are monitored by Dropbox, but that has never been an issue before.
Yes. Reloaded without extensions and I am seeing the same behavior: I can change a file on disk and it will not be reloaded in Brackets.

This is a pretty frustrating bug. Are there previous builds I could grab and try? Maybe I could at least help track down when the issue was introduced.

@andyburke Thanks for trying out and posting the results. I can recall that there were some file system related issues with Yosemite 10.10.0, which was crashing Dropbox. But those issues have been resolved with 10.10.1 update. Are you experiencing the issue with non Dropbox folders as well? Also it would be great if you can check this with our previous version that is 1.1. Here is the link to the installer.
https://github.com/adobe/brackets/releases/tag/release-1.1

I vote for a preference here:  never display changed files dialog and silently update (current behavior, default) and always display dialog. A possible third option (though unlikely) is to disable file watching altogether and never update files on disk. These are the same options Notepad++ has (Settings>Preferences>MISC.).
Personally, I like the auto-update, but I can see where the prompt comes in handy. In fact, I have N++ set to prompt me, and it has saved by bacon more than once when I messed up and Brackets silently updated but it didn't.
Isn't there already a dialog for this prompt in place? In so, than all that's probably needed to add this is a preference.

@nethip Interesting, it does seem Dropbox-related: I tested things in a folder that is not watched by dropbox and Brackets behaved as expected:

Adding a file in the directory resulted in a file showing up in the bracket's folder view
Editing the resulting file in another editor resulted in Brackets properly displaying the updated contents

I downgraded to the 1.1 release you linked and I am still seeing the bug.
I then quit Dropbox, hoping that would alleviate the issue. However, it does not seem to have any effect: the bug is still persisting within the Dropbox-managed folder, even without Dropbox running.
I tried it within a few folders under my dropbox, thinking that it might have to do with the number of files within the folders (some of my projects are very large). However, that seemed to have no effect: both large and small folders under my Dropbox folder exhibit the bug (not reflecting changes on disk).
What could be special about the Dropbox folder that's causing this behavior? It also just started when I got 1.2, but persists even when downgrading to 1.1.
I'm at a loss as to what the likely cause would be.

Taking discussion of my issue to: #7638

My personal bit: I like the auto-update too, but I don't get why the undo steps are cleared, if something changes my document, I should be able to do ctrl+z to revert that change.


Track SMS Content Length

User Story - Business Need
We need to be aware of excessive content length in SMS. This ticket is to start sending the metric to statsd so we can make informed decisions.

[ ] Ticket is understood, and QA has been contacted (if the ticket has a QA label).

User Story(ies)
As a Veteran
I want SMS sent to me to be a reasonable length
So that there is less confusion regarding the notification
Additional Info and Resources

The send_sms_to_provider method creates the data for the "content" field used in send_sms
The data is changed to a string in the method call, that should probably be pulled to a little before the method call so we can set the gauge
Setting a statsd gauge in python

This is 1600 characters with at most 45 characters per line (about what Android has as default):
Lorem ipsum odor amet, consectetuer
adipiscing elit. Posuere facilisi natoque
tincidunt pellentesque rutrum aenean. Taciti
nisl pulvinar ex magnis faucibus nascetur.
Ultricies hendrerit bibendum urna ultricies
iaculis massa varius egestas. Aliquam aliquam
 egestas neque; dolor parturient maecenas
dictum. Malesuada litora aptent dignissim
egestas metus inceptos taciti. Imperdiet
congue eget donec condimentum parturient
fermentum. Lorem class varius nostra
facilisis convallis cubilia? Sem pulvinar
parturient turpis fringilla ipsum tellus.

Montes consequat inceptos hendrerit libero
consequat magna. Nec in pharetra potenti
egestas dignissim. Luctus ullamcorper mi
rhoncus erat quisque malesuada. Ex magna odio
 consequat mauris molestie metus mi
ullamcorper. Dictum curabitur amet maecenas
conubia dui parturient. Nascetur tellus enim
duis vehicula et ex sapien ipsum senectus.
Non felis adipiscing sociosqu lobortis
integer commodo luctus.

Fermentum adipiscing quis elit ut purus urna.
 Maecenas mattis euismod sem cras sagittis
pretium fermentum a. Eros montes netus
cubilia enim ligula. Vitae class mattis
imperdiet vel sem quis accumsan augue.
Dignissim quis senectus fusce sed cursus
fringilla interdum elementum nec. Ante
torquent curabitur placerat porta feugiat
donec fringilla. Nascetur nullam finibus
feugiat conubia lectus et efficitur auctor
hac.

Nunc turpis mattis netus id consectetur massa
 primis vulputate. Dictum nisi platea
penatibus semper in. Pulvinar rutrum nunc;
lacus cras ut metus. Montes in ex phasellus
penatibus purus torquent sagitt.

Acceptance Criteria

[ ] A statsd gauge is used to record/set the content length e.g. len(content)
[ ] Metric appears in Datadog
[ ] This work is added to the sprint review slide deck (key win bullet point and demo slide)

QA Considerations

Potential Dependencies

Hey team! Please add your planning poker estimate with Zenhub @cris-oddball @EvanParish @k-macmillan @kalbfled @MackHalliday @mchlwellman

I have added the gauge, and it's in Datadog, however the documentation for a gauge says that it only submits the last value received during a flush interval, the others are dropped, which isn't ideal. I think Datadog uses a flush interval of 10 seconds.  To address this shortcoming, Datadog has added a histogram metric type to it's statsd implementation.  We use our own statsd client derived from the basic python implementation.  A statsd datagram looks like this namespace.metric:value|type.  The type field is a single character, and for the new histogram type, Datadog is using h, so I added a corresponding histogram method to our internal statsd client. I've tested it against Datadog and against a local Graphite instance (which doesn't support histograms). The metric shows in both, although I don't know why it does in Graphite.

PRs for utils and api merged. Holding deploy until after resolving C&P send issue. Will be up by EOD.


Add P2PKH address derivation

This adds the module Address that should contain address derivation functions. This PR implements only p2pkh for mainnet. Should we support testnet/other networks?

Now part of https://github.com/tgalal/motoko-bitcoin/pull/7


Convert to Current Swift Syntax

Whenever I try and install the newest SwiftyJSON through cocoapods and then open the project I am prompted with convert to current swift syntax even though SwityJSON 3 supports Swift 3.
If I say later it says Module Compiled with Swift 3 cannot be imported in Swift 2.3

What version of Cocoapods you using? You should have 1.1 RC 2 if you want it to work.

I am using 1.1.0.rc.2. I reinstalled cocoapods and I no longer get the prompt to switch but it tells me "No Such Module 'SwiftyJSON'"

Fixed it by wiping the pod information inside Xcode then deleting all files associated with SwiftyJSON and starting the install from scratch. I must not have completely removed the old version


Vector support

This PR adds support for decoding Cassandra v5's new VECTOR types.
https://cassandra.apache.org/_/blog/Apache-Cassandra-5.0-Features-Vector-Search.html

Hi, any chance of making requested changes?

issues fixed in #186


Error on posting payload from Splunk SDK client service

Describe the bug
Not able to post the payload from the Splunk SDK client service
To Reproduce
Steps to reproduce the behavior:
I have used the below code to push the logs
def push_splunk():
    service = splunklib.client.connect(host='my_host', port='8088', token='my_token')
    response = service.post(path_segment='/services/collector/event',
                            body={"index": "my_index", "event": {"message": "Test Message"}})

But I am getting an error as
HTTP 400 Bad Request -- b'{"text":"Invalid data format","code":6,"invalid-event-number":0}'
I have referred to the below links but nothing helps me to solve the problem, Can you please help me with this?
https://community.splunk.com/t5/Getting-Data-In/HTTP-event-collector-error-with-data-format/m-p/352755
https://medium.com/adarma-tech-blog/splunk-http-event-collectors-explained-2c22e87ab8d2
Expected behavior
It should successfully post the content to Splunk
Logs or Screenshots
HTTP 400 Bad Request -- b'{"text":"Invalid data format","code":6,"invalid-event-number":0}'
Splunk (please complete the following information):

Version: [8.0.2]

SDK (please complete the following information):

Version: [1.6.19]
Language Runtime Version: [Python 3.6.13]

Hi @mvgadagi , event API endpoint requires the body parameter as String, here in the code JSON is being passed for body which is the reason for the error.
--> Here just add the body within quotes and the event will be pushed to the Splunk
i.e - body='{"index": "my_index", "event": {"message": "Test Message"}}'
Here are some references for the same
--> Test case for Event where JSON.dumps is used for the message being passed in body
--> Splunk Docs for Event Formats, in the curl example commands given, the event body being passed is covered within quotes(')
Note:- the references you referred also use the event body i.e after -d in curl command is covered within quotes
Would request you to try with the suggested change and let us know in case you still face the issue.

Closing the Issue as we haven't received any response. @mvgadagi Please reopen the Issue if you still face the error.


Call tracer should ignore calls made from the library it uses

I spotted that the call tracer ignores the debugger's methods but not the methods from related libraries, like irb for coloring. So it looks like this:

This PR fixes the issue:

@ko1 I've rebased it on the master and it should be ready for review now.


Failure to run-tests for tvOS due to missing framework

I'm running xctool 0.2.7 and Xcode 7.1.1.
I'm guessing that tvOS is not supported, but I ran into a problem that I'm hoping might be simple to address so I figured I'd ask.
Here's the message:
Failed to query the list of test cases in the test bundle: dyld: Library not loaded: /System/Library/Frameworks/UIKit.framework/UIKit
  Referenced from: /Users/bclark/myapp/build_output/tvos/test/dev-sdk-appletvsimulator/Debug-appletvsimulator/MyApp.app/MyApp
  Reason: image not found

This framework doesn't exist in /System/Library/Frameworks - I'm guessing this is some place in xctool that's falling back to logic for OS X when detecting a non-iOS SDK. The tvOS SDK is structured very similarly to the iOS one.
Running the equivalent command directly with xcodebuild works as expected.

Is your test bundle linked against correct framework?
P.S. Yes, we aren't going to support tvOS and tvSimulator in the nearest future. Unless someone from open source community will jump and do it. I doubt it would be super hard.


Daily limits?

Are there any daily limits?  I've tried to translate a long file and it will translate only a portion of it.  Then I try running 'trs' from command line and Google returns blank responses to any requests.  I wait 24 hours and then it works again.  Any advice?

Could you catch this and report to stderr?


Grant Application: Roloi - XCM Payment Automation

Project Abstract
Our project enables the automation of ink! smart contract transactions in a recurring way. Using XCM, we can schedule transactions on Shiden (Astar) with the assistance of Turing's (OAK) functionalities.
We will deliver a generic project as a public good that teams can use to automate cross-chain transactions:

A Typescript Next.js UI with abstractions that simplify the most tricky parts for developers such as Accounts connection, Proxy Accounts, and XCM messages creation.
An ink! Smart Contract that handles token transfers between Shiden accounts.

Grant level

[ ] Level 1:  Up to $10,000, 2 approvals
[ ] Level 2:  Up to $30,000, 3 approvals
[x] Level 3:  Unlimited, 5 approvals (for >$100k: Web3 Foundation Council approval)

Application Checklist

[x] The application template has been copied and aptly renamed (roloi-xcm-payment-automation.md).
[x] I have read the application guidelines.
[x] Payment details have been provided (bank details via email or BTC, Ethereum (USDC/DAI) or Polkadot/Kusama (USDT) address in the application).
[x] The software delivered for this grant will be released under an open-source license specified in the application.
[x] The initial PR contains only one commit (squash and force-push if needed).
[x] The grant will only be announced once the first milestone has been accepted (see the announcement guidelines).
[ ] I prefer the discussion of this application to take place in a private Element/Matrix channel. My username is: @_______:matrix.org (change the homeserver if you use a different one)

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Hi @Noc2! We updated our application:

Fixed image
Updated technical details section to better describe the current process
Added a new "Our solution" section to describe our generic software solution
Added a new "Wireframes" section to display our UI initial wireframes

Thanks again for your useful feedback!

Hi @Noc2 @keeganquigley!
Thank you both for your quick feedback. We’ll stay tuned for any news.

@pcorrado-np thank you for the application. I think the scope of the grant interesting, however the price charged is very high. The wireframes presented has 5 pages and the price for develop them (M1) is almost 42k. Furthermore, M2 scope is to develop a smart contract, which probably you already have part of it developed in the previous grant or because of the kind of business NeoPower runs. Adding XCM to it should not be a big deal, and the price to do it is 28k. Could you consider going for a level 2 grant?

Hello @dsm-w3f, thank you for taking the time to review our application.
We truly appreciate your feedback and understand your concern regarding the pricing.
Regarding Milestone 1, the main value of the proposal is about the Typescript logic, React hooks, and code encapsulation to ensure seamless integration with XCM in a generic way. Additionally, our proposal includes the development of the pages to create proxy accounts in a straight-forward way from both the developer and user perspective. What takes most of the estimation and effort in this milestone is the development of fully-typed reusable code for other teams.
Regarding Milestone 2, we don’t have an existing smart contract that does this. Our previous grant last year was about token streaming, and we don’t have any other smart contracts developed for Roloi so far. This development would be completely new to enable on-chain recurring transactions and integrate Kusama chains. This is important for our project because it requires robust testing since the canary DApp will interact with actual money.
From our experience while learning and teaching the Polkadot tech stack, we made sure that the estimations are accurate considering documentation, design, development, peer reviews, and testing.
We firmly believe that the tools developed through this grant will not only be valuable to our own product, to bring on-chain payment automation, but will also be beneficial to numerous other teams in the ecosystem. As part of our commitment to the community, we will actively share this in the various developers' communities we are engaged with, specifically in LATAM.
We appreciate the opportunity to apply for this grant, and we remain open to further discussions or clarifications if needed.
Thank you for your consideration, and we look forward to your response.

Since I haven't used OAK before, I'm not aware how recurring transactions are created and how much of this workflow can be automated away. It seems to me that the creation manual top-up would still be necessary. If I'm reading this correct, you want to build a smart contract to simplify managing recurring payments and a UI to simplify the setup. Is that correct?

Hi, @semuelle. Thank you so much for the review!
OAK recurrently triggers XCM transactions to the origin chain according to the previously received recurring settings. At the origin chain side, a proxy account takes the transaction that includes the encoded extrinsic to be executed (in this case, the extrinsic will be a smart contract transaction).
The creation and funding of proxy accounts will be integrated into the configuration of the recurring settings and will be simplified for the user with the UI.
And you are right:

The Smart Contract will be in charge of storing and validating the users recurring settings and the execution of transfers.
The main goal of the UI is to simplify the complexities of XCM and the OAK automation process, providing a great UI/UX for configuring recurring payments.

Thanks again for your time! We’ll stay tuned.

Thanks for the replies, @pcorrado-np. A couple more questions:

Could you integrate the wireframes directly into the application document? Otherwise they will soon be lost.
The wireframes show a list of "incoming" payments. Incoming where? How does that work? I assumed the list would be showing the payments set up in the smart contract.
 Wouldn't this architecture be possible by directly interacting with the pallet?

Hi, @semuelle. Thanks again for your questions. We really appreciate the feedback to improve our application.

Thanks! We now updated the application document by adding the wireframe images.

Users will be able to connect their wallets to the UI. The incoming and outgoing payments refer to the incoming and outgoing transfers of the connected wallet. Not the entire contract history. This will be useful to have a usage example of a message that selects the payment flows of a given wallet.

We want to leverage ink!’s capabilities to enable on-chain validation rules for any business model other teams may have. This allows for more complex models other than just scheduling token transfers. For now, the smart contract will validate that the scheduled time matches with the executed time, and not just trust on a third party message.
In the future, we also want to include the XCM communication with OAK through the contracts.

@pcorrado-np this looks really interesting. The proposal is well-written and describes most concepts concise and clearly. However, I do the following 2 questions:

what's the transport mechanism you're going to use to send XCM messages, I'd assume it's HRMP? Is it roughly the same process as described in this blog article that describes the integration of oak with moonriver?
there are various types of hooks in react, hence the term "hook" is quite generic. Could you elaborate on the hooks you're planning to implement? Maybe some rough pseudocode could help as well, just to understand better how they're going to work.

Finally, I agree that the price is rather high. But before giving a decision on my approval, I'd like to see the above unclarities resolved.

Hi @takahser. Thanks for the review!
Exactly! The process is similar and uses HRMP.
Unlike the Moonriver process, we set up the task execution and send it through XCM via UI code using Polkadot.js. When the task is triggered by OAK, a proxy account on the Astar side will execute the configured transaction.
Regarding the hooks, we'll create our owns. Some tasks that can be encapsulated are:

Creating proxy accounts on both sides
Depositing funds into these accounts to cover fees,
Wrapping the recurring transaction to be executed via Astar Proxy Account
Transmitting the recurring task configuration via XCM.

These are some of the hooks that we have in mind, but we are certain that many more will come up during development. We’ll also make sure to validate the naming and data modeling according to the standard guidelines.
For example:

Thanks again!

Hi, @takahser! We greatly appreciate your review and questions.
Here are some additional comments to offer more insights about our proposal:

Regarding HRMP and your explanations on the React Hooks: I believe it'd be good if you could integrate this into the document. It's better to have it there since in the convo it runs at risk of getting lost.

We can update our proposal with that information. Thanks for the feedback!

So the key difference between your and Moonriver's implementation would be the web UI, is that correct?

No, not exactly. Moonriver's implementation takes a pallet-side XCM management approach. This way a Solidity contract can create an XCM message in order to send to OAK and schedule a future transaction. OAK receives and triggers the Solidity contract message on Moonriver's side when it's time to execute the transaction.
This implementation is only useful for Solidity contract developers that will fully depend on the usage of a pallet.
With our implementation, we will provide XCM Typescript tooling for UI developers. Currently, using XCM on the UI requires a complex setup process using JavaScript.
This is actual feedback from developers that attempted to build products using XCM (including ourselves), that is why we want to improve this experience.

Apart from that, I agree with @dsm-w3f that the price is rather high. I think 6 months for M1 is quite a long time for writing some hooks, the payment page which is basically a web form and some wallet integration. Especially if I consider that you're planning to have 4 developers working on it simultaneously while there seem to be only 2 devs in your team.

We see now that “React hooks” doesn’t describe our M1 proposal well enough. We will implement React Typescript tooling to work with XCM. This includes hooks, types, state management and generic components.
The UI pages are intended to showcase a use-case example of recurring payments, while our main focus is on the reusable modules of our project.
About our team, we are only describing our core team on the proposal. But for the FTE we are also considering designers, engineers and testers that work with us.
The estimation is 6 weeks (not months), considering design, development, documentation, peer reviews and testing. Based on our experience working with cross-discipline teams and the Polkadot infrastructure, we believe that the estimation is accurate.

I have a similar feeling about M2. Especially since it seems you can reuse a significant portion of the code from the Moonriver implementation. Don't get me wrong, there's nothing wrong with that, but it lowers the value your project adds to the community in contrast to a project that comes up with something completely novel.

Moonriver’s announcement was about opening an HRMP channel to be able to generate an XCM message and pass it to OAK at the pallet level.
That is not providing abstractions for transaction scheduling, just a communication channel between Moonriver and OAK.
That is why we can’t reuse the code for M2. We will develop an ink! smart contract that will store the scheduling configuration of each transaction, validate and execute when OAK triggers the smart contract message. This is something completely new for the ink! ecosystem.

For these reasons I'm not going to approve it either at the current price point. I think lowering it to a level 2 grant would make sense here. Given you already got 3 approvals, a level 2 version of your proposal would be approved.

We understand the concern about the price point and appreciate your feedback. We also believe that some relevant technical details of our proposal weren’t clear enough based on some of the questions we received.
We genuinely believe that this project will introduce a simpler way for developers to build automation products using Typescript + ink! + XCM. Our main goal is to add value to the community, but we also need to make sure that our team members are fairly compensated for their work.
We can refine the description of our proposal and deliverables to provide a clearer explanation of the solution and the work required.
Would you reconsider your decision based on the information provided? A level 2 grant seems too low for the work involved.
Thanks again for taking the time and effort to provide valuable feedback! 🙏🏻
We are open to providing more details and further reviewing our proposal to start working on this project.

@pcorrado-np

About our team, we are only describing our core team on the proposal. But for the FTE we are also considering designers, engineers and testers that work with us.

Keep in mind that our program is tailored to support software development efforts, but not design efforts. Also, if there are more software engineers on your team, you're supposed to list them as well. Quote from our README:

You will be asked to provide the GitHub profiles of your team members as part of your application, which we will examine for past activity and code quality.

I have read and hereby sign the Contributor License Agreement.

Hi @dsm-w3f @takahser @Noc2 @semuelle @keeganquigley,
Thank you all for your valuable feedback.
We've decided to lower the proposal's price by removing 1 FTE for each milestone.
NeoPower will cover the costs of the remaining resources.
We are looking forward to starting working on this project.
Thanks!

To keep you informed about the status:
We are making significant progress with the Milestone 1.
We have detected a deviation from our original estimation and we are currently targeting to deliver Milestone 1 during the 2nd week of November.
We’ll keep track of the status through this PR.
Thank you for your support!

Hi @pcorrado-np how is milestone 2 coming alone?

Hi @pcorrado-np how is milestone 2 coming alone?

Hi @keeganquigley,
It's taking longer than expected but we are making progress with the features. Sorry for the delay and not reporting that.
We hope to reach the second milestone the week of February 5.
We'll keep you touched.

Hi @keeganquigley!
Status update:
We are close to finalizing the delivery of Milestone #2 in the upcoming days.
Everything is working as expected, and now wrapping up docs and article.
Thanks for your patience!

Sounds good @pcorrado-np thanks for the update, much appreciated!


ReAnalyzeVersions should re-fetch repos if necessary

There are some errors that don't clear when re-analysing versions. I.e. the version will be re-tried in every batch because the exception will prevent the version from being updated and falling out of the candidate selection:
[ WARNING ] Error: Git.tag failed: Shell command failed:
command: "git tag"
path:    "/checkouts/github.com-migueldeicaza-swiftsh"
message: "ShellOut encountered an error
Status code: 1
Message: "/bin/bash: line 0: cd: /checkouts/github.com-migueldeicaza-swiftsh: No such file or directory"
Output: """ (id: EE02A7C5-C1A8-4663-B644-128F25E3ABD3) [component: re-analyze-versions]

This can be mitigated by manually checking out the repo to the expected directory but really the re-analysis stage should catch the error and clone the repo instead.
We don't re-analyse often, so if this turns out to be anything more than a trivial fix we should just live with it. Normally, checked out directories shouldn't disappear anyway.

Some more details: I believe this happens when we call getPackageInfo in ReAnalyzeVersions and the repo has changed name between what the current version and the checkout the normal version processing has checked out into.
It would be tricky to adjust for this at this time, because we're re-using the analysis steps from Analyze.swift in ReAnalyzeVersions as much as possible. Due to the complicated nature of the EventLoopFuture function signatures, it isn't easy to just drop in an error handler that catches this exception, checks out the correct path, and retries.
I suspect this would be a bit easier once we've moved these two pieces to async/await. Will revisit this then.

This is actually a fairly frequent problem and can hinder re-analysing all versions. The problem is the following:

in prod we run a 3 node swarm
only one node runs the analyze process, which is the one that checks out the repos
this node changes from time to time as we deploy
effectively all nodes end up having more or less all repos checked out but there are holes depending on which nodes processed given packages
because we run re-analysis on a single node, that node will come across holes
when running with something like --limit 10 ones there are 10 packages stacked up that can't be processed due to missing checkouts the re-analysis process stalls

A workaround for now is probably to run the re-analysis process on all three nodes such that one of them will have the required checkout. That's not great but since we're close to being done now probably the best way to proceed for now.

Even easier than running it on multiple nodes: just run analyze for the package before running re-analyze-versions.


[CELEBORN-543][PERF] handleUnregisterShuffle may occupy RPC thread

What changes were proposed in this pull request?
Fix handleUnregisterShuffle may occupy RPC thread
Why are the changes needed?
Does this PR introduce any user-facing change?
How was this patch tested?

ping @RexXiong @waitinfuture

ping @pan3793 @waitinfuture @RexXiong

IMO it's not a good idea to call lifecycleManager.unregisterShuffle(shuffleId, true) in setStageEnd, because they are separate events, and in most cases unregister shuffle happens after stage end, so I think it's rare that handleUnregisterShuffle encounters that isStageEnd is false.