Shift not working in standard mode

Hi Cedrozor,
It seems that in standard mode shift isn't working properly. CAPS lock seems to work - although often it's reversed. But Shift does not work most of the time. Often double pressing and then holding shift may work, sometimes holding shift for a few seconds before typing works, and other times it doesn't. Often after getting shift to function, letting go of shift and typing still results in capital letters.
Is there any chance of a fix for this issue as well as issue #204. I'd be happy to pay you a few hundred dollars to get this fixed.
Cheers,
Will

Hi, what are the local (OS, browser) and remote (OS) specs ?

Hi,
Using Chrome locally. Myrtille is running on a Windows 2019 server, installed using your msi file.
The RDP is using GUID/PCB ID to connect via internal RDP (port 2179) on hyper-v back end.
We have a solution that works fine and has no keyboard issues with wsgate/freerdp but it hangs frequently due to them using outdated EHS integration – and hasn’t been updated in quite a long time.
Cheers,
Will
From: cedrozor<EMAIL_ADDRESS>Sent: Wednesday, 27 May 2020 5:00 AM
To: cedrozor/myrtille<EMAIL_ADDRESS>Cc: willkruss<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [cedrozor/myrtille] Shift not working in standard mode (#221)
Hi Will, what are the local (OS, browser) and remote (OS) specs ?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/cedrozor/myrtille/issues/221#issuecomment-634215034 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7W7NB3WC7IPE6GJNTCNBTRTQGTVANCNFSM4NJDC5TA .  https://github.com/notifications/beacon/AA7W7NEJDFXLH2NBE46ZO3LRTQGTVA5CNFSM4NJDC5TKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEXGVU6Q.gif

Hi mate,
Just letting you know we have gone with the wsgate/freerdp solution at this stage. Your solution does look good, but requires the keyboard mapping issues to be fixed.
Cheers mate,
Will
From: cedrozor<EMAIL_ADDRESS>Sent: Wednesday, 27 May 2020 5:00 AM
To: cedrozor/myrtille<EMAIL_ADDRESS>Cc: willkruss<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [cedrozor/myrtille] Shift not working in standard mode (#221)
Hi Will, what are the local (OS, browser) and remote (OS) specs ?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/cedrozor/myrtille/issues/221#issuecomment-634215034 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7W7NB3WC7IPE6GJNTCNBTRTQGTVANCNFSM4NJDC5TA .  https://github.com/notifications/beacon/AA7W7NEJDFXLH2NBE46ZO3LRTQGTVA5CNFSM4NJDC5TKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEXGVU6Q.gif

Hi @willkruss, in order to help me to fix the keyboard issues, could you please tell me what was the OS used with Chrome and what was the OS of the target server? (Windows or Linux xRDP)
I'm trying to cross match the issue with https://github.com/cedrozor/myrtille/issues/222
Thanks

Hi,
Windows 10 with Chrome.
OS of the host server is Windows 2016 Hyper-V RDP (connection using a PCID/UID) on port 2197.
The eventual target VM is both Linux and Windows, same behaviour on both.
Thanks,
Will
From: cedrozor<EMAIL_ADDRESS>Sent: Sunday, 7 June 2020 12:49 AM
To: cedrozor/myrtille<EMAIL_ADDRESS>Cc: willkruss<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [cedrozor/myrtille] Shift not working in standard mode (#221)
Hi @willkruss https://github.com/willkruss , in order to help me to fix the keyboard issues, could you please tell me what was the OS used with Chrome and what was the OS of the target server? (Windows or Linux xRDP)
I'm trying to cross match the issue with #222 https://github.com/cedrozor/myrtille/issues/222
Thanks
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/cedrozor/myrtille/issues/221#issuecomment-640072605 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7W7NESONBVZJGZID775TLRVJJM5ANCNFSM4NJDC5TA .  https://github.com/notifications/beacon/AA7W7NEMFC6CVEPJI6EFQMDRVJJM5A5CNFSM4NJDC5TKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEYTLXHI.gif

Hey, we got the exactly same problem, Shift is not working. Tested with Chrome, Edge and Mozilla with latest myrtille version. Myrtille is installed on Windows Server 2019 Core

the next version will include an on-screen virtual keyboard in case of any keyboard or send text issue.


[WebView] Use WKWebView instead of UIWebView

It would be nice if there were a way to use WKWebView instead of UIWebView when using the <WebView> component, as WKWebView has substantial performance benefits.

It also comes with problems. Would like to differentiate between the 2 as separate components.

@drkibitz What kinds of problem? The only thing I can think of is that it requires iOS 8. Maybe we can choose UIWebView when on iOS 7 and lower, otherwise use WKWebKit.

WKWebView is known to be "broken" or at least unfinished. Here's a relevant thread about why (for instance) Google Chrome for iOS still uses UIWebView: https://code.google.com/p/chromium/issues/detail?id=423444
A relevant fragment of the discussion:

A partial list of regressions relative to UIWebView that we’re currently aware of:

There is no cookie management API, which means there is no obvious way to clear/manage cookies
Protocol handlers no longer work, which breaks several very important features
POST bodies are missing from delegate callbacks, which breaks certain aspects of form handling

We should have WebViewIOS and WKWebViewIOS instead of a generic WebView component. Anyone up for the renaming and providing both implementations?

While still buggy, it looks like Telerik has implemented a WKWebView plugin for Cordova that gracefully degrades to UIWebView in <= iOS 7. I agree with @vjeux that we could go ahead and split the components, setting the stage for a working WKWebView.

+1 on @vjeux's suggestion.

Why do we need both?

@spicyj we should be able to expose all the platform features and let the developer pick the one that makes sense for their use case.

I agree in principle but my understanding is that UIWebView is preferable only if you need a synchronous API (or need to do low-level configuration), neither of which are super compelling reasons from the JS side. I'm okay either way I guess but the UIWebView/WKWebView vs WebViewIOS/WKWebViewIOS suggested naming is odd to me.

Isn't there still an issue with WKWebView in that it cannot load from a local resource, it has to load over http? Telerik included a mini-http server in the Cordova plugin to workaround this issue, as far as I remember. Just mentioning, as it may not be as simple as a drop-in replacement.

+1   I use Semantic-ui  to build my mobile app on iOS , and it kind of slow , maybe the WKWebView will speed it up

Does anyone happen to know if the JIT is enabled in JSContext on desktop?

^ cc @ide

Labeled as good first task as per @vjeux's comment:

We should have WebViewIOS and WKWebViewIOS instead of a generic WebView component. Anyone up for the renaming and providing both implementations?

I am reserving this!

@brentvatne, thanks for the label! Reserving as it seems nobody is currently working on it.

I second @joewood concern about the WKWebView being buggy. You can't load html from a local resource, nor does it support the html5 app cache, so pages cannot work offline.  This is not an issue with standard WebView.

I'm going to close this because it doesn't seem to be going anywhere - I think there would be some value in having a WKWebViewIOS component so it's available as an option at least - if anyone has any use for this and wants to implement it, please do so and open a PR :smile:

Agreed @brentvatne it would be nice as a separate component.  Also, FYI in IOS9 there will be a SFSafariViewController

@oveddan - ooh, looks cool! shares cookies with Safari

@brentvatne I'll work on creating WKWebView component - I think it could be useful with something like what @joewood mentioned Telerik did, with some sort of min-http server, that serves a local file over an http server.  If the packager can serve html, then someone can build something similar for these static files. That mini-file-server would probably be a separate project.

@oveddan Any progress on this? I'd like to use WKWebView for SceneVR because the javascript performance is much higher than UIWebView.

@bnolan Sorry, no I haven't had a chance to work on this, nor do I have time in the near future to.

I am actively working on this, and have a nearly complete implementation of the API currently supported by React Native for UIWebView. I should be ready to open source the project next week, and then I'll open a PR for it here.
This is my first attempt to contribute to React Native, so I would definitely expect the PR to go through no small amount of scrutiny by the core team as I revise it, but I plan to release it as a standalone first.
There is also qrush/react-native-wkwebview which is barebones but works if you need something right away. I used it for a little while myself before deciding to fully implement it.
The main differences between Nick Quaranto's project and what I'm working on are:

Taking into account iOS API differences between UIWebView and WKWebview, support for as much of React Native's current API from WebView.ios.js as possible: bounces,  html, injectedJavaScript, onNavigationStateChange, and all other events
AQWebView.m has built-in spinners for loading and pull to refresh behaviors. I've removed these and exposed the events, so you can implement your own state transitions using onNavigationStateChange and renderLoading, as with React Native's WebView.
Improved internal handling of local paths and file:// URLs. WKWebView has various issues with this, depending on the version.

It's based on React Native master but I'm backporting it to v0.14.x as well.

Just want to chime in that WKWebView support (in addition) would be most welcomed, for performance reason. For http webviews, WKWebView in iOS 9 is pretty much superior now with no downside.
And for reference, in iOS 9 there's also Safari View Controller which chiefly gives users webviews that shares same login cookies as from Safari itself, but at cost of no ability to inject JS or control what goes on inside that Safari View. There's support for RN at https://github.com/naoufal/react-native-safari-view

@jordansexton do you have progress on this? Would be an enabler for me. I am a backend dev, learning react.js seems a bit steep to me. So I am looking to have vue.js in a (decent) webview and use react-native as a backend and for heavy lifting. Sorry all.

I completed this but never open sourced it as I no longer use it. I use the
stock React Native webview based on UIWebView. I may be able to release it
if you're interested, but I recommend against using it unless you really
know you need WKWebView. The limitations of it are extreme and even that is
understating things. No cache control or support for NSUrlProtocol inside
of your app is a dealbreaker.
Jordan
On Apr 7, 2016 7:49 PM, "Robert Jefe Lindstädt"<EMAIL_ADDRESS>wrote:

@jordansexton https://github.com/jordansexton do you have progress on
this? Would be an enabler for me. I am a backend dev, learning react.js
seems a bit steep to me. So I am looking to have vue.js in a (decent)
webview and use react-native as a backend and for heavy lifting. Sorry all.
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/facebook/react-native/issues/321#issuecomment-207150673

@jordansexton I'd love to see what you worked on!

@jordansexton Would also like to see it!

Sure thing, I'll check it into a public repo today (Tuesday) if I have time, and tomorrow if not.

This is alpha software. WKWebView itself was unable to meet my needs, and I don't use it any more.
Since that's out of the way: npm install react-native-wkwebview
https://github.com/jordansexton/react-native-wkwebview
I believe it was based on RN 0.20 (as seen in the dev dependencies of package.json). I can't make any guarantees, but I'll see what I can do to update it for more recent RN versions, and also to turn it into an iOS lib for easy integration. You should be able to (ab)use it right away by just dragging files into your Xcode project.

The original code was very closely based on React Native's WebView for iOS at the time it was written (for React Native 0.14, I believe), and then updated a bit for 0.20, but not since. This was internal to another project so you won't find any helpful git history, and you will find wacky artifacts in this library against numerous versions of RN. These are what I'll be trying to smooth out, but it's definitely usable if you're willing to hack around. I suspect it's currently compatible with RN 0.19+ but I can't say for sure.

Thanks @jordansexton, I will be looking more into your library over the next few weeks. A quick glance through it does look similar to what RN has internally. We will potentially be using this for a project at the company I work at and if you're unwilling to maintain it we may be able to take over maintenance. Though would appreciate any work in getting it working with latest RN releases.
What sort of wacky artifacts are we talking about?

Haha similar indeed. I basically just copied the RN code as of 0.14 originally and just changed things as needed or eliminated things where they weren't supported in WKWebView or needed for my use at the time. There is also a small addition that was made to handle target="_blank", and mailto: or other non-http/s links.
I updated the code for 0.20 essentially using the same hacky process. It's hard to say what kind of wacky artifacts you may encounter (too wacky to think about!) but it really just comes down to API changes in RN that may not be reflected in the code or my knowledge of RN (limited) or iOS (more limited).
I say all this mostly as a big disclaimer so that when somebody says "it doesn't work", I can just point to this and be like "yah, I told you", and then try to fix the issue once I'm aware of it.

We have recently open sourced our implementation of WKWebView on React Native. The implementation derives from React Native's WebView and some code from @jordansexton 's implementation. We try to maintain maximum compatibility with WebView. We've also implemented onShouldStartLoadWithRequest, which uses WKNavigationDelegate internally.
We use this component in our app and will try to make sure it works with latest React Native. (our App usually upgrades to latest RN asap). For anyone interested, here is the repo: https://github.com/CRAlpha/react-native-wkwebview (currently not on NPM).

I really believe we should reopen this issue. (or #10456)

WebView was deprecated by Apple, WKWebView is strongly recommended:

Fix performance issues (it uses the Nitro Javascript engine)

New iOS features, like 3D Touch support

In my view, it's time for it to be the new default.

@brunolemos agreed. the current webview implementation has issues with the javascript bridge as well https://github.com/facebook/react-native/issues/11594
the only fix I see is something like this http://www.joshuakehn.com/2014/10/29/using-javascript-with-wkwebview-in-ios-8.html

Is there any kind of discussion with the RN core team about this? Coming deprecation of UIWebView, likely in iOS 11, ought to be addressed.

Just use https://github.com/CRAlpha/react-native-wkwebview it already exists and is up to date.

I remember trying this package, but it either didn't build or run. That was a few months ago though.

It's a time! We definitely have to replace UIWebView with WKWebView.
We can do this in two simple steps:

Create a new fully compatible component WKWebView; live with it and test it carefully;
Replace old one with it! (Previous implementation can live as a separate package.)

Honestly, we, at Facebook, do not use this component so much, so it is unlikely that I or one of my colleagues will have time to dive into this, so... PRs is welcome!

I made an attempt in #16535
Unfortunately I am a JS Dev, maybe someone could look this PR over and help improve it?

As @shergin mentioned earlier in the thread, we don't use WebView much here at Facebook. You can tell by the vast quantity of issues that are opened against WebView here on the repo.
One of my responsibilities is to ensure the open source community's needs are met. Be assured I'm aware of the need for a better solution here. As it happens, we don't have plans to make changes here so I am updating the issue to make it clear the contribution has to come from the community for the time being.

@hramos are you interested in this PR: https://github.com/facebook/react-native/pull/16792? I can rebase to latest and merge some other bugfix commits if there is interest in taking the changes.
@grabbou mentioned on that PR that it was unlikely to get merged, so I was going to make a standalone project when I got the time.

How about merging https://github.com/CRAlpha/react-native-wkwebview into RN proper? It's still iOS only, but since we are talking about WKWebView that's the half that matters. It's a very active repo (illustrating exactly the demand for a better WebView), and have a better (working) 2 way messaging between React Native and WkWebView than RN's own, IMHO.
I'm using https://github.com/alinz/react-native-webview-bridge in my app WonderSwipe, and I'd migrate to react-native-wkwebview when I have time.

@bowerman0 I echo @ide's comment and it's something I've commented on other WebView issues. We are reluctant to import PRs that touch WebView, and would rather have the community work on third party modules to fit their needs.
On our side, we do need to agree on what the future of the build-in WebView module should be in order to avoid any further confusion here when people try to use this module as opposed to a better supported third party module.

@fungilation how do you think react-native-wkwebview might benefit from being merged into the core React Native repo (aside from an increase in visibility)?

That and de-fragmentation. If react-native-wkwebview is more actively developed and better (in multiple ways IMO) than RN's (stagnant) WebView, doesn't the community benefit from a better out-of-box iOS WebView that's built-in and will be more used/supported?

While WebView is the "official way to do it", that's where 99% will go. IMO not necessarily it need to be merged, a note in the docs like it was made with react-navigation could be enough. But merged in the core passes more confidence that it will be maintained.

Maintainer of react-native-wkwebview here. I agree that there are limited benefits for this repo to be merged into core, one of them being Expo support (Expo still uses UIWebView). But I think this is more of an issue for Expo, rather than React Native.
IMO, React Native should focus on the "core" bridges and this component should happily live as a 3rd party component. This makes it easier for people to actually contribute to this project. Also, it means flexible release schedule (as opposed to the monthly release schedule of React Native)
And I agree with @brunolemos that improving React Native docs is a better way to make people more aware of the decision regarding UIWebView and WKWebView and point them to the solution.

Could react-native-wkwebview  be "linked" in RN proper (in documentation if not in code, though being built-in without install would be nicer), as a new WebViewiOS in RN? Then there's no downside even for "touching" existing WebView.

@hramos The WebView should either be updated to WKWebView or move out of the react-native package. (Both are equally ok with me, personally.) The UIWebView-based WebView can cause unexpected problems for developers that use it. Because the official docs use WebView, most developers probably do not look for alternatives until they encounter problems.

UIWebView is officially deprecated in iOS 12 (I don't think it was in iOS 11, but forgive me if I'm wrong about that) https://developer.apple.com/documentation/uikit/uiwebview

Good riddance. Get with the times and migrate to WKWebView.

@hramos any word on if there are going to be instructions on building on iOS from source anytime soon? I would love to poke around on this and other issues, but as I believe we discussed before, the only instructions to build from source are for Android and they are here https://facebook.github.io/react-native/docs/building-from-source.html. Am I missing something? Are there any guides for building from source for iOS?

Also/ or if someone could explain it to me, I would be happy to make a PR to the docs

@ckknight If lack of action is blocking a major feature for you, consider using https://github.com/CRAlpha/react-native-wkwebview. We are using it in production with no problems.

👋 related to this whole WebView topic, just wanted to let you know that "something is moving": within the core we decided to try an planning an "extraction" of this component into its own separate repo for easier maintenance.
You can follow along and give feedback here: https://github.com/react-native-community/discussions-and-proposals/pull/3

Because https://github.com/react-native-community/react-native-webview uses WKWebView by default, I'm not planning to migrate this issue over there. This issue can probably be closed.

Sounds good to me - closing

Should we consider WebView to be officially deprecated from RN and migrated to react-native-community?

Basically, yes. It will be properly announced when 0.57.0 is released but yeah, that's what will happen.


Fix Prometheus TSDB file layout explanation in comparison doc

https://prometheus.io/docs/introduction/comparison/ has "Prometheus also creates one local file per time series". This is no longer true for Prometheus 2.x and needs to be updated.

The comparison doc is out of date in a few places, this is on my todo list.


Update CONTRIBUTING.md

Related Issue
Closes: #653
Describe the changes you've made ✍
Fixed the typo

Hi @Shalini469717, please fix your Pull request title, we have talked about it in the Pull request section of the Contributing Guidelines.

Fixed it. Maybe we can mention that the title should start with a lowercase letter as well in the Lint. I changed the title again once I got the error.

Fixed it. Maybe we can mention that the title should start with a lowercase letter as well in the Lint. I changed the title again once I got the error.

Well that is the reason we have put in our docs, and all the first time contributors are specially expected to go through the docs.

Amazing work, This looks good to me, will be merged soon 🎉.

Thank you for the help


recipe compoundVariants responsive

Description
When compoundVariants are added to a Recipe, it loses its responsive property and values cannot be passed in a responsive manner.
//Type '{}' is not assignable to type '"BRAND" | "BRAND_GRAY" | "STRONG" | "TEXT_STRONG" | "OBVIOUS" | "SUBTLE" | undefined
// In this case it is good
export type AvatarVariantProps = {
[key in keyof AvatarVariant]?: ConditionalValue<AvatarVariant[key]> | undefined;
};
//But this problem occurs when compoundVariants are added
export type ButtonVariantProps = {
[key in keyof ButtonVariant]?: ButtonVariant[key] | undefined;
};
And ConditionalValue type is not applied
JS Framework
React (TS)
Panda CSS Version
0.24.2
Browser
No response
Operating System

[ ] macOS
[X] Windows
[ ] Linux

@mahanfakhimi Please add a playground reproduction that showcases the issue

This is by design @mahanfakhimi.
Recipes with compound variants cannot be applied responsively. I believe it's mentioned in the docs as well.
Consider rendering two versions of the same element and show/hide based on breakpoints, or build the responsiveness into the recipe variants.


App start port should be change from ACKEE_PORT to `PORT

closes #70

Pull Request Test Coverage Report for Build 234

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 57.118%

Totals

Change from base Build 232:
0.0%

Covered Lines:
1212

Relevant Lines:
1868

💛  - Coveralls

Thanks for the PR! We just need to make sure that this change is backward compatible. ACKEE_PORT should be used first, PORT as a fallback and 3000 when non of them is defined.

@electerious Great idea! Forgot about this.

@electerious should be done now 👍


change const -> var

replaces https://github.com/thejameskyle/react-required-if/pull/2.

Please don't bump the version in a PR

@thejameskyle un-bumped

published

Awesome thanks!


Max Slave

Hello is 10 the max of slave or can i use 50 slave in the define

A higher HASHRATE can cause individual items to crash. I had a set of 12 parts, after 2-3 hours it worked 9. I made two sets of 6 and it works stably


Black camera on first open

Steps to Reproduce
I could only reproduce this on release mode, not profile or debug. On a Samsung Galaxy A21s, the first time I open the camera (with the CameraAwesome widget), there's a black screen with the following error message:
I/flutter (15729): Camerawesome start after resume state failed: PlatformException(CAMERA_MUST_BE_INIT, init must be call before start, , null)

If I close the route and open it again, the initialization error no longer happens, seems to be only the first time that it's opened.
Expected results
Camera initializes and viewfinder is displayed
Actual results
There was an error initializing the camera
About your device

Brand
Model
OS

Samsung
Galaxy A21s
Android 12, OneUI 4.1

Flutter Doctor
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 3.3.3, on Microsoft Windows [Versi¢n 10.0.22621.521], locale es-ES)
[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.0)
[✓] Chrome - develop for the web
[✓] Android Studio (version 2021.3)
[✓] Connected device (3 available)
[✓] HTTP Host Availability

https://github.com/Apparence-io/camera_awesome/blob/4539779d41466bd605f7dc682c018cd7314c77f4/lib/camerapreview.dart#L180

This should be fixed with the reworked version #150


Stryker Equality ignore  once not working

Stryker Version:
"@stryker-mutator/core": "^6.2.3",
"@stryker-mutator/karma-runner": "^6.2.3",
"@stryker-mutator/typescript-checker": "^6.1.2",

// Stryker disable all followed by // Stryker restore all works

Even without the once keyword it still won't work

The name of the mutators is EqualityOperator.
Let's warn users when a mutator name doesn't match any of the enabled mutators.

I've been working on a PoC for this feature and I could use some feedback.
My current implementation shows the following:

Which in it's turn reflects in the HTML:

A couple of questions:

Is this solution what was expected of this feature, or did I misinterpret it?
I wanted to introduce a new Status along the lines of Unknown, but this requires a change to the Schema which is probably used by more stryker projects? Not sure if I can just add something there without discussing it more broadly. Or is it acceptable to use Ignored for this?
I'm also not happy with the diff view, but it seems like these are required with my current implementation (I'm currently (ab)using the Mutant class to get it to display in the clear-text-reporter). It doesn't really highlight the real error. It would be cool if I could create a diff where the Unknown Operator is highlighted and perhaps a close match can be suggested. Might be tricky though.

Thoughts?

I've been working on a PoC for this feature

😍

A couple of questions:

Is this solution what was expected of this feature, or did I misinterpret it?

Well, having it end up in the report itself is nice, but a lot of work. I was thinking more along the lines of a warning message being logged.

I wanted to introduce a new Status along the lines of Unknown, but this requires a change to the Schema which is probably used by more stryker projects? Not sure if I can just add something there without discussing it more broadly. Or is it acceptable to use Ignored for this?

Yeah, introducing a new status is a lot of work and is technically a breaking change. We should be hesitant to add new.

I'm also not happy with the diff view, but it seems like these are required with my current implementation (I'm currently (ab)using the Mutant class to get it to display in the clear-text-reporter). It doesn't really highlight the real error. It would be cool if I could create a diff where the Unknown Operator is highlighted and perhaps a close match can be suggested. Might be tricky though.

Wow yes, definitely. Like the HTML reporter does? Feel free to open an issue for that 😍. You might even be able to share code between mutation-testing-elements (or mutation-testing-metrics) and Stryker, but at least you will be able to copy the code: https://github.com/stryker-mutator/mutation-testing-elements/blob/8702f65404bb05aa90d5b9f3d134312d91cd4cd9/packages/elements/src/lib/code-helpers.ts#L320

I'll stash the changes for adding them to the report itself, though it does not differ to much from logging it as a warning. It works, but perhaps I made it to difficult so I would love some feedback on it. I'll open the PR for it.

"Unused 'Stryker disable' directive. Mutator with name 'Equality' was not found. Directive found at: src/foo:23:1"

Sounds good to me.

Wow yes, definitely. Like the HTML reporter does? Feel free to open an issue for that 😍. You might even be able to share code between mutation-testing-elements (or mutation-testing-metrics) and Stryker, but at least you will be able to copy the code: https://github.com/stryker-mutator/mutation-testing-elements/blob/8702f65404bb05aa90d5b9f3d134312d91cd4cd9/packages/elements/src/lib/code-helpers.ts#L320

👍 I'll write an issue for it. Not sure what exactly you mean with like the HTML reporter does? I assume that you get a clickable preview of the mutation but in this case a suggestion with a close match?


Generate neuroglancer config for all tomograms

Generate neuroglancer configs for all tomograms, even if they're not default. This will help users with being able to visualize all the tomograms before downloading them.
For s3 import

Remove check in neuroglancer config generation.  (https://github.com/chanzuckerberg/cryoet-data-portal-backend/blob/main/ingestion_tools/scripts/importers/visualization_config.py#L37)
Update how neuroglancer path is added to the tomogram metadata (https://github.com/chanzuckerberg/cryoet-data-portal-backend/blob/2bad245f8ca41f5950de74a35401252ef0592830/ingestion_tools/scripts/importers/tomogram.py#L142)

For v1 api db import
In the db_import, set the neuroglancer_config to None for tomograms that have is_visualization_default= false.
This is required because the v1 api does not support is_visualization_default field that is used by frontend to identify the tomogram to be displayed in the single dataset page.
For v2 api db import
No changes are required for db_import.

HOLD for now: Jun to surface to design first


Add Support for AWS Launch Template Configuration

Reference Issues or PRs
This is an extension version of the fantastic work suggested and developed by @joneszc on his original PR #2621
The significant difference from his PR is moving out the template and AMI-type constructs into its pydantic schema (also to allow users to customize the template further). This also reduces the logic conditionals with terraform HCL as well.

What does this implement/fix?
Put a x in the boxes that apply

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds a feature)
[ ] Breaking change (fix or feature that would cause existing features not to work as expected)
[ ] Documentation Update
[ ] Code style update (formatting, renaming)
[ ] Refactoring (no functional changes, no API changes)
[ ] Build related changes
[ ] Other (please describe):

Testing

[ ] Did you test the pull request locally?
[ ] Did you add new tests?

How to test this PR?

Any other comments?

@viniciusdc How can we help getting this PR prioritized for review & merge? We need this capability in order to make progress on our air-gapped deployment investigation.

@joneszc can you try out these changes and check if the MIME user_data file generated workers as expected? I will do a deployment this evening as well.

How to test
Those are valid configs that should work:
    test:
      instance: m5.xlarge
      min_nodes: 0
      max_nodes: 1
      gpu: false
      single_subnet: false
      permissions_boundary:
      launch_template:
        node_prebootstrap_command: |
          #!/bin/bash
          echo "Hello, Nebari!"
    test2:
      instance: m5.xlarge
      min_nodes: 0
      max_nodes: 1
      gpu: false
      single_subnet: false
      permissions_boundary:
      ami_type: CUSTOM
      launch_template:
        ami_id: ami-0c3f3b5f2f3f3f3f3
        node_prebootstrap_command: |
          #!/bin/bash
          echo "Hello, Nebari!"

launch_template can also be defined at the provider level and will affect all node_groups, though for ami_id to be passed the ami_type needs to be set to CUSTOM

      ami_type: CUSTOM
      launch_template:
        ami_id: ami-0c3f3b5f2f3f3f3f3
        node_prebootstrap_command: |
          #!/bin/bash
          echo "Hello, Nebari!"

@viniciusdc
Initial test resulted in the following error

@joneszc can you try out these changes and check if the MIME user_data file generated workers as expected? I will do a deployment this evening as well.

Hello @viniciusdc
My question with this ami_type check is whether a CUSTOM gpu-enabled AMI will be possible if a manually entered "CUSTOM" value is constantly overridden by the base AL2_x86_64_GPU ami_type.
I personally think the original Nebari approach of inferring the ami_type based on condition of gpu true/false would be best combined with an additional switch to auto-trigger a value of CUSTOM if the ami_id is specified by the user.
Also, the options to set ami_id & pre_bootstrap_command should be independent and not mutually inclusive.  Will this validator here prevent the user from running a custom AMI unless including a pre_bootstrap_command?  We have use cases for employing custom-ami without running commands, custom-ami with running commands, as well as running commands without custom-ami.

@joneszc @tylergraff I am testing this right now:

My question with this ami_type check is whether a CUSTOM gpu-enabled AMI will be possible if a manually entered "CUSTOM" value is constantly overridden by the base AL2_x86_64_GPU ami_type.
I personally think the original Nebari approach of inferring the ami_type based on condition of gpu true/false would be best combined with an additional switch to auto-trigger a value of CUSTOM if the ami_id is specified by the user.

You are right; that's what I had in mind when first including that validator, mainly to avoid the case where the user misplaced something in yaml. But I agree with you that the current code does more harm to extensibility than anything. I have a fix for that already and will commit shortly.

Also, the options to set ami_id & pre_bootstrap_command should be independent and not mutually inclusive. Will this validator here prevent the user from running a custom AMI unless including a pre_bootstrap_command? We have use cases for employing custom-ami without running commands, custom-ami with running commands, as well as running commands without custom-ami.

I agree that was hindsight on my part, I will address it shortly. The only thing I am curious about is this last part here:

as well as running commands without custom-ami.

I am not sure that will be possible when launching an instance using a launch_template while the ami_id field is optional. During the spawning request itself, it will prompt for one, but I am not entirely sure yet if it will be able to auto-assign one or if it will fail.

while testing found a minor bug with some validations of an internal _nebari method. Currently addressing it
│ /Nebari/nebari/src/_nebari/stages/terraform_state/__init__.py:2 │
│ 39 in deploy                                                                                │
│                                                                                │
│   236 │   def deploy(                                                                            │
│   237 │   │   self, stage_outputs: Dict[str, Dict[str, Any]], disable_prompt: bool = False       │
│   238 │   ):                                                                                │
│ ❱ 239 │   │   self.check_immutable_fields()                                                      │
│   240 │   │                                                                                │
│   241 │   │   with super().deploy(stage_outputs, disable_prompt):                                │
│   242 │   │   │   env_mapping = {}                                                               │
│                                                                                │
│ /Nebari/nebari/src/_nebari/stages/terraform_state/__init__.py:2 │
│ 75 in check_immutable_fields                                                                     │
│                                                                                │
│   272 │   │   │   bottom_level_schema = self.config                                              │
│   273 │   │   │   if len(keys) > 1:                                                              │
│   274 │   │   │   │   print(keys)                                                                │
│ ❱ 275 │   │   │   │   bottom_level_schema = functools.reduce(                                    │
│   276 │   │   │   │   │   lambda m, k: getattr(m, k), keys[:-1], self.config                     │
│   277 │   │   │   │   )                                                                          │
│   278 │   │   │   extra_field_schema = schema.ExtraFieldSchema(                                  │
│                                                                                │
│ /Nebari/nebari/src/_nebari/stages/terraform_state/__init__.py:2 │
│ 76 in <lambda>                                                                                │
│                                                                                │
│   273 │   │   │   if len(keys) > 1:                                                              │
│   274 │   │   │   │   print(keys)                                                                │
│   275 │   │   │   │   bottom_level_schema = functools.reduce(                                    │
│ ❱ 276 │   │   │   │   │   lambda m, k: getattr(m, k), keys[:-1], self.config                     │
│   277 │   │   │   │   )                                                                          │
│   278 │   │   │   extra_field_schema = schema.ExtraFieldSchema(                                  │
│   279 │   │   │   │   **bottom_level_schema.model_fields[keys[-1]].json_schema_extra or {}       │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'dict' object has no attribute 'test'

Hi @tylergraff @joneszc while reviewing and testing this proposed changes (using launch_templates) I found the following very important information on AWS docs:

When using a custom AMI (ami), overrideBootstrapCommand must also be set to perform the bootstrapping.
overrideBootstrapCommand can only be set when using a custom AMI.
When a launch template is provided, tags specified in the nodegroup config apply to the EKS Nodegroup resource only and are not propagated to EC2 instances.

A node group can enter a DEGRADED status with a message similar to the following error:

"The Amazon EC2 Launch Template : lt-xxxxxxxxxxxxxxxxx has a new version associated with your Autoscaling group, which is not managed by Amazon EKS. Expected Launch Template version: x".

This error occurs when the Amazon EC2 launch template version for your managed node group doesn't match the version that Amazon EKS creates. Existing node groups that don't use a custom launch template can't be directly updated. To resolve this error, create a launch template and version with your preferred settings. Then, use the launch template to create the node group. If the new node group is launched from your custom template, then create new versions of the template. You can use this template without placing the node group in a DEGRADED status.

and also this one:

Managed node groups are always deployed with a launch template to be used with the Amazon EC2 Auto Scaling group. When you don't provide a launch template, the Amazon EKS API creates one automatically with default values in your account. However, we don't recommend that you modify auto-generated launch templates. Furthermore, existing node groups that don't use a custom launch template can't be updated directly. Instead, you must create a new node group with a custom launch template to do so.

This is not a big problem as we now have ways to control this so that it never happens, but I just wanted to signal it as in our docs, we will be raising this launch_template parameter as an immutable one. This means that only new deployments or new node_groups will be able to benefit from it.
Also, when using launch_templates, upgrades to kubernetes versions and AMIs are not performed automatically anymore on the UI, so usage of the nebari deployments to upgrade the available kuberntes versions will be required.

Looking great!
[terraform]:   # module.kubernetes.aws_eks_node_group.main[3] must be replaced
[terraform]: -/+ resource "aws_eks_node_group" "main" {
[terraform]:       ~ ami_type               = "AL2_x86_64" -> (known after apply)
[terraform]:       ~ arn                    = "arn:aws:eks:us-east-1:******:nodegroup/vini-template-dev/test/----" -> (known after apply)
[terraform]:       ~ capacity_type          = "ON_DEMAND" -> (known after apply)
[terraform]:       ~ disk_size              = 0 -> 50 # forces replacement
[terraform]:       ~ id                     = "vini-template-dev:test" -> (known after apply)
[terraform]:       ~ instance_types         = [ # forces replacement
[terraform]:           + "t2.micro",
[terraform]:         ]
[terraform]:       + node_group_name_prefix = (known after apply)
[terraform]:       ~ release_version        = "1.29.6-20240910" -> (known after apply)
[terraform]:       ~ resources              = [
[terraform]:           - {
[terraform]:               - autoscaling_groups              = [
[terraform]:                   - {
[terraform]:                       - name = "eks-test-----"
[terraform]:                     },
[terraform]:                 ]
[terraform]:               - remote_access_security_group_id = ""
[terraform]:             },
[terraform]:         ] -> (known after apply)
[terraform]:       ~ status                 = "ACTIVE" -> (known after apply)
[terraform]:         tags                   = {
[terraform]:             "Environment"                                             = "dev"
[terraform]:             "Owner"                                                   = "terraform"
[terraform]:             "Project"                                                 = "vini-template"
[terraform]:             "k8s.io/cluster-autoscaler/node-template/label/dedicated" = "test"
[terraform]:             "propagate_at_launch"                                     = "true"
[terraform]:         }
[terraform]:       ~ version                = "1.29" -> (known after apply)
[terraform]:         # (6 unchanged attributes hidden)
[terraform]:
[terraform]:       - launch_template {
[terraform]:           - id      = "lt-02c4517df4f8e2bbc" -> null
[terraform]:           - name    = "test" -> null
[terraform]:           - version = "1" -> null
[terraform]:         }
[terraform]:
[terraform]:       - update_config {
[terraform]:           - max_unavailable            = 1 -> null
[terraform]:           - max_unavailable_percentage = 0 -> null
[terraform]:         }
[terraform]:
[terraform]:         # (1 unchanged block hidden)
[terraform]:     }
[terraform]:
[terraform]:   # module.kubernetes.aws_eks_node_group.main[4] will be created
[terraform]:   + resource "aws_eks_node_group" "main" {
[terraform]:       + ami_type               = (known after apply)
[terraform]:       + arn                    = (known after apply)
[terraform]:       + capacity_type          = (known after apply)
[terraform]:       + cluster_name           = "vini-template-dev"
[terraform]:       + disk_size              = 50
[terraform]:       + id                     = (known after apply)
[terraform]:       + instance_types         = [
[terraform]:           + "t2.micro",
[terraform]:         ]
[terraform]:       + labels                 = {
[terraform]:           + "dedicated" = "test1"
[terraform]:         }
[terraform]:       + node_group_name        = "test1"
[terraform]:       + node_group_name_prefix = (known after apply)
[terraform]:       + node_role_arn          = "arn:aws:iam::******:role/vini-template-dev-eks-node-group-role"
[terraform]:       + release_version        = (known after apply)
[terraform]:       + resources              = (known after apply)
[terraform]:       + status                 = (known after apply)
[terraform]:       + subnet_ids             = [
[terraform]:           + "subnet-******",
[terraform]:           + "subnet-******",
[terraform]:         ]
[terraform]:       + tags                   = {
[terraform]:           + "Environment"                                             = "dev"
[terraform]:           + "Owner"                                                   = "terraform"
[terraform]:           + "Project"                                                 = "vini-template"
[terraform]:           + "k8s.io/cluster-autoscaler/node-template/label/dedicated" = "test1"
[terraform]:           + "propagate_at_launch"                                     = "true"
[terraform]:         }
[terraform]:       + tags_all               = {
[terraform]:           + "Environment"                                             = "dev"
[terraform]:           + "Owner"                                                   = "terraform"
[terraform]:           + "Project"                                                 = "vini-template"
[terraform]:           + "k8s.io/cluster-autoscaler/node-template/label/dedicated" = "test1"
[terraform]:           + "propagate_at_launch"                                     = "true"
[terraform]:         }
[terraform]:       + version                = (known after apply)
[terraform]:
[terraform]:       + scaling_config {
[terraform]:           + desired_size = 0
[terraform]:           + max_size     = 1
[terraform]:           + min_size     = 0
[terraform]:         }
[terraform]:     }
[terraform]:
[terraform]:   # module.kubernetes.aws_launch_template.main["test"] will be destroyed
[terraform]:   # (because key ["test"] is not in for_each map)
[terraform]:   - resource "aws_launch_template" "main" {
[terraform]:       - arn                     = "arn:aws:ec2:us-east-******:launch-template/lt-02c4517df4f8e2bbc" -> null
[terraform]:       - default_version         = 1 -> null
[terraform]:       - disable_api_stop        = false -> null
[terraform]:       - disable_api_termination = false -> null
[terraform]:       - id                      = "lt-02c4517df4f8e2bbc" -> null
[terraform]:       - instance_type           = "t2.micro" -> null
[terraform]:       - latest_version          = 1 -> null
[terraform]:       - name                    = "test" -> null
[terraform]:       - security_group_names    = [] -> null
[terraform]:       - tags                    = {} -> null
[terraform]:       - tags_all                = {} -> null
[terraform]:       - user_data               = "TUlNRS1W***FsbCAteSBodG9wCgoKCgotLT09TVlCT1VOREFSWT09LS0K" -> null
[terraform]:       - vpc_security_group_ids  = [
[terraform]:           - "sg-******",
[terraform]:         ] -> null
[terraform]:
[terraform]:       - block_device_mappings {
[terraform]:           - device_name = "/dev/xvda" -> null
[terraform]:
[terraform]:           - ebs {
[terraform]:               - iops        = 0 -> null
[terraform]:               - throughput  = 0 -> null
[terraform]:               - volume_size = 50 -> null
[terraform]:               - volume_type = "gp2" -> null
[terraform]:             }
[terraform]:         }
[terraform]:
[terraform]:       - metadata_options {
[terraform]:           - http_endpoint               = "enabled" -> null
[terraform]:           - http_put_response_hop_limit = 0 -> null
[terraform]:           - http_tokens                 = "required" -> null
[terraform]:           - instance_metadata_tags      = "enabled" -> null
[terraform]:         }
[terraform]:     }
[terraform]:

running with the example bellow:
    test:
      instance: t2.micro
      min_nodes: 1
      max_nodes: 1
      gpu: false
      single_subnet: false
      launch_template:
        pre_bootstrap_command: "#!/bin/bash\n# This script is executed before the node is bootstrapped\n# You can use this script to install additional packages or configure the node\n# For example, to install the `htop` package, you can run:\n# sudo apt-get update\n# sudo apt-get install -y htop"
      permissions_boundary:

ami_id is not required; terraform resource handles that accordingly when applying the node_group. I haven't tested the custom ami_id yet (I could attest it switches to CUSTOM, though), as I don't have a custom image.

ami_id is not required; terraform resource handles that accordingly when applying the node_group. I haven't tested the custom ami_id yet (I could attest it switches to CUSTOM, though), as I don't have a custom image.

Hello @viniciusdc
I just tested the custom ami_id option and the custom-ami nodegroups failed to join the cluster...

It appears you changed the format of the env variable statements set in the bootstrap.sh command.
I had specifically instantiated those variables formatted as ${<variable>} to avoid the following error:
Cloud-init v. 19.3-46.amzn2.0.2 running 'modules:final' at Tue, 17 Sep 2024 19:34:47 +0000. Up 18.47 seconds.
+ CLUSTER_NAME='{{ cluster_name }}'
+ B64_CLUSTER_CA='{{ cluster_cert_authority }}'
+ API_SERVER_URL='{{ cluster_endpoint }}'
+ /etc/eks/bootstrap.sh '{{' cluster_name '}}' --b64-cluster-ca '{{' cluster_cert_authority '}}' --apiserver-endpoint '{{' cluster_endpoint '}}'
2024-09-17T19:34:56+0000 [eks-bootstrap] INFO: starting...
2024-09-17T19:34:56+0000 [eks-bootstrap] INFO: --b64-cluster-ca='{{'
2024-09-17T19:34:56+0000 [eks-bootstrap] INFO: --apiserver-endpoint='{{'
2024-09-17T19:35:04+0000 [eks-bootstrap] INFO: Using kubelet version 1.29.6
2024-09-17T19:35:04+0000 [eks-bootstrap] INFO: Using containerd as the container runtime
Created symlink from /etc/systemd/system/multi-user.target.wants/sys-fs-bpf.mount to /etc/systemd/system/sys-fs-bpf.mount.
ΓÇÿ/etc/eks/configure-clocksource.serviceΓÇÖ -> ΓÇÿ/etc/systemd/system/configure-clocksource.serviceΓÇÖ
Created symlink from /etc/systemd/system/multi-user.target.wants/configure-clocksource.service to /etc/systemd/system/configure-clocksource.service.
2024-09-17T19:35:04+0000 [eks-bootstrap] INFO: Using IP family: ipv4
base64: invalid input
Exited with error on line 415
Sep 17 19:35:04 cloud-init[3100]: util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-002 [1]

As you can see above, the env variables don't render on the node when enclosed in {{'  '}}
Additionally, when invoking a custom ami_id, but not employing any pre_bootstrap_command, an empty user_data section is populated. However, this empty user_data part does not seem to prevent nodes from joining the cluster:

Finally, the following conditional, checking whether or not ami_type is set to "CUSTOM", does not appear to be functioning:

The bootstrap.sh command fails to render and the node fails to join the cluster.
I was able to continue testing by, instead, using the following conditional with --disable-render:
        include_bootstrap_cmd     = each.value.launch_template.ami_id != null ? true : false

Hi @joneszc thank you so much for the testing, indeed while moving working around that template I ended up incorrectly assigning the values there, thanks for catching that. I fixed it this afternoon while testing the CUSTOM logic issue you related above. I had also noticed the blank user_data sections and fixed it. Will finish testing and commit the new changes

I am addressing an issue with the ami_id and launch_template.ami_profile_name that might be related to the faulty logic you saw with the CUSTOM. After you continued testing with the no render parameter, were you able to get your instance running with the custom code?

I am addressing an issue with the ami_id and launch_template.ami_profile_name that might be related to the faulty logic you saw with the CUSTOM. After you continued testing with the no render parameter, were you able to get your instance running with the custom code?

Hello @viniciusdc
Yes! testing both pre_bootstrap_command and setting a custom ami_id succeeded after using --disable-render while adjusting the include_bootstrap_cmd conditional as mentioned and also the user_data.tftpl environment vars format:
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="

--==MYBOUNDARY==
Content-Type: text/x-shellscript; charset="us-ascii"

%{ if node_prebootstrap_command != null }
${node_prebootstrap_command}
%{ endif }

%{ if include_bootstrap_cmd == true }
--==MYBOUNDARY==
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
set -ex

CLUSTER_NAME=${cluster_name}
B64_CLUSTER_CA=${cluster_cert_authority}
API_SERVER_URL=${cluster_endpoint}

/etc/eks/bootstrap.sh $CLUSTER_NAME --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL
%{ endif }

--==MYBOUNDARY==--

Testing this is now complete, I ensured it was backward compatible with an existing deployment:
Deployment with custom IAM (testing cluster)

Attested pre_bootstrap_command works

For more details here's the doc's PR https://github.com/nebari-dev/nebari-docs/pull/525

Its worth noticing that the AWS provider seems to have an issue with the update of scaling_config.desired_size field. So, if you are using a custom AMI and you would like to scale up from the nebari config, right now you will need to recreate the group to make that work or use the console UI to scale (advised) that only affects deployments whose node_group has the launch_template and uses a custom ami_id.
I will address this inconsistency with a follow up PR in another time.


Build problem

Background
when i try compile with
GOOS=windows GOARCH=amd64 go build -v -o lnd.exe -ldflags "-X main.Commit=v0.6-beta" github.com/lightningnetwork/lnd
for windows
i hv this erroe
GOOS=windows GOARCH=amd64 go build -v -o lnd.exe -ldflags "-X main.Commit=v0.6-beta" github.com/lightningnetwork/lnd
github.com/lightningnetwork/lnd/routing/route
# github.com/lightningnetwork/lnd/routing/route
routing/route/route.go:119:4: cannot use 0 (type int) as type [1]byte in field value

Steps to reproduce
at ubuntu 18.04
first i do
sudo apt install go-dep
then
wget https://dl.google.com/go/go1.12.3.linux-amd64.tar.gz
then
sudo tar -C /usr/local -xzf go1.12.3.linux-amd64.tar.gz
then
export PATH=$PATH:/usr/local/go/bin
then
go get -d  github.com/lightningnetwork/lnd
then
cd /home/ubuntu/go/src/github.com/lightningnetwork/lnd
then
dep init
then
dep ensure -v
then
GOOS=windows GOARCH=amd64 go build -v -o lnd.exe -ldflags "-X main.Commit=v0.6-beta" github.com/lightningnetwork/lnd
and give error
GOOS=windows GOARCH=amd64 go build -v -o lnd.exe -ldflags "-X main.Commit=v0.6-beta" github.com/lightningnetwork/lnd
github.com/lightningnetwork/lnd/routing/route
# github.com/lightningnetwork/lnd/routing/route
routing/route/route.go:119:4: cannot use 0 (type int) as type [1]byte in field value

@mraksoll4 do you get the same error with make?

The commit indicates you are trying to compile 0.6-beta, but the file referenced in the error did not exist in 0.6.

We also don't use dep anymore. You'll need to build with go modules: https://github.com/lightningnetwork/lnd/blob/master/docs/INSTALL.md#installing-lnd

problem was at download latest lightning-onion from master, not version from comint what was need.


Faster but still general binop broadcasting

Speed: (just on my machine, so take w/ grain of salt) 0.27s for this version. It was 0.4s. The version before we general broadcasting was 0.25s. Based on the complexity of generated code (and the fact that it has no branches), simple broadcasts should have a similar level of performance to the simple-broadcast-specific code that was there before.
Complexity: Costs about 6 new LOC in complexity.
What/How:
In binary_op, we get rid of the loop in the kernel, instead computing indices with nested expressions.
Prior to compiling the operation code, we combine any adjacent dimensions that we can, and ignore dimensions that are both 1. I broke out codegen and compilation into a new function, get_binop_prg, which has a larger cache and depends only on the binop code and the type of broadcast (this avoids doing a bunch of python string ops every time we want to compile something we've already compiled). The larger (600 vs. 128) cache is needed since there will end up being quite a few possible types of broacasts (sum_i^N 3**i is a loose upper bound, N being the maximum number of dimensions). Probably most programs will only use a fraction of this, though, and worst case is just having to recompile some things repeatedly.

Okay, looks like the problem is that len(dimlist) is sometimes greater than CL_DEVICE_MAX_WORK_ITEM_DIMENSIONS (which can be as low as 3). This'll take some messing around to fix.

Speed on my machine is still around 0.27s. (note: the command I'm running for all of these tests is python test/test_ops.py TestOpsGPU.test_broadcast_partial, I'm not running the full set of tests.)

Very clean!


SAK-32131 Fix issue with lowercase growing string.

Some characters get longer when they become lowercased and this was causing the portal code to incorrectly inline the tool. In severe cases this would cause an out of bounds exception.
In other cases the returned page would contain multiple closing body tags.
The refactor here makes it possible to write a unit test for the code without having to mock large parts of the request.

I slightly modified the debug statement. LGTM


fix(config): restrict config.lighthouseOptions.onlyCategories values and …

Description
When I was writing my config I added different categories to the lighthouseOptions.onlyCategories parameter like this:
{
    [...]
    lighthouseOptions: {
        onlyCategories: ['accessibility', 'best-practices', 'seo', 'performance']
    }
}

When adding them, I ignored their order and just appended new elements in the end.
When I generated a report I noticed that this order was adopted by the tab list on the report page.
So my Navigation looked like this:

Overview
Accessibility
Best Practices
SEO
Performance

But once I clicked on the navigation elements, they would open another tab.
I could click Accessibility but it would open the Performance view.
The same switch could be observed with the guage element next to them.
Accessibility displayed the average value of the Performance category.
Everything except the label would be related to the element, which would in the regular order be position in this spot.
To fix this issue, I added some code to order the user inputs in the same order as the default config.
Additionally this code filters out every none standard or mistyped values as they would also confuse the system.
Linked Issues
none
Additional context
Things to evaluate:

I'm not sure if restricting the selectable values to ['performance', 'accessibility', 'best-practices', 'seo'] can have any side effects on custom configuration. That's probably something you have more insights to determine.
An alternative solution would be to fix the behaviour of the TabList element and allow other orders. But I think I prefer this solution as it is uniform with with the lighthouse outputs.

Looks like a good solution for the time being, thanks!


Consider exposing the packet short_id as a label for relabling

I use hostnames like «short_id».packethost.net in my infrastructure, and it would be nice to be able to use them via packet-sd  as well.

fixed in https://github.com/packethost/prometheus-packet-sd/commit/3e8965ce7ef68501e5037c932221cf05d184350d


Boost fails to build

warning: Import libs were not present in V:/vcpkg/packages/bzip2_x86-windows/debug/lib
Found 4 error(s). Please correct the portfile:
V:\vcpkg\ports\bzip2\portfile.cmake
Error: Building package bzip2:x86-windows failed with: POST_BUILD_CHECKS_FAILED

Full log:
T:\>vcpkg install boost
-- CURRENT_INSTALLED_DIR=V:/vcpkg/installed/x86-windows
-- DOWNLOADS=V:/vcpkg/downloads
-- CURRENT_PACKAGES_DIR=V:/vcpkg/packages/bzip2_x86-windows
-- CURRENT_BUILDTREES_DIR=V:/vcpkg/buildtrees/bzip2
-- CURRENT_PORT_DIR=V:/vcpkg/ports/bzip2/.
-- Downloading http://www.bzip.org/1.0.6/bzip2-1.0.6.tar.gz...
-- Downloading http://www.bzip.org/1.0.6/bzip2-1.0.6.tar.gz... OK
-- Testing integrity of downloaded file...
-- Testing integrity of downloaded file... OK
-- Extracting source V:/vcpkg/downloads/bzip2-1.0.6.tar.gz
-- Extracting done
-- Applying patch V:/vcpkg/ports/bzip2/fix-import-export-macros.patch
-- Applying patch failed. This is expected if this patch was previously applied.
-- Applying patch V:/vcpkg/ports/bzip2/fix-import-export-macros.patch done
-- Configuring x86-windows-rel
-- Configuring x86-windows-rel done
-- Configuring x86-windows-dbg
-- Configuring x86-windows-dbg done
-- Package x86-windows-rel
-- Package x86-windows-rel done
-- Package x86-windows-dbg
-- Package x86-windows-dbg done
-- Applying patch V:/vcpkg/ports/bzip2/auto-define-import-macro.patch
-- Applying patch failed. This is expected if this patch was previously applied.
-- Applying patch V:/vcpkg/ports/bzip2/auto-define-import-macro.patch done
-- Performing post-build validation
Import libs were not present in V:/vcpkg/packages/bzip2_x86-windows/debug/lib
If this is intended, add the following line in the portfile:
    SET(VCPKG_POLICY_DLLS_WITHOUT_LIBS disabled)
Import libs were not present in V:/vcpkg/packages/bzip2_x86-windows/lib
If this is intended, add the following line in the portfile:
    SET(VCPKG_POLICY_DLLS_WITHOUT_LIBS disabled)
The following DLLs have no exports:

    V:/vcpkg/packages/bzip2_x86-windows/debug/bin/libbz2.dll
    V:/vcpkg/packages/bzip2_x86-windows/bin/libbz2.dll

DLLs without any exports are likely a bug in the build script.
There should be no empty directories in V:/vcpkg/packages/bzip2_x86-windows
The following empty directories were found:

    V:/vcpkg/packages/bzip2_x86-windows/debug/lib
    V:/vcpkg/packages/bzip2_x86-windows/lib

If a directory should be populated but is not, this might indicate an error in the portfile.
If the directories are not needed and their creation cannot be disabled, use something like this in the portfile to remove them:

    file(REMOVE_RECURSE ${CURRENT_PACKAGES_DIR}/a/dir ${CURRENT_PACKAGES_DIR}/some/other/dir)

-- Performing post-build validation done
Found 4 error(s). Please correct the portfile:
    V:\vcpkg\ports\bzip2\portfile.cmake
Error: Building package bzip2:x86-windows failed with: POST_BUILD_CHECKS_FAILED
Please ensure sure you're using the latest portfiles with `vcpkg update`, then
submit an issue at https://github.com/Microsoft/vcpkg/issues including:
  Package: bzip2:x86-windows
  Vcpkg version: 0.0.74-unknownhash

Additionally, attach any relevant sections from the log files above.

Hi @pravic,
It looks like bzip2 failed to build. Could you attach the log files:
V:/vcpkg/buildtrees/bzip2/config-x86-windows-rel-out.log
V:/vcpkg/buildtrees/bzip2/config-x86-windows-rel-err.log
V:/vcpkg/buildtrees/bzip2/package-x86-windows-rel-err.log
V:/vcpkg/buildtrees/bzip2/package-x86-windows-rel-out.log

...
-- Downloading http://www.bzip.org/1.0.6/bzip2-1.0.6.tar.gz...
...
-- Applying patch V:/vcpkg/ports/bzip2/fix-import-export-macros.patch
-- Applying patch failed...
...

This patch should have worked on freshly downloaded sources. Maybe there are some issues with your copy of git. These logs may also help:
V:/vcpkg/buildtrees/bzip2/patch-x64-windows-0-err.log
V:/vcpkg/buildtrees/bzip2/patch-x64-windows-0-out.log

bzip2 compilation ended without errors.
config-x86-windows-dbg-err:
CMake Warning:
  Manually-specified variables were not used by the project:

    Boost_COMPILER
    CMAKE_EXPORT_NO_PACKAGE_REGISTRY
    CMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY
    CMAKE_FIND_PACKAGE_NO_SYSTEM_PACKAGE_REGISTRY

config-x86-windows-dbg-out:
-- The C compiler identification is MSVC 19.0.24215.1
-- The CXX compiler identification is MSVC 19.0.24215.1
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/cl.exe -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/cl.exe
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/cl.exe -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: V:/vcpkg/buildtrees/bzip2/x86-windows-dbg

patch-x86-windows-0-err is empty.

libbz2,dll was compiled successfully. And yes, it contains no export, as my first log said (DLLs without any exports are likely a bug in the build script.).

I'd like to make sure that the patch [1] applied correctly to your sources. Could you run
>type buildtrees\bzip2\src\bzip2-1.0.6\bzlib.h | findstr "BZ_API"

from a command prompt in your Vcpkg directory?
If there is any mention of WINAPI in the output, the patch wasn't applied. In that case, please try deleting buildtrees\bzip2\ and doing a fresh install of bzip2.
If this fails, please post the patch logs (both -out AND -err) again, because they should explain why the patch didn't apply.
[1] https://github.com/Microsoft/vcpkg/blob/master/ports/bzip2/fix-import-export-macros.patch

Well, I found what's wrong.
vcpkg_apply_patches unable to find git program, because I use git.cmd (no idea how find_program works - in ideal world it should not care about program extension, it just should try to execute any of PATHEXT). And since noone checks find_program result, cmake tries to operate with GIT-NOTFOUND program to apply patches.

There is a bug about that: https://cmake.org/Bug/bug_relationship_graph.php?bug_id=9879.
It seems, it is not fixed still: https://gitlab.kitware.com/cmake/cmake/issues/9879

Resolved by #698, works well now. Thanks for help!


How to 360 rotation gauge.

Hi,
Not Really an ng-gauge question, but maybe someone knows here. How do you make a smooth full circle SVG amimation transition? If you amimate from say 320 degrees to 10 degrees, it will not go over 0, but rather Go the other way around. How is this done? I see it in action with no h
Gauge!

same for me will be helpful

I found how. You have to use some math. See if degree change is more than 180, if so, check if what way you go over the 0 or 360, then split the animation:
Say from 350 to 10 degrees. Total degree change is 350 - 10 = greater ten 180. Animate 350 to 359. Them 0 to 10.
It’s the only way I found. Works well !

// Special cases to smooth out passing between 359 to/from 0
// if more than half the circle, it could need to go over the 359 / 0 values
if ( Math.abs(diff) > 180 ) {
// In what direction are we moving?
if (Math.sign(diff) == 1) {
if (oldAngle == 359) {
// special cases
this.oldAppWindAngle = "0";
this.appWindAnimate.nativeElement.beginElement();
} else {
this.newAppWindAngle = "359";
this.appWindAnimate.nativeElement.beginElement();
this.oldAppWindAngle = "0";
this.newAppWindAngle = changes.appWindAngle.currentValue.toFixed(0);
this.appWindAnimate.nativeElement.beginElement();
}
} else {
if (oldAngle == 0) {
// special cases
this.oldAppWindAngle = "359";
this.appWindAnimate.nativeElement.beginElement();
} else {
this.newAppWindAngle = "0";
this.appWindAnimate.nativeElement.beginElement();
this.oldAppWindAngle = "359";
this.newAppWindAngle = changes.appWindAngle.currentValue.toFixed(0);
this.appWindAnimate.nativeElement.beginElement();
}
}
} else {
this.appWindAnimate.nativeElement.beginElement();
}

您的邮件我已收到，谢谢！


Add support for Intercom events

Reference: Track events in Intercom

Oops! I forgot I already did this in #2.

Reference: Track events in Intercom

<EMAIL_ADDRESS>

#11


Translators missing on the citation page

I noticed that on the citation page, the references and the BibTeX do not include the translators. I feel you should add these! This might be as simple as adding translators in the appropriate places, as I've done below, but it probably makes sense to translate the titles in their entirety.

Wadler, Philip and Wen Kokke. Programming Language Foundations in Agda. Translated by Fangyi Zhou, Oling Cat, and Rongxiao Fu. Available at https://agda-zh.github.io/PLFA-zh/. 2019.

@Book{plfa2019,
  author     = {Philip Wadler and Wen Kokke},
  title      = {Programming Language Foundations in {A}gda},
  translator = {Fangyi Zhou and Oling Cat and and Rongxiao Fu},
  note       = {Available at \url{https://agda-zh.github.io/PLFA-zh/}},
  year       = 2019,
}

Ah... thank you! I just noticed that.
This issue will be solved after #64, cause we are sync PLFA-zh with plfa.github.io for Agda 2.6.0, it has a lot of work. 😅
In Chinese, the cite of PLFA-zh should be

《编程语言基础：Agda 语言描述》，Philip Wadler 与 Wen Kokke 著，Fangyi Zhou、Oling Cat 与 Rongxiao Fu 译。可访问 https://agda-zh.github.io/PLFA-zh/ 阅读。

By the way, your name "Wen Kokke" can be transliterated into "温·考克" in Chinese, and "Philip Wadler" should be "飞利浦·瓦德勒".
@roger-uw @fangyi-zhou 话说咱还不知道乃们的中文名字= =？

周方易

傅荣枭

From: Oling Cat<EMAIL_ADDRESS>Sent: Tuesday, July 16, 2019 3:12:32 PM
To: Agda-zh/PLFA-zh
Cc: roger-uw; Mention
Subject: Re: [Agda-zh/PLFA-zh] Translators missing on the citation page (#65)
Ah... thank you! I just noticed that.
This issue will be solved after #64https://github.com/Agda-zh/PLFA-zh/issues/64, cause we are sync PLFA-zh with plfa.github.io for Agda 2.6.0, it has a lot of work. 😅
In Chinese, the cite of PLFA-zh should be
《编程语言基础：Agda 语言描述》，Philip Wadler 与 Wen Kokke 著，Fangyi Zhou、Oling Cat 与 Rongxiao Fu 译。可访问 https://agda-zh.github.io/PLFA-zh/ 阅读。
By the way, your name "Wen Kokke" can be transliterated into "温·考克" in Chinese, and "Philip Wadler" should be "飞利浦·瓦德勒".
@roger-uwhttps://github.com/roger-uw @fangyi-zhouhttps://github.com/fangyi-zhou 话说咱还不知道乃们的中文名字= =？
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/Agda-zh/PLFA-zh/issues/65?email_source=notifications&email_token=ADD2HJIGN2AOKVYYB7MP4E3P7VYGBA5CNFSM4ID3SK3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ75WJI#issuecomment-511695653, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ADD2HJLN4QMPQHMPH57W4O3P7VYGBANCNFSM4ID3SK3A.

This issue will be solved after #64, cause we are sync PLFA-zh with plfa.github.io for Agda 2.6.0, it has a lot of work. 😅

Ah, yeah, I'm sorry for that 😅


Alexey/2802 get rid of termlike transforming

Fixes #2802
Scope:
Estimate:

Review checklist
The author performs the actions on the checklist. The reviewer evaluates the work and checks the boxes as they are completed.

[ ] Summary. Write a summary of the changes. Explain what you did to fix the issue, and why you did it. Present the changes in a logical order. Instead of writing a summary in the pull request, you may push a clean Git history.
[ ] Documentation. Write documentation for new functions. Update documentation for functions that changed, or complete documentation where it is missing.
[ ] Tests. Write unit tests for every change. Write the unit tests that were missing before the changes. Include any examples from the reported issue as integration tests.
[ ] Clean up. The changes are already clean. Clean up anything near the changes that you noticed while working. This does not mean only spatially near the changes, but logically near: any code that interacts with the changes!

Closing this PR by the same reason the corresponding task was closed: #2802


Native cloud providers with GCP and AWS python lib

The goal is to connect to a user provided VPC from the stitching point.

[x] should experiment with AWS and GCP python client. Try use the python code (with some test cases) to attach an existing AWS VPC to a DirectConnect service or GCP VPC to a (creating) VlanAttachment.
[x] will be adding such native AWS/GCP provider to fabfed with @abessiari 's help.

@zlion (July 24-Aug 11) experiment with AWS and GCP python client. Try use the python code (with some test cases) to attach an existing AWS VPC to a DirectConnect service or GCP VPC to a (creating) VlanAttachment.
Next step (late Aug) will be adding such native AWS/GCP provider to fabfed with @abessiari 's help.

Sure, I got it.
Sent from my iPhone

On Jul 20, 2023, at 8:55 AM, xi-yang @.***> wrote:

@zlion (July 24-Aug 11) experiment with AWS and GCP python client. Try use the python code (with some test cases) to attach an existing AWS VPC to a DirectConnect service or GCP VPC to a (creating) VlanAttachment.
Next step (late Aug) will be adding such native AWS/GCP provider to fabfed with @abessiari 's help.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.

@zlion
For aws we may want to look at AWS SDK for Python (Boto3).
https://boto3.amazonaws.com/v1/documentation/api/latest/index.html
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/directconnect.html

Initial versions of python scripts that add given VPC to direct connect(AWS) and interconnect(GCP) have been checked into branch "develop-lzhang9" and  PR#65 has been created.

Verified that the GCP provider creates and destroys the network resource as expected. There is one thing I missed in the sample code that the "pairing key" should be read out and passed to Fabric provider. Good job @abessiari .

Yes I will have that part ready for you sometime tomorrow. Thanks.

There are some issues when creating a slice in the RENCI site. It could result from bugs in fablib according to Komal preliminary investigation.

Failed to create Al2s network with the Notebook testing script "create_al2s.ipynb"

Failed to print the facilityport,

Exception: 'FacilityPort' object has no attribute 'get_node'
Traceback (most recent call last):
  File "/tmp/ipykernel_152/2316175781.py", line 52, in <module>
    print(f"facility_port_iface {fabric_facility_port_iface}")
  File "/opt/conda/lib/python3.10/site-packages/fabrictestbed_extensions/fablib/interface.py", line 82, in __str__
    if self.get_network():
  File "/opt/conda/lib/python3.10/site-packages/fabrictestbed_extensions/fablib/interface.py", line 678, in get_network
    for net in self.get_slice().get_networks():
  File "/opt/conda/lib/python3.10/site-packages/fabrictestbed_extensions/fablib/interface.py", line 649, in get_slice
    return self.get_node().get_slice()
  File "/opt/conda/lib/python3.10/site-packages/fabrictestbed_extensions/fablib/interface.py", line 658, in get_node
    return self.get_component().get_node()
AttributeError: 'FacilityPort' object has no attribute 'get_node'

I came across an issue when submitting a slicet: "Couldnt create VRF: Couldnt create Endpoint: Couldnt find an available Unit." as shown below. Already opened a ticket to Internet2 to start the investigation.
failed lease update- all units failed priming: Exception during create for unit: ef3a16c9-be74-4032-864e-45eab3a72045 (PlaybookException(Playbook has failed tasks: results: None, error_text: Couldnt create VRF: Couldnt create Endpoint: Couldnt find an available Unit., error: 1), [(core1.loui.net.internet2.edu, HundredGigE0/0/0/24), (agg4.ashb.net.internet2.edu, Bundle-Ether5)])#all units failed priming: Exception during create for unit: ef3a16c9-be74-4032-864e-45eab3a72045 (PlaybookException(Playbook has failed tasks: results: None, error_text: Couldnt create VRF: Couldnt create Endpoint: Couldnt find an available Unit., error: 1), [(core1.loui.net.internet2.edu, HundredGigE0/0/0/24), (agg4.ashb.net.internet2.edu, Bundle-Ether5)])#
There are two interface resources to be allocated: "core1.loui" and "agg4.ashb". The latter one can be controlled via input parameters to the method facility_port(). But the former one is automatically/dynamically  picked up by the process between Fabric and Internet2.

@abessiari @xi-yang
Investigated the above issue with the AMHandler/test_oess_handler.py and found

the "oess_handler.py" works as expected.
the problem is related to the "VLAN tag", i.e. changing VLAN tag fixes the problem.

Further thoughts:
As the method "add_facility_port()" automatically pick up the tag from an internal pool, take into consideration,

how to pick up a valid VLAN tag from the pool?
need to retry with different tag if previous one fails with errors "Could not find an available Unit"

Another case of "VRF not found".
Check the log files on al2sam shown below indicates that he VLAN 3500 on the endpoint core1.wash is already in used.
[nrig-service@al2sam ~]$ cd ControlFramework/fabric_cf/authority/al2s-am/
[nrig-service@al2sam al2s-am]$ ls -ltr logs/*-handler.log
https://al2s.net.internet2.edu/oess-service-basic/vrf.cgi?method=provision&name=MySlice-al2s-3-296256999979&description=MySlice-al2s-3-296256999979&local_asn=55038&workgroup_id=1482&endpoint=%7B%22node%22%3A%20%22core1.wash.net.internet2.edu%22%2C%20%22bandwidth%22%3A%200%2C%20%22interface%22%3A%20%22HundredGigE0/0/0/28%22%2C%20%22tag%22%3A%20%223500%22%2C%20%22jumbo%22%3A%201%2C%20%22cloud_account_id%22%3A%20null%2C%20%22entity%22%3A%20%22al2s%22%2C%20%22peers%22%3A%20%5B%7B%22bfd%22%3A%200%2C%20%22ip_version%22%3A%20%22ipv4%22%2C%20%22peer_ip%22%3A%20%22<IP_ADDRESS>/24%22%2C%20%22peer_asn%22%3A%20%22398900%22%2C%20%22local_ip%22%3A%20%22<IP_ADDRESS>/24%22%2C%20%22md5_key%22%3A%20%22secret%22%7D%5D%7D&endpoint=%7B%22node%22%3A%20%22agg4.ashb.net.internet2.edu%22%2C%20%22bandwidth%22%3A%201000%2C%20%22interface%22%3A%20%22HundredGigE0/0/0/7%22%2C%20%22tag%22%3A%20%22101%22%2C%20%22jumbo%22%3A%201%2C%20%22cloud_account_id%22%3A%20%22296256999979%22%2C%20%22entity%22%3A%20%22al2s%22%2C%20%22peers%22%3A%20%5B%7B%22bfd%22%3A%200%2C%20%22ip_version%22%3A%20%22ipv4%22%2C%20%22peer_ip%22%3A%20%22<IP_ADDRESS>/24%22%2C%20%22peer_asn%22%3A%20%2264512%22%2C%20%22local_ip%22%3A%20%22<IP_ADDRESS>/24%22%2C%20%22md5_key%22%3A%20%220xzsEwC7xk6c1fK_h.xHyAdx%22%7D%5D%7D

Komal submitted a request on the the algorithm changes regarding the VLAN issue.
https://fabric-testbed.atlassian.net/browse/FIP-714
Change Broker Allocation Algorithm to spread resources across available hosts more evenly as opposed to first fit.

AWS
Found an issue with the method "find_available_dx_connection()" described as follows:
In one test case,  a connection is ordered but an exception is raised from this method. The reason is that multiple connections might have the same name but one is in state "deleted", which ends up with timeout exception.
A fix is committed and ready for reviewing.
GCP
Tested with Fabfed that workflow is created successfully and the GCP VMs can be reached from the Fabric node via ping and ssh.

One observation I had with the AWS provider is that it would not work with a VPC that already had a routing table created with subnets being associated to.
The fabfed provider would create a separate routing table and VGW which is disconnected with the existing routing table.
A better flow would be:

check if a VPC has a VGW associated; if so, use that VGW to associate with the DX VIF. Otherwise, create and associate a VGW.
check if a VPC has an existing main routing table. If so enable route propagation to the VGW with this existing routing table. Otherwise, create a new routing table and enable the propagation (also prompt a warning message that users need to associate any existing subnets to this routing table)


SAML assertions cannot be serialized (returns empty strings).

Trying to use Microsoft.IdentityModel.Tokens.Saml to generate SAML assertions.
Seeing a strange issue when trying to serialize the assertions, from looking up examples elsewhere in github etc.. The following should work:
var assertion = new Saml2Assertion(new Saml2NameIdentifier("foo")) {
	Subject = new Saml2Subject(new Saml2NameIdentifier("bar"))
};

using var stringWriter = new StringWriter();
using var xmlWriter = XmlWriter.Create(stringWriter);

var serializer = new Saml2Serializer();
serializer.WriteAssertion(xmlWriter, assertion);

// Expect to see XML string
Console.WriteLine(stringWriter.ToString());

Expected to see the XML as string, however this outputs an empty string.
Versions:
Have not been able to get this working with current (6.8.0) version and .NET 5.
Tried some older versions of the library such as 5.4.0 to make sure this wasn't a regression but this wasn't the case.
===
I have some suspicions that perhaps writing assertions doesn't work without a "proper" SAML assertion complete with signing credentials. Is this the case?
PS A similar behavior is exhibited when trying to serialize a SAML Token with a Saml2SecurityTokenHandler

Closing this, it was user error.
using var xmlWriter = new XmlTextWriter(stringWriter);

It appears the XmlWriter.Create() returns something very different from what I expected.


feat: serverSide validation error markers

@bripkens removed server-side word

We need to deduplicate errors on the same path: if two errors are found on the same path, the server-side error takes precedence, see attached screenshot.


Special GC (Incremental GC with Central Object Table)

Do not merge - this is only an experiment for comparison.
Special GC (Incremental GC with Central Object Table)
Incremental, generational, and compacting garbage collector based on a central indirection table for object addresses.
Objective: An incremental GC with a different design than the evacuation-compacting GC (https://github.com/dfinity/motoko/pull/3837).

No partitioned heap like in the evacuation-compact GC.
Easily combinable with generational collection.
Simple in-place full-heap compaction.
No pointer update phase.

Properties:

Fast reclamation of short-lived objects with generational collection.
Full-heap incremental snapshot-at-the-beginning marking.
Incremental compaction enabled by a central object table.

Design
The key concept of this garbage collector is to introduce a central object table over which all references to an object are indirected. This allows atomic movement of objects without having to search and update their incoming pointers. Based on this mechanism, the GC can perform incremental heap compacting, similar to a simple mark-and-compact GC.
Central Object Table
Each object has its object id that serves to reference the object. Thereby, all references are indirected via a central object table, that maps an object id (object reference) to the corresponding object address.
This enables fast moving of objects in the incremental GC by only updating the address of the corresponding object in the table. Objects also carry their id in the header to allow fast reverse lookup, i.e. determining the object id for a given object address.
The table object is allocated in the heap and can also be moved (e.g. when growing). A global pointer denotes the current table location. Allowing relocation of the table is more expensive in terms of object address lookup (extra indirection via the global object table pointer), but allows a significantly simpler implementation, since the table can grow at any time without having to move other objects.
The ids of reclaimed garbage objects are recycled in the object table using an inlined free list.
When running out of free object ids, the table is currently relocated to double-sized new table at a different location in the heap. Pre-amortized growth could be used in future to obtain O(1) worst-case per insertion.
Garbage Collection
Garbage collection is performed in two steps:
Young Generation Collection (Blocking)
The young generation collection runs non-incrementally, blocking the mutator. This is acceptable as the generation tends to be small and the GC work should adapt to the allocation rate
It is scheduled:

Before each GC increment of the old generation, for simplifying incremental collection.
After a certain amount of new allocations, i.e. when young generation has exceeded a threshold.

Young generation collection aims at fast reclamation of short-lived objects, to reduce GC latency, which is particularly relevant in an incremental GC.
The young generation collection requires an extra root set of old-to-young pointers. Those pointers are caught by a write barrier and recorded in a remembered set. The remembered set lives in the young generation and is freed during the young generation collection.
The compaction phase can use the simple object movement enabled by the central object table provided for the incremental GC.
Old Generation Collection (Incremental)
Old generation collection runs incrementally in multiple GC increments, where the mutator can resume work in between. It is scheduled to run after young generation collection, such that it can ignore generational aspects.
New objects are always first allocated in the young generation and become only visible to the incremental GC after they have been promoted to the old generation (i.e. survived the preceding young generation collection).
The incremental collection of the old generation is based on two phases:

Mark: Incremental snapshot-at-the-beginning marking. The GC marks at least all objects that have been reachable at the start time of the incremental GC run. A pre-update write-barrier detects relevant concurrent pointer overwrites by the mutator between GC increments and marks the corresponding objects to guarantee snapshot-at-the-beginning consistency. Concurrent allocations to the old generation (being promotions from the young generation) are conservatively marked.
Compact: Incremental compaction supported by the central object table. After the mark phase, the marked objects of the old generation are compacted towards the bottom of the generation. Objects can be easily moved one-by-one since all incoming references to a moved object can be atomically updated in the central object table. Concurrent allocations to the old generation during this phase (promotions from the young generations) are marked because they have to be retained and compacted during this incremental GC run.

GC increments are currently only scheduled on empty call stack such that we can ignore pointers on the call stack for the root set.
In-Header Mark Bit
The GC uses a mark bit in the object header instead of mark bitmaps since there is no performance advantage by using the mark bitmap for skipping garbage objects. This is because the object ids of garbage objects need to be freed in the object table and therefore, the compaction phase must visit all objects (marked and unmarked).
GC Configuration

Increment limit: Regular increment bounded to 4,000,000 steps (approximately 400 million instructions).
Young generation collection: When exceeding 8 MB young generation size.
Old generation collection: When the old generation has grown more than 2.5 times its size or passed the critical limit of 3.5GB (on 4GB heap size).

The configuration can be adjusted to tune the GC.
Measurement
The following results have been measured on the GC benchmark with dfx 0.12.1. Special denotes this new garbage collector, while Incremental here refers to the other incremental evacuation-compact GC (https://github.com/dfinity/motoko/pull/3837).
The Copying, Compacting, Generational GC are based on the original runtime system without the header extension (for object id). No denotes the disabled GC based on the runtime system also with the header extension. Measurement results are rounded to two significant figures.
Scalability
Summary: Like the other incremental GC, the special GC scales up to the full heap size.
Average amount of allocations for the benchmark limit cases, until reaching a limit (instruction limit, heap limit, dfx cycles limit).

GC
Avg. Allocation Limit

Special
140e6

Incremental
140e6

No
47e6

Generational
33e6

Compacting
37e6

Copying
47e6

Performance
Total number of instructions (mutator + GC), average across all benchmark cases:

GC
Avg. Total Instructions

Special
2.9e10

Incremental
2.0e10

Generational
1.9e10

Compacting
2.2e10

Copying
2.0e10

45% slower than the other incremental and copying GC. 31% slower than the compacting GC. 52% slower than the basic generational GC.
Memory Size
Allocated WASM memory space, benchmark average:

GC
Avg. Memory Size

Special
260 MB

Incremental
290 MB

No
500 MB

Generational
190 MB

Compacting
190 MB

Copying
270 MB

12% smaller than the other incremental GC. 4% smaller than the copying GC. 37% larger than the compacting and generational GC.
Overheads
Additional mutator costs implied by the incremental GC:

Object table indirection

Dereferencing requires an extra look up in the object table.
Two more instructions (add and load) for each field or array element access (and object header access).

Write barrier:

Pre-update barrier during the mark phase: Marking the target of overwritten pointers.
Post-update barrier: Recording old-to-young generation pointers in the remembered set.

Around 45% mutator overhead for the object table indirection and write barrier.
Testing
The special GC is enabled by --incremental-gc compiler flag (like the other incremental GC).

RTS unit tests
In Motoko repo folder rts:
make test

Motoko test cases
In Motoko repo folder test/run and test/run-drun:
export EXTRA_MOC_ARGS="--sanity-checks --incremental-gc"
make

GC Benchmark cases
In gcbench repo:
./measure-all.sh

Discussion
Advantage

Low-latency reclamation: Fast memory reclamation due to the use of generational collection on top of the incremental collection. No update phase is needed in the incremental GC.

Limitations

Object table growth: Currently, for implementation simplicity, the object table is extended by copying the entries to a double-sized copy, which causes an unbound interruption in the incremental GC. This could be improved by using a pre-amortized table growth implementation.
Object table shrinking: Due to the fragmentation, the object table cannot be easily compacted without having to update all references.
Indirection costs: The performance costs of central object table iteration is noticeable by around 45%.
Iterating objects: In contrast to other GCs using a mark bitmap, all free objects need to be also iterated for freeing the object id.

Conclusion
The runtime costs of central object table indirection are not extremely high but relevant at around 45%. Moreover, the growth of the central object table is non-trivial. Both incremental GCs have similar code size complexity with around 5,200 - 5,500 LOC (additions and deletions): While the special GC avoids the partitioned heap, it introduces the central object table.
Except for the higher memory footprint, the incremental evacuation-compact GC shows superior performance according to the benchmark.
Related PRs

Incremental GC (based on evacuation-compaction): https://github.com/dfinity/motoko/pull/3837
Generational GC: https://github.com/dfinity/motoko/pull/3495

I wonder if this approach would be much more viable if we could just use a second, isolated memory for the object table. Unfortunately, the replica doesn't support this I think, but wasm does. Or use stable memory, once fast enough, but probably not efficient enough in combination with regions. @luc-blaeser


Command-line switch not accepted as charger control switch

Version of the custom_component

Version 2.0.1
Configuration
Home assistant v2024.7.4 running in docker with this integration downloaded using HACS
Describe the bug
When trying to configure the integration it does not accept the switch entity that I have that controls the EV charger. I get the error "Brytare för laddningsstyrning hittas inte.". Yet the switch entity exists and can be used as switch everywhere else in HA. I have been using it for many years from automations et.al.
If I substitute the switch entity in the configuration with one for a zigbee switch it is accepted.
The switch is a command-line switch that is configured like this:
command_line:
  - switch:
      name: the_name
      command_on: "..."
      command_off: "..."
      command_state: "..."
      value_template: '...'

Debug log
There are no debug logs from this integration that shows that anything goes wrong when I try to fill in the config parameters.

2024-09-03 12:20:26.659 DEBUG (MainThread) [custom_components.ev_smart_charging.config_flow] EVChargingControlConfigFlow.__init__
2024-09-03 12:20:26.659 DEBUG (MainThread) [custom_components.ev_smart_charging.config_flow] EVChargingControlConfigFlow.async_step_user
2024-09-03 12:21:05.879 DEBUG (MainThread) [custom_components.ev_smart_charging.config_flow] EVChargingControlConfigFlow.async_step_user

Neither does normal custom switch. I don't have start/stop control in the charger, and the car (vw id3) doesn't have a switch to start/stop charging, but separate button entities. So I created a switch that turn_on --> service: button.press (start) and turn_off --> service: button,press (stop).
However, the integration doesn't allow this. "Charger control switch entity is not a switch."

Use automation for start/stop charging.
Like this

Perfection. Thanks.
Just noting that it seems to me that the integration's own manual start/stop buttons don't seem to work. I do have similar buttons in the VW integration itself. They are actually bit weird, they exist and they work, but they are not visible in the VW integration screen, only in the developer tools - where the description looks identical to the ones created by the VW integration.
Problem with these VW buttons is that they are obviously not connected to the EV smart charging integration, so stopping manually charging will not be relayed to the charging_charging -sensor. Therefore, I created instead a script that presses both start buttons or both stop buttons. One starts/stops the charging and the other just notifies the EV Smart Charging integration. And script triggered when switch.ev_smart_charging_smart_charging_activated switch changes state.
Still in testing phase. Testing is slow as the battery is mostly full.


[RFE] Support updating existing document

Describe the solution you'd like
Currently once you import a document and create mappings, you can't update document schema with keeping existing mappings. When a document schema is updated, e.g. fields are added/removed, you need to start from scratch again. Allow updating documents without discarding existing mappings.
When a field is removed, and that field is in some mappings, we should remove those mappings. Maybe we can show some warning in dialog with showing the reason, "Since field A is removed, corresponding mappings will be removed"

Hi @igarashitm , I also need this feature as this is very useful. This is pain to start mapping from scratch, if there is any change in source and target document.

Hi, has anyone had an update on this or has been working on this feature?. I am currently working on this, but when I tried to get the fields from the imported document, they are empty

So - to be clear - the enhancement is to allow the import of an existing document name (i.e. update the document).  The enhancement does not entail dynamically adding fields into the document from the UI.  Any missing field in the updated document that is currently mapped will result in a dialog challenge stating that the resulting mapping will be deleted y/n.
i.e. The update of source file myfile.json results in the elimination of a mapped field(s).  These mappings will be removed.  Proceed?
All other mappings will remain intact.  Does  that sound correct?

That's right. Modifying a field individually has a separate issue - https://github.com/atlasmap/atlasmap/issues/1417

Right - so for the first pass at this - if the updated document removes or changes the type of a mapped field the mapping is removed (post challenge).  We can revisit recalculating the mapping with the updated field type in a future enh.  I think the lowest hanging fruit is just where a user adds an unrelated field to a document and doesn't want to lose their existing mappings.
Since we have the document management service and we have the docdefs and mappings already established I was thinking that would be a good place to see if we can support this.  We'll definitely need an updated REST api once we're ok with updating.  wdyt?

I would want to avoid having more logic in UI, actually if we have hands I want to move more logic from UI atlasmap-core to backend. That's far better for testing and maintaining than in typescript toolchain.

Ok - I'll look pushing as much as possible to the backend.   Good point w.r.t. dynamic inspection.  It could get hairy.

https://github.com/atlasmap/atlasmap/issues/2214#issuecomment-1035472021

This would need an update icon, as you can import multiple Documents from a same schema. And the "Update Document" icon needs to be next to the "Remove Document" icon.

This one turned out to be even more complicated - blocked by the server side field search (#603) in order to handle children fields that's already used in the mappings.

Hi @davidg2019 ,
There is work around for that If you want to update schema after creating mapping. You can directly change schema in .adm file by renaming it to .zip file. The grammar is wriiten is json and easy to understand.

+1 on this feature, that workaround is very painful for something that should be an OOTB feature IMHO. If you are dealing with legacy interfaces then yes the source/target documents may not change but when using AtlasMap to iterate on an integration with a new interface/endpoints having to remap everytime the source or target changes makes it impossible to use the tool effectively.