integration-test: Change assert_loaded to a macro with timeouts

We often need to wait a bit until the program or link gets unloaded after dropping.
Also, using a macro makes it clear in which particular test the panic happened.
Signed-off-by: Michal Rostecki<EMAIL_ADDRESS>

cool. you're too fast and furious! I've confirmed it worked.
aya   fix-integration-tests  via  v1.66.0-nightly took 6s
❯ RUST_BACKTRACE=full cargo xtask integration-test --libbpf-dir ~/projects/oss/libbpf/
    Finished dev [unoptimized + debuginfo] target(s) in 0.33s
     Running `target/debug/xtask integration-test --libbpf-dir /home/hnidoaht/projects/oss/libbpf/`
       Fresh core v0.0.0 (/home/hnidoaht/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core)
       Fresh unicode-ident v1.0.4
       Fresh rustc-std-workspace-core v1.99.0 (/home/hnidoaht/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/rustc-std-workspace-core)
       Fresh proc-macro2 v1.0.46
       Fresh compiler_builtins v0.1.79
       Fresh quote v1.0.21
       Fresh rustversion v1.0.9
       Fresh aya-bpf-cty v0.2.1 (/home/hnidoaht/projects/oss/aya/bpf/aya-bpf-cty)
       Fresh syn v1.0.101
       Fresh aya-bpf-bindings v0.1.0 (/home/hnidoaht/projects/oss/aya/bpf/aya-bpf-bindings)
       Fresh aya-bpf-macros v0.1.0 (/home/hnidoaht/projects/oss/aya/aya-bpf-macros)
       Fresh aya-bpf v0.1.0 (/home/hnidoaht/projects/oss/aya/bpf/aya-bpf)
       Fresh integration-ebpf v0.1.0 (/home/hnidoaht/projects/oss/aya/test/integration-ebpf)
    Finished dev [unoptimized + debuginfo] target(s) in 0.15s
  INSTALL  bpf.h libbpf.h btf.h libbpf_common.h libbpf_legacy.h bpf_helpers.h bpf_helper_defs.h bpf_tracing.h bpf_endian.h bpf_core_read.h skel_internal.h libbpf_version.h usdt.bpf.h
   Compiling integration-test v0.1.0 (/home/hnidoaht/projects/oss/aya/test/integration-test)
    Finished dev [unoptimized + debuginfo] target(s) in 2.56s
[sudo] password for hnidoaht:

1 more question, it won't show any outputs? Just finished.

1 more question, it won't show any outputs? Just finished.

Because we didn't add any outputs. :sweat_smile: I can add them in a follow-up PR next days.

1 more question, it won't show any outputs? Just finished.

Because we didn't add any outputs. sweat_smile I can add them in a follow-up PR next days.

awesome! I feel like I've learned a lot from joining discord and reading all of your(s) code 😄 .
Btw, I was pulled from work and away from aya for about a week to travel with my family. hopefully since next week I can dig deeper into aya code and contribute anything no matter how small to this charming project.


Export the DINED database in standard formatted csv files and sharing those on 4TU repository.

This would allow us to work with the dined data more easily for the time being. Once we have an open API to approach the DINED database, this would not be necessary anymore.

How could the dataset look like?
How should a study look like?
Each row is an individual:
| individual_id | sex | units | age | study_id | measure_n  | measure_n+1 |  measure_n_unit

Ideas and options:

Separate each study with its name and metadata (can in a folder or at the root)
Separate files for the studies one for the metadata, here we refer to metadata as measures description.
There is a description about the study, this can be also a separate metada or summary file. (currently is in json, perhaps this could be turned into more of a readme style kind of format)

measures.json could be a specification, perhaps a dined specification. Think about this as a specification that could be described with json schema for instance.
measure_n could be a type as well, for instance we could do type checking for certain measures like the type can only be positive or a float, and it needs to be of a certain class like mm???

This has been solved as well as the generation of metadata that describes the set of standardized files as a data-package using frictionless standards and frictionless framework.


Features on Rust homepage are unintelligible jargon

Features listed on Rust’s homepage are implementation details, rather than benefits for the user.
Now that I know Rust I appreciate how useful these features are, but before using Rust I never thought I'd want a language with "move semantics" or "trait-based generics". In fact, such terms make it sound complicated and off-putting.

move semantics — to someone coming from C it is absolutely meaningless term, but "you can guarantee there's only one copy of a struct" or "you can enforce an object is not used after it's freed" is marvelous.
trait-based generics — I’ve first heard of "traits" in PHP, and generics in Java, and neither of these seemed interesting.
type inference —  doesn’t sound as great as “you get strict type-checking without having to write every type manually”
minimal runtime — could be “can be linked with C programs with no hassle” and “works on microcontrollers”

I appreciate it's very hard to replace that list, because it's impossible to explain these things in 3 words. For most of these features I had no idea I want them until I used them.
And it's missing some other features (some of which are not strictly language features, but make Rust great in practice):

error handling as convenient as exceptions, as predictable as error codes,
standard package manager that makes dependencies easy to use and reliable,
easy to use built-in unit testing,
clear and helpful error messages,
developed in the open, with cooperation with the community.

I think Elixir website presents the features well:
https://elixir-lang.org/
Each feature is a section with description and example code. It's easier to show e.g. pattern matching than to describe it.

This is certainly something most people agree on.

I support this deeply. It took me more than a year from seeing the main page to discovering what Rust really was about. Something really should be done about that!

Yep, the website is getting a complete overhaul as part of Rust 2018, and this issue will definitely be addressed there.
I'm going to close this issue though as it's out of scope for this repo.


[] build failure

Host Environment

OS: Windows 10 Enterprise
Compiler: Visual Studio 2019 Enterprise 16.5.3
VCPKG 2020-06

To Reproduce
vcpkg install libmysql
OR
vcpkg install icu
Failure logs
Output on the console:
C:\vcpkg-2020.06>vcpkg install libmysql
Computing installation plan...
The following packages will be built and installed:
  * icu[core]:x86-windows
  * libevent[core,thread]:x86-windows
    libmysql[core]:x86-windows
  * lz4[core]:x86-windows
  * xxhash[core]:x86-windows
Additional packages (*) will be modified to complete this operation.
Starting package 1/5: icu:x86-windows
Building package icu[core]:x86-windows...
-- Using cached C:/vcpkg-2020.06/downloads/icu4c-67_1-src.tgz
-- Using source at C:/vcpkg-2020.06/buildtrees/icu/src/c-67_1-src-39ba50b26e
-- Acquiring MSYS Packages...
CMake Error at scripts/cmake/vcpkg_execute_required_process.cmake:72 (message):
    Command failed: C:/vcpkg-2020.06/downloads/tools/msys2/msys64/usr/bin/bash.exe --noprofile --norc -c "pacman -S --noconfirm --needed make automake1.15"
    Working Directory: C:/vcpkg-2020.06/downloads/tools/msys2
    Error code: 1
    See logs for more information:
      C:\vcpkg-2020.06\buildtrees\icu\msys-pacman-x86-windows-err.log

Call Stack (most recent call first):
  scripts/cmake/vcpkg_acquire_msys.cmake:115 (vcpkg_execute_required_process)
  ports/icu/portfile.cmake:78 (vcpkg_acquire_msys)
  scripts/ports.cmake:76 (include)

C:\vcpkg-2020.06\buildtrees\icu\msys-pacman-x86-windows-err.log:
error: mingw32: key "4A6129F4E4B84AE46ED7F635628F528CF3053E04" is unknown
error: key "4A6129F4E4B84AE46ED7F635628F528CF3053E04" could not be looked up remotely
error: mingw64: key "4A6129F4E4B84AE46ED7F635628F528CF3053E04" is unknown
error: key "4A6129F4E4B84AE46ED7F635628F528CF3053E04" could not be looked up remotely
error: msys: key "4A6129F4E4B84AE46ED7F635628F528CF3053E04" is unknown
error: key "4A6129F4E4B84AE46ED7F635628F528CF3053E04" could not be looked up remotely
error: database 'mingw32' is not valid (invalid or corrupted database (PGP signature))
error: database 'mingw64' is not valid (invalid or corrupted database (PGP signature))
error: database 'msys' is not valid (invalid or corrupted database (PGP signature))

Additional context
x64 version also fails. I get the error trying to install libmysql
The latest branch works fine, I have not tried 2020-07

This does appear to be fixed in the later release and in the master branch so I'm guessing it can be closed.

Thanks for posting this issue. I install icu successfully on the latest VCPKG. I am closing this issue for now. Please reopen this issue if this is still a problem for you.


Fixing full_fact function and docs updated

construct_df, used by full_fact needed some fix to be used with float. Examples in the docs before this fix are wrong, and some values were false so I fixed it as well.

Can this be merged? I have this bug too

This change is required with newer versions of pandas. Is this code still maintained?

This change is required with newer versions of pandas. Is this code still maintained?

Idk, I hope so. The author commited a few this year.


extract: T/UT is supposed to have no underscore in attributes

This one just adds a fallback for _context (context), _comment (comment), _charlimit (charlimit) and _tags (tags) so that usage of T and UT will get it's params out.
According to the documentation for Angular the params are without _ on the Angular T / UT component. This was handled for key but not for the other ones that are sent to Transifex. I should have seen it before :(
Includes one test to check that it does correct for _context, the rest just follows the same logic.

@pablotransifex Do you re-call what is the case with _ in reserved variables?

@pablotransifex Do you re-call what is the case with _ in reserved variables?

@nbasili I remember we had in the beginning the TX Native _ variables in the T/UT as parameters, but for being more standard, we removed the _ in the component parameters list.


Report uncovered as errors with GithubActionOutputFormatter

resolves #510

should this be something general (not just for the github error formatting?)

I also updated the (default) TableOutputFormatter but that was just using red instead of yellow as color.
If one interested for the same behavior for junit or graphviz I would be happy about an PR. The XmlOutputFormatter doesn't report uncovered yet and the ConsoleOutputFormatter might be removed the next time.

oh I see. thx for the insights.


Fix Quickstart for FedRAMP

Proposed changes

Checklist

[x] I have signed the MongoDB CLA
[x] I have added tests that prove my fix is effective or that my feature works
[x] I have added any necessary documentation (if appropriate)
[x] I have updated e2e/E2E-TESTS.md (if a new command or e2e test has been added)
[x] I have run make fmt and formatted my code

Further comments

➜  ~/workspace/mongoCLI git:(CLOUDP-94313) ✗ ./bin/mongocli atlas quickstart
You are creating a new Atlas cluster and enabling access to it.

Press [Enter] to use the default values.

Enter [?] on any option to get help.

[Set up your Atlas cluster]
? Cluster Name [This can't be changed later] Quickstart-1626180399
? Cloud Provider AWS
? Cloud Provider Region US_GOV_EAST_1

We are deploying Quickstart-1626180399...
? Do you want to load sample data into Quickstart-1626180399? Yes

[Set up your database authentication access details]
? Database User Username Quickstart-1626180399
? Database User Password [Press Enter to use an auto-generated password 'SH1pV3NIb3tZ'] SH1pV3NIb3tZ

[Set up your database network access details]
? Access List Entry [Press Enter to use your public IP address '<IP_ADDRESS>'] <IP_ADDRESS>

[Connect to Quickstart-1626180399]
? Do you want to access Quickstart-1626180399 with MongoDB Shell? Yes

Creating your cluster... [It's safe to 'Ctrl + C']
-
Loading sample data into your cluster... [It's safe to 'Ctrl + C']
-
Now you can connect to your Atlas cluster with: mongosh -u Quickstart-1626180399 -p SH1pV3NIb3tZ mongodb+srv://quickstart-1626180399.jxkfe.mongodbgov-dev.net

Current Mongosh Log ID: 60ed8e81f3c64601168050ff
Connecting to:    mongodb+srv://quickstart-1626180399.jxkfe.mongodbgov-dev.net/
Using MongoDB:      4.4.5-1-fips-openssl-1.0.2
Using Mongosh Beta: 0.8.2

For mongosh info see: https://docs.mongodb.com/mongodb-shell/

atlas-13pzho-shard-0 [primary]>

This just missed the 1.19 release :(


Score incongruous for repeated words

Examples:

apple apple:

Score: 3/4

apple apple : (note the trailing space)

Score: 1/4

apple apple apple

Score: 2/4

Especially the first example seems way off.

Yeah, I found this issue too, and I'm definitely concerned about it since I'm planning to have a longer minimum password length than most sites (why does everyone only do 8?), but I don't want people to use an insecure password but just repeat it and then have my password checker not notice. I cooked up this little script to find repeated strings (based on LZ77-style compression algorithms): https://gist.github.com/laughinghan/bf0ff29e11d6ef36881a92e4a35abb98
Wanna help me integrate it into their scoring system and turn it into a PR?

Related (or possibly duplicate of) #216


SAK-29397 remove standalone gradebook app as well as other modules previously marked for removal

There is currently an unused standalone app inside the gradebook source that is unused, however requires updating whenever APIs change etc.
This should be removed.
Also, the other bits that are marked with the remove-10 build profile.

I looked at doing this awhile back but held off because removing this looked like it removed a bunch of unit tests for gradebook. Is this something we want to do without relocating them? I think this would be okay if the new UI got in place and these weren't used at all anymore but it looked like many of these were testing the service and just were never moved. A subtask back of SAK-29101.

 Unknown if they work or what. The code has been out of the build for a while. Happy for someone to volunteer to move them, but I won't be updating an out of date module when I change the API in the coming days.

p.s. new UI will be in trunk in the coming weeks. Though I will be refactoring the API this week. Chicken and egg ;)

The samigo and gradebook standalone versions have caused me to lose countless hours of work while searching in the wrong places.  This is welcome cleanup.

Is this ok to be merged?


build: remove dependency on modified folly repository

Currently we pull a modified folly repository to be able to use it header only.
The only difference in this repository is adding the folly-config.h file.
Pull this patch into this repo instead and use upstream folly.
Test plan:

CI build. I have local build issues at the moment.

Noice. Any reason not to use a more recent release of folly?

Nope! This is the upstream base of the old modified branch, I'll try the latest hash, should be fine.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 60.78%. Comparing base (8d83ae1) to head (e434767).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #499   +/-   ##
=======================================
  Coverage   60.77%   60.78%
=======================================
  Files         126      126
  Lines       12488    12488
  Branches     2014     2014
=======================================
+ Hits         7590     7591    +1
+ Misses       3941     3940    -1
  Partials      957      957

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


feat: added support for MM flask v10.24.1 and added starkNet_estimate…

…AccountDeployFee

I have an issue if I update my flask and I already had the flask + snap installed.
The dapp asks me to install the latest snap which is 1.2.1 here but then I get the error : "Failed to decrypt snap state, the state must be corrupted." once installed

If I uninstall the snap and reinstall, it works fine.

@Julink-eth pushed another commit, for the install snap error, I guess its because your existing and new snap version are both in v1.2.1 and the dApp was calling the old snap which is not compatible with Flask v10.25, and hence u saw the "snap state" error message.

I have an issue if I update my flask and I already had the flask + snap installed. The dapp asks me to install the latest snap which is 1.2.1 here but then I get the error : "Failed to decrypt snap state, the state must be corrupted." once installed
If I uninstall the snap and reinstall, it works fine.

Nevermind https://www.notion.so/0-27-1-24767c05a7684851ab856fbe8a2370b5

@Julink-eth pushed another commit, for the install snap error, I guess its because your existing and new snap version are both in v1.2.1 and the dApp was calling the old snap which is not compatible with Flask v10.25, and hence u saw the "snap state" error message.

Yes exactly and I think the only thing we could do is display a modal for the user to reinstall their snap because they are going to be stuck otherwise


Fixes #40926 for HTTP response on write error.

Fixed behaviour of ResponseHeaderTimeout in http.Transport - even in case property was set it was not honoured.
According to documentation if ResponseHeaderTimeout is set and fully writing the request happens implementation shall wait for specified time for response before sending error.
Steps to reproduce:
Prepare a POST/PUT request with body.
Try to send it to the server requiring authentication.
The server respond with authentication request and closed the client connection (to avoid sending huge file from not authenticated source)

Result the method RoundTrip returned error only without response.
Expectation response will be returned if received within specified time span.

@googlebot I consent.

@tz70s @bcmills @zhangfannie @cuonglm can you please consent? I just rebased my code to resolve conflicts so it contains your commits.

@googlebot I consent.

@googlebot I consent.

If you want to go ahead with this change, I would suggest to squash and rebase your commits into a single one authored just by yourself. That should make the CLA bot happy.

If you want to go ahead with this change, I would suggest to squash and rebase your commits into a single one authored just by yourself. That should make the CLA bot happy.


cmd/go: remove -buildmode=shared (not c-shared)

-buildmode=shared has been broken since modules, and it is apparently unused.
See https://github.com/golang/go/issues/47257#issuecomment-897243256.
Let's remove it.

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

I really want to use this buildmode=shared mode.

@rsc how has it been broken since modules?

Just to clarify, this is not related to --buildmode=plugin, right?

@electricface Why do you want to use it?  Are you using it today?
@scott-cotton You can't use -buildmode=shared with a module and then use -linkshared to link against it from other modules.  The whole idea of -buildmode=shared doesn't fit very well with modules.  Each separate program can be linked against different versions of dependent modules, so if it did work people would in practice wind up with many different versions of the shared libraries of the same modules.
@ainar-g Correct, this is not about -buildmode=plugin, although plugins have their own, different, problems.

@ianlancetaylor
I am a debian-baseed Linux Desktop Environment developer. Our team have developed lot of daemon programs with golang. But these programs have significantly larger binary file size and use more ram than C or C++ programs. We currently working on these two problems. An idea is making some modules pluggable, which means only load modules that we need.
As other C or C++ programs are using shared-library, we are researching on some shared-library-like mechanism of golang. Inspired by buildmode=plugin and goloader, we found a way that bases on buildmode=shared to make modules pluggable recently, you can found a demo here. This solution has not been using in production environment, because of some problems hasn't been solved yet. I found out that use shared library may increase memory use and programs built with -linkeshared can not be debugged using gdb and dlv.
After this commit, our "unoffical plugin" solution can work fine with go 1.14.9~1.16.7. Programs built in this way can run stably. But this won't work on go 1.17. Program will exit with a "fatal error: unreachable method called. linker bug?"
  We aren't going to support -buildmode=shared just to support plugins, since we already have -buildmode=plugin.

I know that using static linking can eliminate the problems caused by the mismatched version of the dependent dynamic library.
But in the linux desktop system ecosystem, C or C++ languages are mostly used. These languages generally use dynamic libraries to save memory. The main reason for not using -buildmode=plugin is that static linking cannot save memory. For example, if the modules are divided into smaller ones and a lot of modules are added, each module occupies a relatively large amount of memory, and the overall occupancy will eventually become unacceptably large. Compared with C or C++ languages, they only use less memory to implement these functions, while the golang implementation uses several times the memory.
There is even an idea in our team that we must change the development language if we can't reduce the memory. I personally like to use golang for development, so I plan to study this part of the technology.

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

@rsc not sure if this is a good place to give feedback about the proposal process, but this is the first time (since the proposal process exists) that I missed an issue. I was mildly interested in this (not enough now to vehemently protest), but even if I'm subscribed to the proposal process issue and I normally read minutes every week, I missed this. I took 10 days off for vacation, and those 10 days covered the only 2 emails where this issue was mentioned, before today when it was approved. Again, I don't mind about this specific issue at this point but I wanted to raise a general point.
Maybe a proposal should stay a minimum time in the proposal process before being closed, so that even people that don't work on Go full time have time to notice and react even with personal life happening in-between.

To limit binary file size, I use command go install -buildmode=shared std to install libstd.so. And I build my application by go build -linkshared. I have been using this in  production environment for a long time. I hope I can still use it in the furture.
Can we keep  -buildmode=shared std, it's useful.

Our company has been using the shared mode to reduce the binary size for a long time. Currently, there is no better way to reduce the binary size. The plugin mode is not helpful in most of time because the entire runtime package needs to be compiled.

Russ wrote:

The only problem I'd still be concerned about is the handling of debug information, but that should be doable

One thing we can do that would make the problem more tractable in general would be to move DWARF type DIE generation out of the linker and into the compiler (this is an idea that we've toyed with the in the past but haven't gotten to).  Doing that would definitely help for the "link shared against libstd.so" use case.

No change in consensus, so declined.
— rsc for the proposal review group

So...
Do you have any legal review board? Obviously not,
What shall we do with legal license requirements taht shared library is ok but compiling into static is not?
I think a sane programming language should have shared library support
It seems we have to move to rust as Go is getting crazier everyday

@kiap I don't know if you realize that this proposal was declined and was not implemented.
There is a new suggestion in https://github.com/golang/go/issues/47788#issuecomment-954890659.  It would be helpful to know whether that would or would not work for your case.

I, for one, was unaware of that, despite the fact that I follow the issue tracker fairly closely a lot of the time. I stopped following this one, however, once the accepted label got applied. Could the labels be updated to reflect the actual status?

To everyone, but especially to @rasky, @zhouguangyuan0718, @Jason7602, and @mlaventure:
would this alternate UX for -buildmode=shared work for your current use cases?

I'm so sorry for my so late reply, @rsc .The new proposal seems more perfect and graceful for smaller binaries.
But I think there is some details to think over and over.

go build -buildmode=shared -o libgo1.17.2.so std golang.org/x/...
go build -linkshared=libgo1.17.2.so cmd1
go build -linkshared=libgo1.17.2.so cmd2
go build -linkshared=libgo1.17.2.so cmd3

If  I don't misunderstand, in this proposal,  cmdx is a main package, all the dependencies of  cmdx  should be contained in libgo1.17.2.so.  It is a new constraint. Maybe we can support some more packages build in the build command or other way.
And more details I can realize is the dependency chain of dynamiclink.
Will it support a commands looks like go build -buildmode=shared  -linkshared=libgo1.17.2.so  -o libcmd.so cmd?  I think this is useful, And comparing with the new proposal, it will not be a significant burden.
The last thing I'm care about is that if the new proposal is implemented, can it support the plugin with linkshared? Looks like go build -buildmode=plugin  -linkshared=libgo1.17.2.so  -o plugincmd.so cmd. I had implemented limited "plugin with linkshared" and I'm planing to use it in our producting enviroment. I think if the new proposal is accepted, it will more easy and less situation to resolve for "plugin with linkshared" .
I'm always focus on shared library or dynamiclink in go. And I'am glad to contribute any code about shared library or dynamiclink in the future.
Thanks.

So, question to people who saw it working - when was it? (my understanding is that because this feature is on a borrowed time right now, there is no point to open a bug report)

@tandr
It is a bug in go 1.17~1.17.2. I had fixed it in go 1.17.3 and master. You can try 1.17.3 or gotip.

Let me explain: I am quoting from @rsc

There are two main reasons that people reach for shared libraries.
No there are three at least: (or even more)

First: Security fix, one shared library for all of its clients/users code
The argument he gave is it is broken unless you are careful; Sir, you need to stick to SemVer and security fix should not break that compatibility, Go is loosey-goosey, breaking everything of course, (My complaint)
Second: Code size
While this argument is true, I never grasped the idea of one exec file do it all you make, while all bosses and clients and techs used to C style library + executables + other stuff,  Deploying a single large exec makes your commercial product weird to them, there are other obvious technical merits, such as space constraint, Which I do not think Go fits in,
When we picked up Go we wanted to see efficiency but got bloated memory usage, did not let us use it for much system-level management of internal Cloud Virtual Machines (VPSes)
The idea seems for Google is memory is cheap so who cares right!  To Me Go is thinking of itself as a replacement for Java or Python, we gravitated to it from C, for system-level management and distributed systems management.
Third: Legal issues,
Let's say LGPL license, so many commercial codes or licensed code based on patent and any commercial code that permits only shared linking, don't ask my inlining cross-code optimization and thus break that, tell the lawyers who wrote them, those who want to break free from liability need to avoid derivative work, and thus shared linking is required.
There could be other reasons I cannot think of now, I cannot tell even it is three use cases.
@rsc wrote:

We've decided to leave -buildmode=shared alone (not delete it) for Go 1.18,
but it's not going to work any better than it does today.
In particular, we're not fixing any bugs in it anymore,
and we still intend to remove it in a future release.

This is exactly what I experienced since 2016, every major release, something breaks the shared library and I had to report it to get it fixed. the ultimate one is modules.
@ianlancetaylor no the proposed solution in the comments does not work,
for exactly the same reasons of (1) (2) and (3)
We create a library directory that holds shared libraries in /usr/local/libs/
if that library exists go uses shared otherwise just pie see below: (You already have broken this in every major release and I reported and got it fixed)
even we do not create libstd.so
we create disaggregate libs
This is how we do it: (for every version) below is for 1.17
If you see multiple libraries in the list because of Go gotchas, this is the only way you can compile disaggregate (beyond a single comment)
`vi lib_list_1.17.txt
runtime sync
runtime/race
internal/unsafeheader
unicode
unicode/utf8
internal/itoa
math/bits
container/list
container/ring
crypto/internal/subtle
crypto/subtle
unicode/utf16
vendor/golang.org/x/crypto/cryptobyte/asn1
internal/nettrace
vendor/golang.org/x/crypto/internal/subtle
math
encoding
internal/goversion
hash/maphash
image/color
internal/cfg
reflect/internal/example1
reflect/internal/example2
image/color/palette
math/cmplx
runtime/metrics
internal/reflectlite
internal/testlog
internal/sysinfo
internal/singleflight
math/rand
errors
sort
io
internal/oserror
path
crypto/elliptic/internal/fiat
vendor/golang.org/x/net/dns/dnsmessage
strconv
syscall
container/heap
strings
bytes
hash
crypto/internal/randutil
hash/crc32
hash/adler32
crypto/hmac
text/tabwriter
vendor/golang.org/x/crypto/hkdf
hash/crc64
vendor/golang.org/x/text/transform
reflect
crypto
bufio
crypto/rc4
encoding/ascii85
encoding/base32
net/http/internal/ascii
go/build/constraint
regexp/syntax
hash/fnv
html
net/http/internal/testcert
compress/bzip2
image
internal/syscall/unix
internal/syscall/execenv
time time/tzdata
plugin
regexp
image/internal/imageutil
image/draw
image/jpeg
context
io/fs
internal/poll
embed
os
internal/fmtsort
encoding/binary
crypto/ed25519/internal/edwards25519/field
crypto/sha512
crypto/cipher
crypto/md5
crypto/sha1
crypto/sha256
encoding/base64
vendor/golang.org/x/crypto/poly1305
fmt
encoding/pem
crypto/ed25519/internal/edwards25519
net
path/filepath
io/ioutil
internal/lazyregexp
crypto/aes
crypto/des
vendor/golang.org/x/crypto/chacha20
vendor/golang.org/x/sys/cpu
index/suffixarray
os/exec
internal/obscuretestdata
runtime/debug
os/signal
vendor/golang.org/x/crypto/chacha20poly1305
compress/lzw
encoding/hex
os/user
net/url
compress/flate
math/big
vendor/golang.org/x/crypto/curve25519
database/sql/driver
debug/dwarf
debug/gosym
debug/plan9obj
database/sql
encoding/csv
archive/zip
compress/gzip
compress/zlib
encoding/gob
encoding/json
debug/elf
debug/macho
debug/pe
archive/tar
encoding/xml
crypto/dsa
crypto/elliptic
encoding/asn1
crypto/rand
log
crypto/ed25519
crypto/rsa
vendor/golang.org/x/text/unicode/bidi
vendor/golang.org/x/text/unicode/norm
vendor/golang.org/x/crypto/cryptobyte
crypto/x509/pkix
vendor/golang.org/x/net/http2/hpack
mime
mime/quotedprintable
net/http/internal
flag
vendor/golang.org/x/text/secure/bidirule
go/token
text/template/parse
crypto/ecdsa
internal/buildcfg
internal/execabs internal/execabs internal/goroot go/build go/internal/gccgoimporter go/internal/srcimporter
vendor/golang.org/x/net/idna
go/scanner
go/constant
internal/xcoff
text/scanner
image/gif
go/ast
image/png
internal/profile
runtime/trace
internal/trace
crypto/x509
net/textproto
vendor/golang.org/x/net/http/httpproxy
text/template
testing
vendor/golang.org/x/net/http/httpguts
mime/multipart
log/syslog
runtime/pprof
go/internal/typeparams
net/internal/socktest
net/mail
go/parser
go/printer
os/signal/internal/pty
testing/iotest
internal/testenv
crypto/tls
testing/fstest
testing/quick
go/doc
html/template
internal/lazytemplate
testing/internal/testdeps
vendor/golang.org/x/net/nettest
go/types
go/format
net/http/httptrace
net/smtp
net/http
expvar
net/http/cgi
net/http/cookiejar
net/http/httptest
net/http/httputil
net/http/pprof
net/rpc
net/http/fcgi
net/rpc/jsonrpc
go/internal/gcimporter
go/importer
`
Another bash file (This is an example)
`#! /bin/bash
GO_DIR=/usr/src/dev
LIB_DIR=/usr/local/lib/project
WORK_DIR=/usr/src/project
STD_LIB_LIST=lib_list_1.17.txt
            IFS_saved=$IFS
            IFS=$'\n'
            for i in \`cat ${WORK_DIR}/${STD_LIB_LIST}\`
            do
                    IFS=$IFS_saved
                    go install -v -ldflags '-s -w' -gcflags "-trimpath ${WORK_DIR}" -buildmode=shared -linkshared -pkgdir "${LIB_DIR}" $i
            done

            IFS=$IFS_saved

`
then to build:
    # -linkshared here  falls back to static linking if the -pkdir is empty or the libraries could not be found
    # we use this auto-detection of shared libraries, to compile shared or static library just by
    # creating the library directory in Makefile and compiling them using lib option above
    # so if no shared library is found static linking is invoked with pie linkshared  otherwise the available libraries
    # are compiled in as dynamic linking

$2 is your project
        go install -v -ldflags '-s -w' -gcflags "-trimpath ${WORK_DIR}" -buildmode=pie -linkshared -pkgdir "${LIB_DIR}" "$2"
All just to satisfy the legal issue!
The build scripts are more complex I just provided a simplified piece
Even our libraries are placed like that
Unless yu assume every single piece of code you got is licensed as BSD o MIT or apache or you wrote it then feel free for using static linking.
If you cannot satisfy dynamic shared libraries requirement, Go is at the level of Perl, PHP, in our usage, as a medium-size business we need to satisfy the legal department, I cannot comment on Small Businesses who do not care about legalities.

Just to comment on my own comment that
The idea of a shared library is to create once and used it as many times
With Go you have to recompile the entire shared library every time you compile (Unless you keep cache) which is more complicated
Go is not C, I see, but cannot change legalities of licenses too.
So it would be very much appreciated if a shared library acted like exactly as C, an OS vendor then shipped one library of a version in disaggregate.

@DeedleFake Sorry, you're right, the history of this proposal has been confusing.
I removed the "Accepted" label.

@kiap I would be very interested in hearing of any Go package anywhere that is distributed under the LGPL.  Given Go's current technology such a package would effectively be under the GPL, so I don't understand why anybody would choose the LGPL.
If there are no Go packages under the LGPL, then I don't understand the legal concerns.
Thanks.

@ianlancetaylor  No it is not, GPL
Technically even shared libraries in C : part of the code is derived from the original by the compiler,
however:
    "A program that contains no derivative of any portion of the Library, but is designed to work with the Library by being
    compiled or linked with it, is called a "work that uses the Library". Such a work, in isolation, is not a derivative work of
     the Library, and therefore falls outside the scope of this License. ",  the LGPL version 2.1

the exception for shared library  allows the part the compiler or linker mixes in
Sample write a wrapper around libraries in C that are LGPL
There are other works that people write and put under MIT, BSD licenses, if you cannot be sure of the origin then you can be held legally liable regardless of the original author mistake or ill-fated decision,

If there are no Go packages under the LGPL, then I don't understand the legal concerns.

To me, you made your decision Sir, and are simply asking to avoid any other licenses that are not MIT, BSD, Apache, and the like. That's a solution of course why did they clutter the OS with shared libraries, right? perhaps at 46, I am old.

Writing a wrapper around LGPL C libraries is a "work that uses the library."  The C libraries can continue to be shared libraries.  There is no need for the Go code itself to be in a shared library.  That works today and will continue to work.

To me, you made your decision Sir, and are simply asking to avoid any other licenses that are not MIT, BSD, Apache, and the like. That's a solution of course why did they clutter the OS with shared libraries, right? perhaps at 46, I am old.

That is not our intent.  It is fine for Go programs to use GPL libraries.  It is fine for Go programs to use LGPL libraries, as shared libraries, if those libraries are written in other languages.  It's true that it's hard for Go code to be licensed under the LGPL; it is in effect the same as being licensed uder the GPL.  But that doesn't seem so bad as even the FSF recommends against using the LGPL (https://www.gnu.org/licenses/why-not-lgpl.html).
The difficulty with shared libraries in Go is not because we don't like shared libraries.  It's because they are technically difficult to support, and nobody has figured out how to do it well.
And for what it's worth I'm a lot more than 46 years of age.

@ianlancetaylor  thanks for addressing our concern but,
A wrapper around the LGPLed C library for easier use in Go, is still a library  (If it functions as a library it is a library)
A solution would be directly using that C-shared library with cgo and avoid the wrapper
The difference is the programmers should now know the inner working of the C library.
the ultimate solution is just to write the entire LGPL/GPL code in Go
Too much work I suppose

The difficulty with shared libraries in Go is not because we don't like shared libraries. It's because they are technically difficult to support, and nobody has figured out how to do it well.

very well understood, Thank you

A wrapper around the LGPLed C library for easier use in Go, is still a library (If it functions as a library it is a library)

Yes, but the Go wrapper is not a derivative of the C code, and as such the Go wrapper can be under a different license.  The C code, under the LGPL, will be in a shared library, and the Go wrapper, not under the LGPL, will not.

@ianlancetaylor

but the Go wrapper is not a derivative of the C code, and as such the Go wrapper can be under a different license

No sir, that's your technical opinion as an engineer, not a legal opinion
LGPL defines these terms on its own legal universe, Basically, it means you cannot circumvent LGPL by actually writing another library and putting another license to it, doing the same thing, otherwise, this is possible with C, or other languages, if making a library in Go of a library of LGPL, it must be LGPL and be a library, the usage of LGPL directly does not fall under this (as I wrote),
It is well understood legally an LGPL library must be dynamically linked or the application violates this,
This is based on LGPL 2.1  otherwise 3.0 is even more difficult.
A "library" means a collection of software functions and/or data prepared so as to be conveniently linked with application
programs (which use some of those functions and data) to form executables.

...
 2. You may modify your copy or copies of the Library or any portion of it, thus forming a work based on the Library, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions:

a) The modified work must itself be a software library.

b) You must cause the files modified to carry prominent notices stating that you changed the files and the date of any change.

c) You must cause the whole of the work to be licensed at no charge to all third parties under the terms of this License.

d) If a facility in the modified Library refers to a function or a table of data to be supplied by an application program that uses the facility, other than as an argument passed when the facility is invoked, then you must make a good faith effort to ensure that, in the event an application does not supply such function or table, the facility still operates, and performs whatever part of its purpose remains meaningful.

If Go does not support shared library for Go libraries, it is legally safe to say, not to write wrappers around LGPL C code and statically link it, it is very easy to show the user is violating LGPL

No sir, that's your technical opinion as an engineer, not a legal opinion

That is correct.  However, it is also my informed opinion due to having worked on free software licensing issues for decades.  I was one of many people who contributed to the design of the LGPL in the first place.

LGPL defines these terms on its own legal universe, Basically, it means you cannot circumvent LGPL by actually writing another library and putting another license to it, doing the same thing, otherwise, this is possible with C, or other languages, if making a library in Go of a library of LGPL, it must be LGPL and be a library, the usage of LGPL directly does not fall under this (as I wrote),
It is well understood legally an LGPL library must be dynamically linked or the application violates this,

Yes.  In the scenario I described, the LGPL code continues to be dynamically linked.  Absolutely nothing changes with respect to the LGPL code.  No part of the LGPL code is modified in any way.  It continues to be under the LGPL, it continues to be distributed as a dynamically linked shared library.  It continues to be possible for the end user to substitute in a different version of the shared library when running the executable.

@ianlancetaylor
Thanks
My problem Lawyers at our company: Neither GPL nor LGPL has been explicitly tested in a court of law, the linking issue subject to MySQL vs. NuSphere case was settled outside of the court.
So we take the most conservative way; defending in court is too expensive. exceeding the boundaries of compliance to a license is not.

@ianlancetaylor
Some folks contacted me from ITF USA
Saying the GPL / LGPL problem with Go can easily be fixed in the most conservative way:
Sending messages from process A (EULA or any license) to process B (GPL licensed), embracing 100% separation
I guess that's doable,
Thanks for @ianlancetaylor 's help and other folks for their input.

@rsc Is there any issue or discussion to track about your -o proposal here?

I've been working on plugin loading and splitting components cleanly (and automatically) across multiple IPC-connected processes, the code is available here: controller-bus.
It could sure use some improvement on the c-shared plugin loading side, (although I know that's unrelated to this issue)

After reading this whole thread it's not clear to me about -buildmode=shared 's future. Can I summarize it as:

plugin and c-shared will stay.
shared will be limited to use cases as below only:

go build -buildmode=shared -o libgo1.17.2.so std golang.org/x/...
go build -linkshared=libgo1.17.2.so cmd1
go build -linkshared=libgo1.17.2.so cmd2
go build -linkshared=libgo1.17.2.so cmd3

@laoshaw There has not been a final decision but what you describe is likely.  More precisely, it's likely that we will only permit a single -buildmode=shared library to be included in a build; it doesn't necessarily have to be the Go standard library although that will presumably be the common case.

Just wanted to check on the status of this. Since the issue is closed, is there another issue/location where I can follow the evolution of this? I don't see it explicitly in the 1.19 beta 1 change log so I'm guessing at best it'll be for 1.20?

@mlaventure I think the status is that we developed an idea (a single shared library) but stalled out on implementing it.  Nothing has changed yet and as far as I know there is no new proposal.

@mlaventure I don't know of a place to track progress.  Sorry.
You should change to a different language if it makes sense for you, of course, but I don't understand the comment.  The existing -buildmode=shared continues to work for now.

@mlaventure I don't know of a place to track progress. Sorry.
You should change to a different language if it makes sense for you, of course, but I don't understand the comment. The existing -buildmode=shared continues to work for now.

Unfortunately it doesn't work for me, the compiler panics when I try to build a package with -buildmode=shared -linkshared:
panic: runtime error: makeslice: cap out of range

goroutine 1 [running]:
cmd/link/internal/loader.(*Loader).LoadSyms(0xc00079a000, 0x670a56)
	/usr/local/go/src/cmd/link/internal/loader/loader.go:2210 +0x9f
cmd/link/internal/ld.(*Link).loadlib(0xc0000da000)
	/usr/local/go/src/cmd/link/internal/ld/lib.go:577 +0x46e
cmd/link/internal/ld.Main(_, {0x20, 0x20, 0x1, 0x7, 0x10, 0x0, {0x0, 0x0}, {0x679c5b, ...}, ...})
	/usr/local/go/src/cmd/link/internal/ld/main.go:249 +0xd8b
main.main()
	/usr/local/go/src/cmd/link/main.go:69 +0x1005

This is using go 1.17.11.

You wouldn't normally use both -buildmode=shared and -linkshared together.  Anyhow perhaps that should be a different issues if there isn't already one open for it.  Thanks.

That is #47455, which could perhaps be reopened — but, to be honest, I would be surprised if it gets much debugging effort until we figure out the future of -buildmode=shared.

You wouldn't normally use both -buildmode=shared and -linkshared together. Anyhow perhaps that should be a different issues if there isn't already one open for it. Thanks.

It used to be the only way I had found to build a shared library and avoid the /usr/local/go/pkg/tool/linux_amd64/link: cannot implicitly include runtime/cgo in a shared library  error. I think there was an issue for it around go 1.7, but the issue seems to still be there. The packages I am building do not depend on CGO.

I've been looking at -linkshared application building recently to see if it's a viable solution for Linux distribution packaging of Go software. The main workflow concept is this:

Distributions provide a -buildmode=shared package that includes libstd.so (in Fedora this is golang-shared).GOROOT=$(pwd) PATH=$(pwd)/bin:$PATH go install -buildmode=shared -v -x std

Applications are built with -linkshared to link against this global libstd.so for the Go standard library.

The idea behind this is to enable updates to Go within the distribution without requiring mass rebuilds of applications due to stdlib patches/fixes. It has the side benefit of creating smaller binaries, but the gains are dependent on other factors as well (e.g. minimal dependencies). However, I do not fully understand the Go build process and the way the compiler works to know if this is even a feasible goal.

As it stands right now this isn't possible because Go attempts to rebuild libstd.so on each build run, which doesn't lend confidence to a stable shared object being produced, and users building software as non-root generally do not have write access to the global GOROOT.
I've been told that it's possible for things to change between patch releases of Go (e.g. 1.19.0->1.19.1) that might introduce reliability issues in this process.
Dependencies using assembly may run into issues

For example, trying to build terraform:$ GOROOT=/tmp/golang go build -o ./bin/terraform -mod=vendor -linkshared .
# github.com/klauspost/compress/zstd/internal/xxhash
asm: xxhash_amd64.s:120: when dynamic linking, R15 is clobbered by a global variable access and is used here: 00092 (/home/mroche/Documents/code/tools/terraform/vendor/github.com/klauspost/compress/zstd/internal/xxhash/xxhash_amd64.s:120)ADDQ	R15, AX
asm: assembly failed

If my understanding is correct, this wouldn't remove the need to rebuild applications if there's a security fix to the compiler itself, only the stdlib.

Does this sound like a use case that could be (or maybe shouldn't be at all) supported by Go tooling?

@omenos, to my knowledge the Go compiler doesn't currently have any notion of “ABI boundaries”, and in particular it can do inlining of function bodies across packages. So there is a substantial risk of subtle bugs when picking up changes to the standard library and toolchain — particularly if a function fixed in a patch release was inlined into the caller when it was compiled.

Distributions might choose to do
go build -buildmode=shared -o libgo1.17.2.so std golang.org/x/...
go build -linkshared=libgo1.17.2.so cmd1
go build -linkshared=libgo1.17.2.so cmd2
go build -linkshared=libgo1.17.2.so cmd3

This sounds like a great solution. I know I'm late to the game and perhaps it's a bit much to ask ;) What would really help in my use case (single repo/module many separate applications) is a build command that can build multiple binaries in one go, and intelligently bundles all shared dependencies into a shared library.
So basically your solution, with automation so the risk of operator mistake is limited.

+1 for shared libraries support with module aware versions
I'm using plugins with their dependencies. And they are very fat. I just want to move the dependencies to a shared library. And I don't want to care about a compatibility etc.
Also, root installation is not acceptable for me too. Currently I'm using separate Go compiler (unlike /usr/lib/go).


getting-started-csharp-mac.md, compile success, deploy fails

Followed instructions in the documentation. avm file was successfully compiled with indicators:

Neo.Compiler.MSIL console app v<IP_ADDRESS>
Find entrypoint:System.Boolean Main.SmartContracts.AContract::Main(System.String,System.Byte[])
convert succ
gen abi succ
write:SmartContract.avm
write:SmartContract.abi.json
SUCC
Executing: mv SmartContract.avm  //neo-local/smart-contracts/SmartContract.avm

However when deploying to neo-local private blockchain via neo-python. Contract fails to execute. See below errors
[D 190911 23:25:33 ApplicationEngine:372] Exception calculating gas consumed 'utf-8' codec can't decode byte 0xcf in position 0: invalid continuation byte
[I 190911 23:25:33 EventHub:58] [test_mode][SmartContract.Execution.Fail] [b29948e9150d5364ab286e7e09049d7292618379] {'type': 'Array', 'value': [{'type': 'String', 'value': 'Execution exited in a faulted state. Any payload besides this message contained in this event is the contents of the EvaluationStack of the current script context.'}, {'type': 'ByteArray', 'value': b'No operation ran'}]}

Found issue was related to dotnet Neo compiler command in documentation:
dotnet neon.dll <your smart contract.dll>
Contract succesfully deployed with additional compatible option i.e.
dotnet neon.dll <your smart contract.dll> --compatible

Pull request available for documentation update to fix this issue


Time formats broken for rails 3.1

We should now add them to the Time::DATE_FORMATs hash instead.

Resolved


Implement Project Task Assignment page

Project task assignment (copy Introduction.js and edit as needed).
Look into changing the axios calls to work with the project task assignment page instead of the introduction.
Will probably require some looking into what data is being passed in, etc.

Project Task Assignment page is set up, the axios calls are not yet present. Need to consult with someone as to what calls to the database are required to make them work.


neo4j: 3.1.2 -> 3.3.2

Semi-automatic update. These checks were performed:

built on NixOS
ran /nix/store/h7gmk2gjradg84isbn8hpb05cirpkind-neo4j-3.3.2/bin/neo4j help got 0 exit code
ran /nix/store/h7gmk2gjradg84isbn8hpb05cirpkind-neo4j-3.3.2/bin/neo4j --version and found version 3.3.2
ran /nix/store/h7gmk2gjradg84isbn8hpb05cirpkind-neo4j-3.3.2/bin/neo4j version and found version 3.3.2
ran /nix/store/h7gmk2gjradg84isbn8hpb05cirpkind-neo4j-3.3.2/bin/neo4j help and found version 3.3.2
ran /nix/store/h7gmk2gjradg84isbn8hpb05cirpkind-neo4j-3.3.2/bin/neo4j-admin help got 0 exit code
ran /nix/store/h7gmk2gjradg84isbn8hpb05cirpkind-neo4j-3.3.2/bin/neo4j-admin --version and found version 3.3.2
ran /nix/store/h7gmk2gjradg84isbn8hpb05cirpkind-neo4j-3.3.2/bin/neo4j-import --help got 0 exit code
ran /nix/store/h7gmk2gjradg84isbn8hpb05cirpkind-neo4j-3.3.2/bin/neo4j-import help got 0 exit code
ran /nix/store/h7gmk2gjradg84isbn8hpb05cirpkind-neo4j-3.3.2/bin/neo4j-shell -h got 0 exit code
ran /nix/store/h7gmk2gjradg84isbn8hpb05cirpkind-neo4j-3.3.2/bin/neo4j-shell --help got 0 exit code
ran /nix/store/h7gmk2gjradg84isbn8hpb05cirpkind-neo4j-3.3.2/bin/neo4j-shell --version and found version 3.3.2
found 3.3.2 with grep in /nix/store/h7gmk2gjradg84isbn8hpb05cirpkind-neo4j-3.3.2
found 3.3.2 in filename of file in /nix/store/h7gmk2gjradg84isbn8hpb05cirpkind-neo4j-3.3.2

cc "@offline"

Success on x86_64-darwin (full log)
Partial log (click to expand)
post-installation fixup
strip is /nix/store/4sdh09gmvl15cy0zb6i7mbvxh5syz206-cctools-binutils-darwin/bin/strip
stripping (with command strip and flags -S) in /nix/store/vmm51lp8akaz7y0yjfp7cwwaw4nndd13-neo4j-3.3.2/bin
patching script interpreter paths in /nix/store/vmm51lp8akaz7y0yjfp7cwwaw4nndd13-neo4j-3.3.2
/nix/store/vmm51lp8akaz7y0yjfp7cwwaw4nndd13-neo4j-3.3.2/share/neo4j/bin/cypher-shell: interpreter directive changed from "/bin/bash" to "/nix/store/pkjmwq7sqrvjg7cjiph6hq0khsmfl6p8-bash-4.4-p12/bin/bash"
/nix/store/vmm51lp8akaz7y0yjfp7cwwaw4nndd13-neo4j-3.3.2/share/neo4j/bin/neo4j: interpreter directive changed from "/usr/bin/env bash" to "/nix/store/pkjmwq7sqrvjg7cjiph6hq0khsmfl6p8-bash-4.4-p12/bin/bash"
/nix/store/vmm51lp8akaz7y0yjfp7cwwaw4nndd13-neo4j-3.3.2/share/neo4j/bin/neo4j-admin: interpreter directive changed from "/usr/bin/env bash" to "/nix/store/pkjmwq7sqrvjg7cjiph6hq0khsmfl6p8-bash-4.4-p12/bin/bash"
/nix/store/vmm51lp8akaz7y0yjfp7cwwaw4nndd13-neo4j-3.3.2/share/neo4j/bin/neo4j-import: interpreter directive changed from "/usr/bin/env bash" to "/nix/store/pkjmwq7sqrvjg7cjiph6hq0khsmfl6p8-bash-4.4-p12/bin/bash"
/nix/store/vmm51lp8akaz7y0yjfp7cwwaw4nndd13-neo4j-3.3.2/share/neo4j/bin/neo4j-shell: interpreter directive changed from "/usr/bin/env bash" to "/nix/store/pkjmwq7sqrvjg7cjiph6hq0khsmfl6p8-bash-4.4-p12/bin/bash"
/nix/store/vmm51lp8akaz7y0yjfp7cwwaw4nndd13-neo4j-3.3.2

Success on x86_64-linux (full log)
Partial log (click to expand)
strip is /nix/store/b0zlxla7dmy1iwc3g459rjznx59797xy-binutils-2.28.1/bin/strip
stripping (with command strip and flags -S) in /nix/store/h7gmk2gjradg84isbn8hpb05cirpkind-neo4j-3.3.2/bin
patching script interpreter paths in /nix/store/h7gmk2gjradg84isbn8hpb05cirpkind-neo4j-3.3.2
/nix/store/h7gmk2gjradg84isbn8hpb05cirpkind-neo4j-3.3.2/share/neo4j/bin/neo4j-admin: interpreter directive changed from "/usr/bin/env bash" to "/nix/store/zqh3l3lyw32q1ayb15bnvg9f24j5v2p0-bash-4.4-p12/bin/bash"
/nix/store/h7gmk2gjradg84isbn8hpb05cirpkind-neo4j-3.3.2/share/neo4j/bin/cypher-shell: interpreter directive changed from "/bin/bash" to "/nix/store/zqh3l3lyw32q1ayb15bnvg9f24j5v2p0-bash-4.4-p12/bin/bash"
/nix/store/h7gmk2gjradg84isbn8hpb05cirpkind-neo4j-3.3.2/share/neo4j/bin/neo4j-shell: interpreter directive changed from "/usr/bin/env bash" to "/nix/store/zqh3l3lyw32q1ayb15bnvg9f24j5v2p0-bash-4.4-p12/bin/bash"
/nix/store/h7gmk2gjradg84isbn8hpb05cirpkind-neo4j-3.3.2/share/neo4j/bin/neo4j-import: interpreter directive changed from "/usr/bin/env bash" to "/nix/store/zqh3l3lyw32q1ayb15bnvg9f24j5v2p0-bash-4.4-p12/bin/bash"
/nix/store/h7gmk2gjradg84isbn8hpb05cirpkind-neo4j-3.3.2/share/neo4j/bin/neo4j: interpreter directive changed from "/usr/bin/env bash" to "/nix/store/zqh3l3lyw32q1ayb15bnvg9f24j5v2p0-bash-4.4-p12/bin/bash"
checking for references to /tmp/nix-build-neo4j-3.3.2.drv-0 in /nix/store/h7gmk2gjradg84isbn8hpb05cirpkind-neo4j-3.3.2...
/nix/store/h7gmk2gjradg84isbn8hpb05cirpkind-neo4j-3.3.2

Failure on aarch64-linux (full log)
Partial log (click to expand)
Package ‘oraclejre-8u161’ in /var/lib/gc-of-borg/nix-test-rs-1/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-1/pkgs/development/compilers/oraclejdk/jdk-linux-base.nix:71 has an unfree license (‘unfree’), refusing to evaluate.

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.


Address flattening issues

See https://docs.jsonata.org/processing#sequences

Ensure SingletonArrayTest passes

Opened a new issue for remaining flattening issues.


Not work with Internal Server Error.

Here's the error log.
[Error  :XUnity.AutoTranslator] Error occurred while retrieving translation.
System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
at System.Net.HttpWebRequest.CheckFinalStatus (System.Net.WebAsyncResult result) [0x00000] in :0
at System.Net.HttpWebRequest.SetResponseData (System.Net.WebConnectionData data) [0x00000] in :0
I tried GoogleTranslateV2 and PapagoTranslate, but both of translations not work with those errors.

Papago should be fixed and GoogleTranslateV2 should work properly.


Fix gas estimation

Fixes #1245.

@MaksymilianDemitraszek Do the release then. Btw. multicall could be included in the release, if the following PR wasn't blocked: https://github.com/software-mansion/protostar/pull/1224.


Improve documentation

We need to flesh out the README to include contribution guidelines, style guides, authors, licensing, etc

Merged in #21


Use embedded-hal traits

To make this lib easier to (re-)use, it should make usage of the traits provided by the embedded-hal crate. The current impls in the hal module are already very similar to them, so porting should not be a big problem.
The only thing that needs clarification is whether to use the single pin or the port wise traits?

First try on implementing the Delay trait from embedded-hal for clerk.
https://github.com/kunerd/f3-discovery-hd44780


Disable removal of native billing adpapter

[Version]
16.0.4
[How to reproduce]

Login as administrator
Select "Manage billing systems"
Add external billing
Set external billing as default
Delete Native Billing

[Expected result]

Delete button isn't active in Native Billing, or
Native Billing cannot be deleted

[Observed result]

Delete button is active in Native Billing, and
Native Billing can be deleted

The definition needs to be clarified. Currently, if native billing is set as 'not default' it can be deleted. This is also in accordance with the documentation.

Native billing must not be deleted.

Removing native billing is not a problem, since it can be restored. This is in accordance with the documentation.

Please write how to restore native billing in help page if this function will be decided.

I added a corresponding note to the online help (operator_manageBillingAdapters.xml). Will be available on 2016-06-02.


LCACHE misses for every native C function call

I have noticed everytime JS calls an external C function the LCACHE will miss.
Testing method:
I looped an external C call 10000 times and LCACHE missed exactly 20000 times. (MISS when ecma_lcache_lookup returns NULL)
This is quite a performance hit. Calling an empty external C function is 10% faster(77282 cycles vs 88476 cycles) with LCACHE completely disabled.
Can anyone confirm this?

Here's how I tested it:
I slightly modified the external function example:
#include <string.h>
#include "jerryscript.h"
#include "jerryscript-ext/handler.h"

/**
 * Add param to 'this.x'
 */
static jerry_value_t
add_handler (const jerry_value_t func_value, /**< function object */
             const jerry_value_t this_val, /**< this arg */
             const jerry_value_t *args_p, /**< function arguments */
             const jerry_length_t args_cnt) /**< number of function arguments */
{
  return jerry_create_undefined ();
} /* add_handler */

int
main (void)
{
  /* Initialize engine */
  jerry_init (JERRY_INIT_EMPTY);

  /* Register 'print' function from the extensions */
  jerryx_handler_register_global ((const jerry_char_t *) "print",
                                  jerryx_handler_print);

  /* Create a JS object */
  const jerry_char_t my_js_object[] = " \
    MyObject = \
    { x : 12, \
      y : 'Value of x is ', \
      foo: function () \
      { \
        return this.y + this.x; \
      } \
    } \
  ";

  jerry_value_t my_js_obj_val;

  /* Evaluate script */
  my_js_obj_val = jerry_eval (my_js_object,
                              strlen ((const char *) my_js_object),
                              false);

  /* Create a JS function object and wrap into a jerry value */
  jerry_value_t add_func_obj = jerry_create_external_function (add_handler);

  /* Set the native function as a property of previously created MyObject */
  jerry_value_t prop_name = jerry_create_string ((const jerry_char_t *) "add2x");
  jerry_set_property (my_js_obj_val, prop_name, add_func_obj);
  jerry_release_value (add_func_obj);
  jerry_release_value (prop_name);

  /* Free JavaScript value, returned by eval (my_js_object) */
  jerry_release_value (my_js_obj_val);

  const jerry_char_t script[] = "var i; for(i=0;i<100000;i++){MyObject.add2x ();}";
  size_t script_size = strlen ((const char *) script);

  /* Evaluate script */
  jerry_value_t eval_ret = jerry_eval (script, script_size, false);

  /* Free JavaScript value, returned by eval */
  jerry_release_value (eval_ret);

  /* Cleanup engine */
  jerry_cleanup ();

  return 0;
}

Now add a static counter to ecma_lcache_lookup() in jerry-core/ecma/base/ecma-lcache.c that increment everytime it returns NULL and log it out.
Mine test outputed:
200017 MISSES

There should be no connection between native functions and lcache. Lcache is designed to store property names, and native functions are callable objects.

The miss numbers is directly connected to the number of times the native function is called. I also tested this with a JS function which resulted in only 9 misses for 100000 calls.

I have narrowed down the lcache misses:

Both cache misses are trying to search for the same property, here are the stack traces:

#1  0x0000000000411ba4 in ecma_lcache_lookup (object_p=0x66a338 <jerry_global_heap+24>, prop_name_p=0x66a768 <jerry_global_heap+1096>) at jerryscript/jerry-core/ecma/base/ecma-lcache.c:219
#2  0x000000000044d312 in ecma_op_resolve_reference_value (lex_env_p=0x66a348 <jerry_global_heap+40>, name_p=0x66a768 <jerry_global_heap+1096>) at jerryscript/jerry-core/ecma/operations/ecma-reference.c:91
#3  0x000000000043fb3b in vm_loop (frame_ctx_p=0x7fffffffa100) at jerryscript/jerry-core/vm/vm.c:827
#4  0x0000000000443beb in vm_execute (frame_ctx_p=0x7fffffffa100, arg_p=0x7fffffffa2e0, arg_list_len=2) at jerryscript/jerry-core/vm/vm.c:2988
#5  0x0000000000443edb in vm_run (bytecode_header_p=0x66a748 <jerry_global_heap+1064>, this_binding_value=27, lex_env_p=0x66a348 <jerry_global_heap+40>, is_eval_code=false, arg_list_p=0x7fffffffa2e0, arg_list_len=2)
    at jerryscript/jerry-core/vm/vm.c:3081
#6  0x000000000042f165 in ecma_op_function_call (func_obj_p=0x66a7c8 <jerry_global_heap+1192>, this_arg_value=72, arguments_list_p=0x7fffffffa2e0, arguments_list_len=2)
    at jerryscript/jerry-core/ecma/operations/ecma-function-object.c:500
#7  0x0000000000403a33 in jerry_invoke_function (is_invoke_as_constructor=false, func_obj_val=1195, this_val=72, args_p=0x7fffffffa2e0, args_count=2) at jerryscript/jerry-core/api/jerry.c:2258
#8  0x0000000000403a80 in jerry_call_function (func_obj_val=1195, this_val=72, args_p=0x7fffffffa2e0, args_count=2) at jerryscript/jerry-core/api/jerry.c:2284

#1  0x0000000000411ba4 in ecma_lcache_lookup (object_p=0x66a338 <jerry_global_heap+24>, prop_name_p=0x66a768 <jerry_global_heap+1096>) at jerryscript/jerry-core/ecma/base/ecma-lcache.c:219
#2  0x0000000000408817 in ecma_find_named_property (obj_p=0x66a338 <jerry_global_heap+24>, name_p=0x66a768 <jerry_global_heap+1096>) at jerryscript/jerry-core/ecma/base/ecma-helpers.c:614
#3  0x000000000042ce4e in ecma_op_object_find_own (base_value=27, object_p=0x66a338 <jerry_global_heap+24>, property_name_p=0x66a768 <jerry_global_heap+1096>) at jerryscript/jerry-core/ecma/operations/ecma-objects.c:513
#4  0x000000000042d055 in ecma_op_object_find (object_p=0x66a338 <jerry_global_heap+24>, property_name_p=0x66a768 <jerry_global_heap+1096>) at jerryscript/jerry-core/ecma/operations/ecma-objects.c:615
#5  0x000000000044d3ac in ecma_op_resolve_reference_value (lex_env_p=0x66a348 <jerry_global_heap+40>, name_p=0x66a768 <jerry_global_heap+1096>) at jerryscript/jerry-core/ecma/operations/ecma-reference.c:115
#6  0x000000000043fb3b in vm_loop (frame_ctx_p=0x7fffffffa100) at jerryscript/jerry-core/vm/vm.c:827
#7  0x0000000000443beb in vm_execute (frame_ctx_p=0x7fffffffa100, arg_p=0x7fffffffa2e0, arg_list_len=2) at jerryscript/jerry-core/vm/vm.c:2988
#8  0x0000000000443edb in vm_run (bytecode_header_p=0x66a748 <jerry_global_heap+1064>, this_binding_value=27, lex_env_p=0x66a348 <jerry_global_heap+40>, is_eval_code=false, arg_list_p=0x7fffffffa2e0, arg_list_len=2)
    at jerryscript/jerry-core/vm/vm.c:3081
#9  0x000000000042f165 in ecma_op_function_call (func_obj_p=0x66a7c8 <jerry_global_heap+1192>, this_arg_value=72, arguments_list_p=0x7fffffffa2e0, arguments_list_len=2)
    at jerryscript/jerry-core/ecma/operations/ecma-function-object.c:500
#10 0x0000000000403a33 in jerry_invoke_function (is_invoke_as_constructor=false, func_obj_val=1195, this_val=72, args_p=0x7fffffffa2e0, args_count=2) at jerryscript/jerry-core/api/jerry.c:2258
#11 0x0000000000403a80 in jerry_call_function (func_obj_val=1195, this_val=72, args_p=0x7fffffffa2e0, args_count=2) at jerryscript/jerry-core/api/jerry.c:2284

the opcode and operands that leads to the caches misses are 42 and 640.
In addition I tested with print ext-handler and confirmed that it also encounter 2 cache misses per call.

@jack9766 hi,
I've looked into this. The problem comes from here:
https://github.com/jerryscript-project/jerryscript/blob/efa88507833f554bcf2dc1f901e3549529bb99da/jerry-core/ecma/base/ecma-lcache.c#L204
To make it work you need to add the external function names to the magic string list. See: http://jerryscript.net/api-reference/#jerry_register_magic_strings

Thank you for the analysis. I think we can close this now. Feel free to reopen if needed.


ImportError: cannot import name 'int_classes' from 'torch._six' (Future Torch 1.9+ Compatibility Fix Required)

Issue description
Current behavior

If we execute ivadomed training on Torch 1.9+ we run into this particular import error, mainly because most existing code bases are only tested with Torch 1.8.X.

A potential solution has been this as a quick fix to make the code base support Torch 1.9.x.

Expected behavior

Expected a successful run but instead received a ImportError: cannot import name 'int_classes' from 'torch._six'  error on the Mac Monterey Platform with Torch 1.10

Steps to reproduce

Execute ivadomed -c config.json --train --path-data data_example_spinegeneric --path-output testRun on a Torch 1.9+ environment with the existing master branch as of 2022-03-13

 An example summary message
```
Traceback (most recent call last):
  File "/Users/en_tetteh/miniforge3/envs/venv_ivadomed/bin/ivadomed", line 33, in <module>
    sys.exit(load_entry_point('ivadomed', 'console_scripts', 'ivadomed')())
  File "/Users/en_tetteh/miniforge3/envs/venv_ivadomed/bin/ivadomed", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/Users/en_tetteh/miniforge3/envs/venv_ivadomed/lib/python3.8/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/Users/en_tetteh/miniforge3/envs/venv_ivadomed/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/en_tetteh/NEUROPOLY/ivadomed/ivadomed/main.py", line 14, in <module>
    from ivadomed import evaluation as imed_evaluation
  File "/Users/en_tetteh/NEUROPOLY/ivadomed/ivadomed/evaluation.py", line 9, in <module>
    from ivadomed import inference as imed_inference
  File "/Users/en_tetteh/NEUROPOLY/ivadomed/ivadomed/inference.py", line 15, in <module>
    from ivadomed.loader.mri3d_subvolume_segmentation_dataset import MRI3DSubVolumeSegmentationDataset
  File "/Users/en_tetteh/NEUROPOLY/ivadomed/ivadomed/loader/mri3d_subvolume_segmentation_dataset.py", line 8, in <module>
    from ivadomed import transforms as imed_transforms, postprocessing as imed_postpro
  File "/Users/en_tetteh/NEUROPOLY/ivadomed/ivadomed/transforms.py", line 19, in <module>
    from ivadomed.loader import utils as imed_loader_utils
  File "/Users/en_tetteh/NEUROPOLY/ivadomed/ivadomed/loader/utils.py", line 11, in <module>
    from torch._six import string_classes, int_classes
ImportError: cannot import name 'int_classes' from 'torch._six' (/Users/en_tetteh/miniforge3/envs/venv_ivadomed/lib/python3.8/site-packages/torch/_six.py)
```

Environment
System description

macOS Monterey - version 12.2.1 (21D62) with torch 1.10+

 Output of pip freeze
absl-py==1.0.0
astor==0.8.1
attrs==21.4.0
bids-neuropoly==0.2
bids-validator==1.9.2
cached-property @ file:///home/conda/feedstock_root/build_artifacts/cached_property_1615209429212/work
cachetools==5.0.0
certifi==2021.10.8
charset-normalizer==2.0.12
click==8.0.4
csv-diff==1.1
cycler==0.11.0
Deprecated==1.2.13
dictdiffer==0.9.0
docopt==0.6.2
fonttools==4.29.1
formulaic==0.2.4
google-auth==2.6.0
google-auth-oauthlib==0.4.6
grpcio @ file:///Users/runner/miniforge3/conda-bld/grpcio_1645230394268/work
h5py @ file:///Users/runner/miniforge3/conda-bld/h5py_1637964070496/work
humanize==4.0.0
idna==3.3
imageio==2.16.1
importlib-metadata==4.11.2
iniconfig==1.1.1
interface-meta==1.2.4
-e git+https://github.com/etetteh/ivadomed.git@8d1e36368a5e6ccceb1fd82f56e4b7b15ddd4bdf#egg=ivadomed
joblib==1.1.0
kiwisolver==1.3.2
loguru @ file:///Users/runner/miniforge3/conda-bld/loguru_1635858978966/work
Markdown==3.3.6
matplotlib==3.5.1
networkx==2.7
nibabel==3.2.2
num2words==0.5.10
numpy @ file:///Users/runner/miniforge3/conda-bld/numpy_1643958864831/work
oauthlib==3.2.0
packaging==21.3
pandas==1.4.1
Pillow==9.0.1
pluggy==1.0.0
protobuf==3.19.4
py==1.11.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pybids==0.14.0
pyparsing @ file:///home/conda/feedstock_root/build_artifacts/pyparsing_1635267989520/work
pytest==7.0.1
python-dateutil==2.8.2
pytz==2021.3
PyWavelets==1.2.0
requests==2.27.1
requests-oauthlib==1.3.1
rsa==4.8
scikit-image==0.19.2
scikit-learn==1.0.2
scipy==1.8.0
seaborn==0.11.2
SimpleITK==2.1.1
six @ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work
SQLAlchemy==1.3.24
tensorboard==2.8.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
threadpoolctl==3.1.0
tifffile==2022.2.9
tomli==2.0.1
torch==1.10.2
torchio==0.18.74
torchvision==0.11.3
tqdm==4.63.0
typing_extensions==4.1.1
urllib3==1.26.8
Werkzeug==2.0.3
wrapt==1.13.3
zipp==3.7.0

Discussion

Per PyTorch  forum discussion here, it seems that IF the environment is Torch==1.9+ (as evident by the pip freeze here showing torch==1.10.2, and the thread linked reporting 1.9), while the codebase of ivadomed is mostly tested against Torch 1.8, it will result in this error. This is not a huge surprise (as ivadomed specifically requires 1.8).

However, the purpose of this issue is to bring people's attention to this necessary change in the future event when we need to * upgrade to Torch 1.9+, that we need to make the necessary change recommended in the thread which is simply:

Other resources showing the similar problem:

https://stackoverflow.com/a/69170538
https://blog.csdn.net/u010684651/article/details/121018127 (solution shown in English as the last few lines)

@etetteh can you please indicate why this issue was closed? has it been resolved? if so how?

The issue is due to the usage of a version of torch higher than what is required for the current ivadomed version.  The issue is not resolved, but not relevant at this time. However, it serves as a guide for the amendment of ivadomed codebase.

@etetteh I have updated your issue a bit to provide a broader context and how and when it will become important. I agree the fix suggested seems rather minor and should not have major consequences so we can open a PR and patch this out to pave the way fur Torch 1.9+ adoption in the near future.
BTW, @etetteh is there any particular reason you used torch 1.10 instead of the 1.8 recommended? Was it just for purely experimental reason to leverage the latest PyTorch 1.10 features? or was it a hardware requirement?

The use of torch==1.10 was purely for experimentation. Specifically, the 1.10 version has new lr schedulers that are currently being used to update torchvision models, which I was experimenting with.
I think you can raise the priority so we incorporate it ASAP, before "it slips through the cracks"

Also, I would like to know if there is a particular reason, why we are using torch version 1.8. Based on my experimentation so far, everything runs fine with 1.10

The only reason was for reproducibility and validated performance and we did a thorough testing to ensure 1.8 is working okay at the time so we locked to that version. If we do the same for 1.10 and incorporate the latest changes, I think that is not a problem. After all, the jump from 1.5 to 1.8 was bigger than 1.9/1.10 I believe.

Closed by #1104. @etetteh please ensure to mention the source issue with resolves attribute to auto close the issue PR was related. This is another important reason to examine the PR template. It tackles all these details. :)
See all keywords available in the github context here: https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue


Use wildcard certificate to expose kube-apiservers of Shoot with trusted certificate

How to categorize this issue?

/area networking usability
/kind enhancement
What would you like to be added:
Operators can already provide a wildcard certificate for the seed cluster's ingress domain which is used to expose the observability components with a trusted certificate. This functionality is undocumented.
The gardenlet's Shoot controller copies the wildcard certificate secret into the shoot namespaces: https://github.com/gardener/gardener/blob/759c2641dab95cba6653924e3462aeec2e1fe56d/pkg/operation/botanist/secrets.go#L309-L335
When constructing the Ingress resources for the observability components, this secret is used in spec.tls[].secretName: https://github.com/gardener/gardener/blob/759c2641dab95cba6653924e3462aeec2e1fe56d/pkg/operation/botanist/monitoring.go#L164-L180
https://github.com/gardener/gardener/blob/759c2641dab95cba6653924e3462aeec2e1fe56d/charts/seed-monitoring/charts/alertmanager/templates/ingress.yaml#L17-L21
Let's enhance gardenlet to also create an Ingress resource for the kube-apiserver of shoots with a subdomain of the seed's ingress domain (similar like it's done for the observability components). Use this wildcard certificate secret to expose it. In order for this to work, kube-apiserver must be additionally configured with --tls-sni-cert-key.
From https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/:

A pair of x509 certificate and private key file paths, optionally suffixed with a list of domain patterns which are fully qualified domain names, possibly with prefixed wildcard segments. The domain patterns also allow IP addresses, but IPs should only be used if the apiserver has visibility to the IP address requested by a client. If no domain patterns are provided, the names of the certificate are extracted. Non-wildcard matches trump over wildcard matches, explicit domain patterns trump over extracted names. For multiple key/certificate pairs, use the --tls-sni-cert-key multiple times. Examples: "example.crt,example.key" or "foo.crt,foo.key:*.foo.com,foo.com".

Why is this needed:
The https://github.com/gardener/terminal-controller-manager could benefit from it since it currently sets this up manually.

/cc @petersutter

⚠ Let's finish https://github.com/gardener/gardener/issues/7232 first before we work on this one.

TBH, I still don't like this request to use a "commonly trusted" certificate for the API server.
Not from a security point of view and not from a kubernetes point of view.
In my opinion, any component wishing to establish trust with any shoot API server should first obtain the corresponding CA bundle from a trusted origin (e.g. the garden cluster). It should then verify the presented server certificate with the CA bundle.
I would consider this protocol for establishing trust even more secure than blindly trusting anybody with a "commonly trusted" server certificate because you establish that this is indeed the authority you want to connect to.
That being said...

Let's enhance gardenlet to also create an Ingress resource for the kube-apiserver of shoots with a subdomain of the seed's ingress domain

I'm concerned about adding an additional path for accessing API servers next to the common path via the istio ingress gateway.
Adding a new path means mechanisms like ACL need to protect all paths to be effective, which is error-prone and hence increases the security risk.
This probably doesn't affect any mechanisms in g/g. However, I'm afraid such an architectural decision might be troublesome in the long run.
Because of this, I vote for making a distinction between the wildcard certificate used for Ingress objects (observability endpoints) and the wildcard certificate for the API server ingress.
Operators could potentially use the same certificate. But it should be possible to use the currently existing ingresses without the side-effect of enabling the additional API server ingress.
If this feature is purely optional, I'm fine with having it in the codebase.

The feature is optional, just like it is today for the observability ingresses. If you don't provide a wildcard certificate secret as operator then nothing will happen. Also, my understanding is that istio-ingressgateway (and thus, the same "path") will be reused once we have tackled #7232. It's just another endpoint with a trusted certificate instead of the self-signed one.

The feature is optional, just like it is today for the observability ingresses. If you don't provide a wildcard certificate secret as operator then nothing will happen

Great, that's good news!

Also, my understanding is that istio-ingressgateway (and thus, the same "path") will be reused once we have tackled #7232. It's just another endpoint with a trusted certificate instead of the self-signed one.

I'm not sure if this is the case. I assume there will be different envoy filter chains, etc. for the existing API server access and the Ingress resources. But I'm not very deep into the topic.
Maybe it makes sense to rely on istio's built-in resources instead of the Ingress API to benefit from the same feature set and most importantly consistency.

https://github.com/gardener/gardener/pull/7518 is the first step. What is missing is the integration into gardenlets shoot controller (concretely, the instrumentation of the kubeapiserver package via the botanist) as described above.
If anybody is interested to implement the second part, feel free to pick it up :)

/assign


can you provide a citation?

I have not written a paper, so I have nothing to cite.

Close this issue. Please contact me if you have any further problems.


DBZ-3 Implements a Debezium Connector for ingesting Postgresql changes via logical decoding

Prerequisites

PostgreSQL >= 9.4 configured to allow logical replication
Debezium postgres-decoderbufs plugin installed in the server

Known issues and limitations

no DDL support - this is a limitation of Postgres' logical decoding feature, which does not provide this information
no automatic failover support - neither warm nor hot standby replicas support logical decoding slots. In other words, logical decoding can only be done from the master server. In case of a cluster failure, after a new master has been promoted, the connector has to be pointed to the new node and restarted
current implementation has only been tested on UX OSes. When dealing with Windows, the plugin has to be(most likely) natively complied on Windows and installed in the Windows Postgres server

Features and description
To be able to perform logical replication, the connector relies on a feature of the Postgres JDBC Driver which has not yet been released. As such, the current PR incorporates and uses a version of the JDBC driver where this feature is present. Once this is finally released, we'll want to update the connector and use the official driver.
Similar to the MySQL connector, this connector supports both taking snapshots and streaming incoming changes. With the exception of the DDL related parts, the PostgreSQL connector supports pretty much the same configuration options as the MySQL connector. It also generates events using the same structure and topics as the MySQL connector.
One significant difference from the MySQL connector is about how "multiple databases" are handled: in MySQL one can actually have different database instances and the MySQL connector reflects this for example when it comes to filters. In PostgreSQL however, the preferred way of doing this is using different schemas within the same server and physical database. As such, this connector reflects this distinction and updates some of the debezium-core classes to handle multiple schemas.
Also, since PostgreSQL has different custom datatypes than MySQL, this PR adds some more semantic types to Debezium which are used to represent this information: io.debezium.data.geometry.Point, io.debezium.data.Xml and io.debezium.data.Uuid
The testing and development for this connector has been done using Debezium's PostgreSQL docker image which is a vanilla Postgres 9.6 Debian installation with the previously mentioned plugin installed.

@rhauch : Travis fails atm before even getting to the Postgresql connector with:
[ERROR] Failed to execute goal io.fabric8:docker-maven-plugin:0.18.1:start (start) on project debezium-connector-mysql: I/O Error: Unable to create container for [debezium/mysql-server-test-database]: Conflict. The name "/database" is already in use by container 23791d3d4a129b7054d44c4929f4b40c529ef7234f6fa11565f9d1f91622c421. You have to remove (or rename) that container to be able to reuse that name. (Conflict: 409) -> [Help 1]
Not sure why that is.....

Yes, that is the same error I was getting when I upgraded the Docker Maven plugin before changing the use of docker.image to docker.filter. The symptom is the build incorrectly starts all the defined Docker containers used in the assembly profile but without the different ports defined in the assembly configuration. Turns out we were misusing docker.image to control which image(s) should be started, which worked only because of a bug in the Maven plugin which was corrected recently.  The correct approach is to instead use docker.filter for the purpose we need, so #143 made this change in addition to the plugin upgrade.
However, if you rebased with latest on master then I'm not sure why it fails on this brach but not on master.

yes, the PR is rebased on the latest master (i.e. post plugin update) and so the PG connector uses docker.filter instead of docker.image

@rhauch: I ran the build locally a number of times and it worked fine. Is there anything else which can be updated in this PR to try avoiding the Docker failure ?
Also, are the names of the containers started by the Maven plugin relevant ? If not, I suspect changing the <namingStrategy> to none might fix this particular problem. There is also a <restartPolicy> which might influence the behavior if a container already exists

@hchiorean No, the names of the containers are not important to the tests, since they simply use the localhost and the correct port. I won't have time to check the semantics of those options until later this afternoon -- any chance you can summarize them or try them locally?

added an additional commit which updates the build system to only use alias naming when required. In other words, alias is naming is still used when docker containers reference other containers via links elements. In all other cases the Docker Maven plugin will use a random name for each container.

well...the current failure is Error starting userland proxy: listen tcp <IP_ADDRESS>:3306: bind: address already in use

@hchiorean right, which is strange. Actually, the build is first running
mvn install -DskipTests=true -Dmaven.javadoc.skip=true -B -V

before it (usually) runs another build with tests, but for some reason this first pass seems to be starting the container. We need to either change this initial run to exclude integration tests, or we should override Travis' default approach of running the build in two passes.

@hchiorean I think I fixed the build issues. Do you want to rebase?

@rhauch: done

@rhauch: I've made similar change to the PG connector Maven docker config, but the latest build failure looks like
[ERROR] DOCKER> [debezium/mongo-initiator:3.2]: Timeout after 30008 ms while waiting on log out 'Replica set is ready'
[ERROR] DOCKER> Error occurred during container startup, shutting down...

at this point Travis is looking more like a hindrance to me....

amended the commit again, changing the default PG port to 4532 to see if this works (assuming it gets past the MongoDB hang)

nope, Mongo still fails...

@hchiorean I'll try to fix that, but how about moving the PostgreSQL connector before the MongoDB connector. Be sure that the build is using a different non-standard port for PostgreSQL.

rebased on latest master and moved the PG connector module before the Mongo module

@hchiorean, do you want to rebase this on master and adjust accordingly? I plan to release 0.3.6 on Dec 21, and it'd be great to release 0.4 before the holidays, too. But that'd mean we need documentation for the PostgreSQL connector. Feel free to create a PR on the website repository and update the PostgreSQL Connector page.
BTW, the changes we talked about in our previous conversation (see #142 for details) were more difficult to get working, so I think it's better if we just plan on trying again to refactor the codebase at a later time (perhaps once we've stopped backporting changes to the 0.3.x branch).

rebased on latest master.
Added a new commit with a significant change: since the Postgres JDBC driver does SNAPSHOT releases, this commit removes all the local sources for the PG driver replacing them with a 42.0.0-SNAPSHOT artifact that contains all the streaming changes.
Moving to a released version will be handled once that's available as part of https://issues.jboss.org/browse/DBZ-158

added a new commit with README and NOTES files for the postgres-connector module

rebased on latest master and fixed an issue around topic naming


Add Step Functions

I want to confirm I didn't miss something. I recently gave a talk on Step Functions after using them for a project. I'd be happy to kick start this if it's actually missing.

Yes, indeed it is not a service we yet cover at all.
Would be great if you'd like to get a first draft out! (Want to link your talk here?) Best way is just to start with a Quip doc that has similar structure to an existing section and then discuss/improve in Slack.
See https://github.com/open-guides/og-aws/blob/master/CONTRIBUTING.md#please-help and def join us in Slack! @QuinnyPig @ThanosBaskous and I or other editors can give you feedback as you go.

Awesome, will do.

I had thought maybe it already existed only to find the links are dead.
Is it common to pre-link before seeing an article? I ask since then I have an idea what is missing and what is not.

Thanks @gliptak


Optional items json! Object and Array

Is something like the following — optionally including entire items in the json! macro — possible, or if not, would it be possible to add it to serde_json?
let include_a = true;
let include_z = false;
let value = json!({
    if include_a {
        "a": 1
    },
    "array": [
        "x",
        "y",
        if include_z {
            "z"
        }
})

Which would produce the Value equivalent to { "a: 1, "array": ["x", "y"] } (but would be dynamic based on the values of include_a and include_z).

In terms of syntax/parsing this seems like a minefield, so maybe such expressions would have to be wrapped in parentheses (which can't appear in JSON) in order to enter “Rust mode” in json!.

I prefer not to support this. Please write:
if include_a {
    value["a"] = json!(1);
}


Not all classes supports headers options and recurring pay in registratoin get endpoint is missing

https://github.com/Mangopay/mangopay2-ruby-sdk/pull/200

Hello @andriiNyzhnik,
Thank you for your support. We a reviewing your PR.
Have a nice day,


Tests pass with unhandled rejections in node 14 and node 12

Tests do pass but unhandled rejections are there in console. You can see that here

then tests don't pass, right? ... interesting, no idea why I wouldn't see this in node 15, will check it, thanks.

found it ... neither node 14 nor 12 have Event interface, and the @ungap polyfill relies on a global document ... bummer

found that too but I was unsure what to do with ungap. Maybe just copy it in the source and refactor it to use a document parameter?

naa, I've dropped one dependency, which feels way better 🥳

😆  I like that commit message Poorlyfill


Update maintainer affiliation

Description
Updating my maintainer affiliation from ambassador to CoreWeave
Related Issues
n/a
Testing
Previewed the markdown
Checklist

[ ] Does my change need to be backported to a previous release?

What backport versions were discussed with the Maintainers in the Issue?

[ ] I made sure to update CHANGELOG.md.
Remember, the CHANGELOG needs to mention:

Any new features
Any changes to our included version of Envoy
Any non-backward-compatible changes
Any deprecations

[x] This is unlikely to impact how Ambassador performs at scale.
Remember, things that might have an impact at scale include:

Any significant changes in memory use that might require adjusting the memory limits
Any significant changes in CPU use that might require adjusting the CPU limits
Anything that might change how many replicas users should use
Changes that impact data-plane latency/scalability

[x] My change is adequately tested.
Remember when considering testing:

Your change needs to be specifically covered by tests.

Tests need to cover all the states where your change is relevant: for example, if you add a behavior that can be enabled or disabled, you'll need tests that cover the enabled case and tests that cover the disabled case. It's not sufficient just to test with the behavior enabled.

You also need to make sure that the entire area being changed has adequate test coverage.

If existing tests don't actually cover the entire area being changed, add tests.
This applies even for aspects of the area that you're not changing – check the test coverage, and improve it if needed!

We should lean on the bulk of code being covered by unit tests, but...
... an end-to-end test should cover the integration points

[ ] I updated DEVELOPING.md with any any special dev tricks I had to use to work on this code efficiently.

[ ] The changes in this PR have been reviewed for security concerns and adherence to security best practices.

Whoops, @ddymko, need the DCO here. 🙂

@kflynn ah I forgot about that! Let me address it

@kflynn fixed - I'll merge it unless you want to reapprove(?)

No need to wait. 🙂


Fails to run

Prerequisites

[X] Write a descriptive title.
[X] Make sure you are able to repro it on the latest version
[X] Search the existing issues.

Steps to reproduce

Clone repo: gh repo clone PowerShell/tree-sitter-PowerShell
Run step 1 from Getting started: npm install
Run step 2 from Getting started: npm start file.ps1
Try npm run generate
Try npm run parse file.ps1

Expected behavior
Not an error. Based on documentation, unsure of success.

Actual behavior
Error occurred.

><EMAIL_ADDRESS>start
> npm run generate && npm run parse

><EMAIL_ADDRESS>generate
> tree-sitter generate

[stdin]:189
      throw value
      ^

ReferenceError: Undefined symbol '_newline'
    at Object.get ([stdin]:210:23)
    at Proxy.class_definition (C:\Users\Kyle L. Davis\Development\tree-sitter-PowerShell\grammar.js:64:16)
    at grammar ([stdin]:283:42)
    at Object.<anonymous> (C:\Users\Kyle L. Davis\Development\tree-sitter-PowerShell\grammar.js:12:18)
    at Module._compile (node:internal/modules/cjs/loader:1112:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1166:10)
    at Module.load (node:internal/modules/cjs/loader:988:32)
    at Module._load (node:internal/modules/cjs/loader:834:12)
    at Module.require (node:internal/modules/cjs/loader:1012:19)
    at require (node:internal/modules/cjs/helpers:102:18) {
  symbol: { type: 'SYMBOL', name: '_newline' }
}

Node.js v18.4.0
Node process exited with status 1

### Error details

```console
`Get-Error` fails to capture error.

Environment data
Name                           Value
----                           -----
PSVersion                      5.1.19041.1682
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.19041.1682
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>

Version
7.2.5
Visuals
No response

This PR https://github.com/PowerShell/tree-sitter-PowerShell/pull/7 should also resolve this problem.
I hope it can be merged soon.


Fix "retain cycle" warnings in GRPCWrappedCall.m

In GRPCWrappedCall.m, three callbacks have warnings that say "Capturing 'self' strongly in this block is likely to lead to a retain cycle."

Weird that XCode didn't warn me! (Maybe it did and I was tired?)


Adding full screen button in Playground website in TextEditor section and in display panel

This might be a trivial thing but when I was using the play ground from my mobile I wanted to use a full screen button in both texteditor and display panel.

https://github.com/BabylonJS/Babylon.js/pull/2406


ec sign and verify example on manual page

Manual page ECDSA_SIG_new describes low-level elliptic curve digital signature algorithm (ECDSA) functions. Low-level are depecated in 3.0.
Issues:

examples are based on deprecated ECDSA_sign and ECDSA_verify;
search engine usually point to these pages.

What about to mark these samples as "historic" and first to refer to more up do date examples?

P.S. Reference to EVP_Digest{Sign|Verify}Init is "hidden", i.e. is easy reader to miss this part. So may be is better to remove esamples.

Rather than deleting the examples I'd prefer to split the man page into two, i.e. one to document the ECDSA_SIG_* functions which are not deprecated, and one to document the remaining deprecated functions. Hopefully then it should be clear that all the functions, including the examples, on the second page are deprecated.


Feature Request: Custom command after login or register

When a user joins do /login or /register after it was successful a custom command after authentication will execute.

Already in premium version


Export KO_FLAGS for consuming scripts

Changes

Export KO_FLAGS for consuming scripts

Per the Slack thread with Vincent.  If you prefer to do it globally. Otherwise we could also isolate the changes as needed into consuming ${knative-repo}/hack/release.sh.
/cc @dprotaso @upodroid @cardil

/hold for others to chime in

/cherry-pick release-1.16

/lgtm

/unhold