Add MFA support when profile assumes role

Describe the feature
Currently, the SDK regognizes the role_arn directive in profiles under the normal config file at ~/.aws/config. When sending a request, it will correctly attempt to assume the specified role. However, if that role requires an MFA token, the SDK will not request an MFA token from the user, and will fail to assume the role.
The feature request is to add support for MFA tokens, analogous to support in boto3 for the same.
Use Case
I use an IAM with limited permissions, which assumes a more privileged role, protected by MFA. The idea is to avoid having long-lived credentials present on my development laptop for a privileged account. When using the AWS CLI, or boto3 in a python script (run from a tty), I am prompted for my MFA key when first authenticating. I'd like to use the same workflow for programmes written against the Rust SDK.
To give a complete picture, my ~/.aws/config looks something like this (with long term credentials for the home profile in ~/.aws/credentials):
[profile home]
region=eu-west-2
role_arn=arn:aws:iam::<my-account>:role/MorePrivilegedRole
source_profile=home
mfa_serial=arn:aws:iam::<my-account>:mfa/jelford-laptop

Proposed Solution
To fit with the "Batteries included, but replaceable" design tenet, I think it would make sense to include:

a default implementation that requests an MFA token on stdin, if available
a plugable API to allow users to bring their own implementation for supplying MFA tokens

The general idea is to:

Load mfa_serial from profile config when present, onto aws_config::profile::credentials::repr::RoleArn
Add a trait ProvideMfaToken that can provide mfa tokens when required in aws_config::profile::credentials::exec::AssumeRoleProvider::credentials
Add an implementation of ProvideMfaToken used by default when stdin is a tty, that sources MFA tokens from stdin
Provide an easy hook on aws_config::default_provider::credentials::Builder to pass in a custom ProvideMfaToken

I'd propose usage looks a bit like this (using my "home" profile name from above), first for the default case:
// using default MFA token provider

let creds = credentials::Builder::default().profile_name("home").build();
let region = region::Builder::default().profile_name("home").build();

let config = aws_config::from_env()
    .credentials_provider(creds.await)
    .region(region)
    .load()
    .await;

let s3_client = aws_sdk_s3::Client::new(&config);
let buckets = s3_client.list_buckets().send().await.unwrap();

And for customizing the ProvideMfaTokenImplementation (assuming a future::ProvideMfaToken analogous to future::ProvideCredentials):
struct MyProvideMfaToken{}

impl ProvideMfaToken for MyProvideMfaToken {
    fn provide_mfa_token<'a>(&'a self) -> future::ProvideMfaToken<'a> ... {
        ...
   }
}
...

let creds = credentials::Builder::default().profile_name("home").mfa_token(MyProvideMfaToken{}).build();
let region = ...

let config = aws_config::from_env()
    .credentials_provider(creds.await)
    .region(region)
    .load()
    .await;

... etc

Other Information
For completeness, here's the detail from tracing when trying to use the profile above with an MFA token, following the proposed "default" code:
2022-05-01T12:11:12.105241Z  INFO send_operation{operation="ListBuckets" service="s3"}:provide_credentials{provider=default_chain}:lazy_load_credentials:load_credentials{provider=Profile}: aws_config::profile::credentials: constructed abstract provider from config file chain=ProfileChain { base: AccessKey(Credentials { provider_name: "ProfileFile", access_key_id: "<my-access-key>", secret_access_key: "** redacted **" }), chain: [RoleArn { role_arn: "arn:aws:iam::<my-account>:role/MorePrivilegedRole", external_id: None, session_name: None }] }
2022-05-01T12:11:12.105271Z  INFO send_operation{operation="ListBuckets" service="s3"}:provide_credentials{provider=default_chain}:lazy_load_credentials:load_credentials{provider=Profile}: aws_config::profile::credentials::exec: first credentials will be loaded from AccessKey(Credentials { provider_name: "ProfileFile", access_key_id: "<my-access-key>", secret_access_key: "** redacted **" }) base=AccessKey(Credentials { provider_name: "ProfileFile", access_key_id: "<my-access-key>", secret_access_key: "** redacted **" })
2022-05-01T12:11:12.105296Z  INFO send_operation{operation="ListBuckets" service="s3"}:provide_credentials{provider=default_chain}:lazy_load_credentials:load_credentials{provider=Profile}: aws_config::profile::credentials::exec: which will be used to assume a role role_arn=RoleArn { role_arn: "arn:aws:iam::<my-account>:role/MorePrivilegedRole", external_id: None, session_name: None }
2022-05-01T12:11:12.105337Z  INFO send_operation{operation="ListBuckets" service="s3"}:provide_credentials{provider=default_chain}:lazy_load_credentials:load_credentials{provider=Profile}: aws_config::profile::credentials: loaded base credentials creds=Credentials { provider_name: "ProfileFile", access_key_id: "<my-access-key>", secret_access_key: "** redacted **" }
2022-05-01T12:11:12.107050Z DEBUG send_operation{operation="ListBuckets" service="s3"}:provide_credentials{provider=default_chain}:lazy_load_credentials:load_credentials{provider=Profile}:load_assume_role{provider=AssumeRoleProvider { role_arn: "arn:aws:iam::<my-account>:role/MorePrivilegedRole", external_id: None, session_name: None }}: aws_endpoint: resolved endpoint endpoint=AwsEndpoint { endpoint: Endpoint { uri: https://sts.eu-west-2.amazonaws.com/, immutable: false }, credential_scope: CredentialScope { region: Some(SigningRegion("eu-west-2")), service: None } } base_region=Region("eu-west-2")
<-- snip: a bunch of http request preparation / sending stuff -->
2022-05-01T12:11:12.199119Z  WARN send_operation{operation="ListBuckets" service="s3"}:provide_credentials{provider=default_chain}:lazy_load_credentials:load_credentials{provider=Profile}: aws_config::profile::credentials: failed to load assume role credentials provider=AssumeRoleProvider { role_arn: "arn:aws:iam::<my-account>:role/MorePrivilegedRole", external_id: None, session_name: None }
2022-05-01T12:11:12.199167Z  WARN send_operation{operation="ListBuckets" service="s3"}:provide_credentials{provider=default_chain}:lazy_load_credentials: aws_config::meta::credentials::chain: provider failed to provide credentials provider=Profile error=An error occurred while loading credentials: An error occurred while loading credentials: Error { code: "AccessDenied", message: "User: arn:aws:iam::<my-account>:user/jelford-laptop is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::<my-account>:role/MorePrivilegedRole", request_id: "..." }

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ConstructionFailure(CredentialsLoadingError(ProviderError { cause: ProviderError { cause: ServiceError { err: AssumeRoleError { kind: Unhandled(Error { code: Some("AccessDenied"), message: Some("User: arn:aws:iam::<my-account>:user/jelford-laptop is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::<my-account>:role/MorePrivilegedRole"), request_id: Some("..."), extras: {} }), ...
<-- snip -->

Acknowledgements

[X] I may be able to implement this feature request
[ ] This feature might incur a breaking change

A note for the community

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue, please leave a comment

Thanks for submitting this well thought-out request/proposal! If you're interested in writing up an RFC or submitting a PR, we'd welcome it.
I think that adding a trait for MfaTokenProvider and the relevant config setters could be a small and quick update. With regards to implementing an MfaTokenProvider that receives input thru StdIn: I'd like to see an RFC on this that collects a bit of info on how other SDKs implemented this feature and what their user experience is like.

Sure, sounds sensible - I've actually started on a little PR so I'll post something on that probably later this week, covering the new trait / config setters.
I'm happy to put together an RFC - is there some documentation around the typical shape of these / what to expect to include? Sorry if I've missed it - I did a quick scan through CONTRIBUTING and the discussions here on GH and didn't see much prior art. No worries if not.
On the piont about other SDKs, so far I have been looking only at boto3, which has the behaviour I describe in OP - is there a sensible group, e.g. "boto + java", that taken together make for a reasonably canonical sample?

Sure, sounds sensible, and thanks for the quick response - I've actually started on a little PR so I'll post something on that probably later this week, covering the new trait / config setters.

I can't wait to see it

I'm happy to put together an RFC - is there some documentation around the typical shape of these / what to expect to include? Sorry if I've missed it - I did a quick scan through CONTRIBUTING and the discussions here on GH and didn't see much prior art. No worries if not.

We don't have a guide for writing RFCs but you can take a look at past ones here. In my opinion a good RFC will state the customer need, define terms, show what the user experience will look like for the new feature, and then dive into the implementation of the feature. We usually include a checklist at the end that summarizes what will be implemented.

On the point about other SDKs, so far I have been looking only at boto3, which has the behavior I describe in OP - is there a sensible group, e.g. "boto3 + java", that taken together make for a reasonably canonical sample?

The CLI/boto3 is generally our gold standard. We also often look to the Java, Javascript, Go, and Kotlin SDKs. The goal is that features should be relatively consistent across SDKs. Unsurprising code is easier to use and reason about.

Sounds good, thanks!

I've put up a preliminary PR for this: https://github.com/awslabs/smithy-rs/pull/1359
A few outstanding questions around the implementation, but as you said, not too big.


Blog: Windows HostProcess containers going to stable

Signed-off-by: Mark Rossetti<EMAIL_ADDRESS>

/sig windows
/milestone v1.26
/cc @brasmith-ms

Hello @marosset! v1.26 Comms shadow here. This feature blog is on a feature tracked for release, the deadline for submitting a draft is the 29th of November; this should be considered the hard limit since we will need to review/edit/discuss the draft, so if at all possible it's better to submit it earlier to avoid any problems.
Any doubts, we're here to help! cc @fsmunoz

Hello @marosset, we're doing a global reminder about submitting a draft for review for all opted-in feature blogs. If it's at all possible, it is very helpful for the release team to have drafts submitted for review before the hard deadline date, to better plan the release dates and avoid missing out. Thank you!

/lgtm
/approve

/assign @onlydole

oh wait, we need to remove the todos
/hold

Updated the TODOs, should be good for review now

Hello, Comms Lead for v1.26 here: this Feature Blog PR is confirmed for the post-release Feature Blog series. We will add the publishing date in the next days. Thank you for your work!

@brasmith-ms @jsturtevant @sftim - I reworded a lot of the intro with https://github.com/kubernetes/website/pull/37370/commits/1d467b22df32bb50e98a10b5b038ab30644ccecd PTAL

Holding until the release day.
/hold

/lgtm
/approve

Marking this Ready for Publishing in Comms tracking, thank you for all the work - minor edits/reviews are still doable until the release date.
/lgtm

I'm going to do another round of updates to incorporate all the feedback tomorrow.

Hi @marosset ,thanks for your work on this. It would be good to be able to merge this as soon as possible, especially considering that this is one of the very first blogs to go out and we don't have much time until the publication date.

/hold cancel
We have a brief opportunity to do updates before publication; otherwise, it's still OK to make corrections for up to 1 year afterwards. Beyond that time, we de-prioritise review effort and only really consider it if the article is obviously misleading.


Use cargo-make to build the packages

As mentioned in #33, here is a first pass at cargo make integration, Running cargo make in the workspace root will build every package with the appropriate target (it will also run rustfmt by default). cargo make can also be run in individual packages to build just that package. This only scratches the surface of  what the tool can do(e.g. it can also be used to install dependencies like flip-link, it can be used as part of your CI, etc. )
You will need to install cargo make to make use of the changes.
I also added a a more understandable compiler error if someone attempts to build fl16-inputmodules with mutually exclusive features enabled.

Thanks! This is pretty cool :)
Can it also work for running commands?
Currently it's quite cumbersome to run the inputmodule-control program:
# Linux
cargo run --target x86_64-unknown-linux-gnu -p inputmodule-control

# Windows
cargo run --target x86_64-pc-windows-msvc -p inputmodule-control

And can I apply the profile options listed here only to the firmware workspace members?
https://github.com/FrameworkComputer/inputmodule-rs/issues/19#issuecomment-1669220692

Running commands should also be fairly trivial, just add something to the effect of
[tasks.run]
command = "cargo"
args = [
    "run",
    "--target",
    "${CARGO_MAKE_RUST_TARGET_TRIPLE}",
]

in the inputmodule-control Makefile. You should then be able to run cargo make run in the inputmodule-control directory  or in the root with --cwd inputmodule-control (as above this is currently untested)

Okay, I'm happy with it, thanks a lot @jscatena88!

Cheers! I'm glad it was helpful


protoc-gen-go: support go_tag option to specify custom struct tags

Is it possible to define a message with custom tags? For example defining a json name that isn't just the lowercase field name or adding a bson tag.

No, there's no support for that, nor any intention to support that.

I'd like to add that this feature would be extremely useful for validation purposes.

+1
Use case: Using protnbuf generated structs along with sqlx package. Would be awesome

Likewise the go library for cloud datastore. Basically, increasing numbers of things make use of struct tags, and without support for them you're left manually duplicating structs? If there are no plans to add this functionality, what is the recommended way to handle it?

+1

+10086

I would love this feature!

I would like to have this feature aswell!

I would love this feature aswell!

This would be handy, as I need to validate JSON payloads for my REST API portion of the service.

Since golang/protobuf will probably never support this.  Here are some solutions.
https://github.com/mwitkow/go-proto-validators is useful for validation on proto structs.
https://github.com/gogo/protobuf can be used in two ways to have custom tags:

You can use the moretags extension
https://github.com/gogo/protobuf/blob/master/test/tags/tags.proto

Or you can use the typedecl extension if you don't want to generate the golang struct.

Need it as well to automate insert/update into the sql database

I've add a plugin retag for grpc.
https://github.com/qianlnk/protobuf/tree/master/protoc-gen-go/retag

Thanks @qianlnk. This really helped me.But i want to use timestamp in my proto file.When i try to use it like this
protoc -I src/gatewayapi/rpcdata/rpcdata --proto_path=src/github.com/google/protobuf/src/ --go_out=plugins=grpc+retag:. src/gatewayapi/rpcdata/rpcdata/api.proto
I m not getting the changes in the tag.May i know any way to achieve it.
Thanks

@qianlnk .thanks.I tried your plugin when i use only message then ther is no problem but when the message have sub message  like this
message SendData {
	message MetaData{
		string Name = 1;//`json:"name;omitempty"`
		int64 Length = 2;//`json:"name;omitempty"`
	}
	message Chunck{
	bytes Data = 1;//`json:"name;omitempty"`
	int64 Position = 2;//`json:"name;omitempty"`
	}
}

This throws the error.Any how the retag can support this???
Thanks...

@RajeshKumar1990 I fixed it.

@qianlnk I am having a problem making this work.... does it support proto3?
syntax = "proto3";
package contracts;

import "Common.proto";
//import "google/protobuf/timestamp.proto";

message Deposit {
  string id = 1;  // `db:"id"`
  string clientId = 2;
  double amount =  3;
  Fiat fiat = 4 ;
  string datetime = 5;
}

Converting with:
protoc -I=biteq.eu/contracts/proto --go_out=plugins=grpc+retag:biteq.eu/contracts/ biteq.eu/contracts/proto/*.proto
Result:
type Deposit struct {
	Id       string  `protobuf:"bytes,1,opt,name=id" json:"id,omitempty"`
	ClientId string  `protobuf:"bytes,2,opt,name=clientId" json:"clientId,omitempty"`
	Amount   float64 `protobuf:"fixed64,3,opt,name=amount" json:"amount,omitempty"`
	Fiat     Fiat    `protobuf:"varint,4,opt,name=fiat,enum=contracts.Fiat" json:"fiat,omitempty"`
	Datetime string  `protobuf:"bytes,5,opt,name=datetime" json:"datetime,omitempty"`
}

The db custom tag on id is missing....
Like everyone using proto, I need this in order to work with (any) orm:)
Thanks for your time

@atamgp the first letter in field must be capitalized.

@qianlnk thank you for the response. I changed the proto to:
string ID = 1;  //`db:"id,omitempty"
Like in the example, only i am using a non existing tag like (json or  xml).
Still no db output:
ID       string  protobuf:"bytes,1,opt,name=ID" json:"ID,omitempty"
If you have another idea please let me know. I am looking into the plugin code right now.
I tested that the code is called with a print statement in de init. So the environment seems ok.

@atamgp have you install my retag plugin?
git clone https://github.com/qianlnk/protobuf.git $GOPATH/src/github.com/golang/protobuf
go install $GOPATH/src/github.com/golang/protobuf/protoc-gen-go

this is my try:
syntax = "proto3";
package contracts;

//import "Common.proto";
//import "google/protobuf/timestamp.proto";

message ZZZ {
  string ID = 1;        //`db:"id"`
  string ClientID = 2;  //`db:"client_id"`
  string Datetime = 3;  //`db:"datetime"`
}

run:
protoc -I/usr/local/include -I. \
   -I$GOPATH/src \
   --go_out=plugins=grpc+retag:. \
test.proto

and result:
type ZZZ struct {
	ID       string ` protobuf:"bytes,1,opt,name=ID"         json:"ID,omitempty"  db:"id"`
	ClientID string ` protobuf:"bytes,2,opt,name=ClientID"   json:"ClientID,omitempty"  db:"client_id"`
	Datetime string ` protobuf:"bytes,3,opt,name=Datetime"   json:"Datetime,omitempty"  db:"datetime"`
}

OK, I have it figured out a bit more :)
message ZZZZ {
string ID = 1;        //db:"id"
string clientID = 2; //db:"id"
double amount =  3;
}
Things I learned:

I have to put the db tag on all fields of a message. Otherwise I get an exception:
panic: runtime error: index out of range (panic: runtime error: index out of range -> indexing [1])
For some fields it is not wanted to be mapped by an ORM. Is it possible to make this optional?
Like you said earlier, field name has to begin with a capital otherwise its ignored. Ofcourse its possible to misuse this aspect to realize optional fields (1) but is less clean.
Command path used. I have this folder structure:
$GOPATH/src/X/Y/Z/zzz.proto
I was using a protoc command from src folder which did not work:

Working: When calling from within Z: protoc  --go_out=plugins=grpc+retag:. zzz.proto
Not working: protoc  -I:X/Y/Z  --go_out=plugins=grpc+retag:. X/Y/Z/zzz.proto
Working: protoc -I:. -I:X/Y/Z  --go_out=plugins=grpc+retag:. X/Y/Z/zzz.proto
4) importing from other proto: I have a common.proto and a zzz.proto. zzz is importing an enum and a message from common.
working: if a use a working command from point 3) above for 1 file at a time e.g.:
protoc -I:. -I:X/Y/Z  --go_out=plugins=grpc+retag:. X/Y/Z/zzz.proto
protoc -I:. -I:X/Y/Z  --go_out=plugins=grpc+retag:. X/Y/Z/common.proto
nor working: same commands which are working for 1 file but in order to process both proto files /*.proto instead of a specific proto. E.g.: protoc -I:. -I:X/Y/Z  --go_out=plugins=grpc+retag:. X/Y/Z/*.proto
errors: ....  is already defined in file ....
It is really nice to process all proto files in 1 time, can you have a look at this?
Last: It is not clean to have all proto files and generated go files in the same folder.
In my case Z is actually named proto and Y is contracts.
I want to have my generated go files put in folder Y instead of Z. Still looking for a working combination command for this...

Has anyone found a solution for converting between type: string and type: bson.ObjectId as well?

Please support this it is extremely useful and cuts down on needless code bloat.

While this idea is super useful in Go, unfortunately no other language that proto supports can really make use of it. As such, changing/supporting this would be outside of the scope of the official Go protobuf package.

@puellanivis, that is not true.  Supporting custom tags would open protobufs to a plethora of possible tooling.

I'm going to re-open this issue, but it does not mean that we're going to support this. I'd like to see more discussion on the feasibility of this. Some thoughts:

I can see the usefulness of this feature for Go-heavy projects, but protobufs were designed a language-agnostic way to interchange data. Baking more Go-specific details in the proto files seems to be antithetical to the philosophy of protobufs.
This features makes the assumptions that generated Go protobufs are always structs with fields. What does it mean if we want the ability to generate opaque protobuf types that are interacted via methods. Performance testing has shown that techniques like lazy unmarshaling can bring significant speed benefits, but is only feasible as an opaque type.
The proper way to support this seems to be using protobuf options, but I don't see any proposal here suggesting a syntax for how that would work.

why not add a plugin like
https://github.com/qianlnk/protobuf/tree/master/protoc-gen-go/retag
This plugin is my original idea, don't take too much effort to attain it.But I believe that many people need it,  so I hope more people can join in.

@qianlnk Looked at your retag plugin as one of several alternatives. I don't really like the idea of cloning it into my golang/protobuf folder, especially since This branch is 8 commits ahead, 28 commits behind golang:master. Would you consider making it a standalone plugin instead?

@dsnet See tags.proto which was referred to by @awalterschulze as an example of using the protobuf options approach.
It could perhaps look something like this:
message Enrollment {
   CourseID uint64 = 1 [(go_tags) = "gorm:\"unique_index:uid_user_course\""];
   UserID uint64 = 2 [(go_tags) = "gorm:\"unique_index:uid_user_course\""];
   int32 foo = 3 [(go_tags) = "sql:\"type:text\""];
   string bar = 4;
}

This would be a very useful addition to avoid a lot of boilerplate, and IMO it would fit nicely in with other language-specific options, such as e.g. java_package to name just one. Just like java_package is ignored when generating Go files, go_tags would be ignored when generating non-Go code.
The only issue that I don't like with the above is the need for the \". It would be nice if that can be avoided, but it is a compromise I don't mind if we can get this feature into the main project.

@dsnet Regarding your point about opaque types. I think there are two main use cases that have come up in this issue.

Boilerplate avoidance for database/storage backends.
Validation of input fields.

I think the first is best dealt with using something like my proposed go_tags, because the tools that process the tags are their own beasts and it seems difficult to interact via lazy functions.
Validation of input fields seems different though. I can imagine that validation could be done as part of lazy unmarshaling, but that would require another option such as in go-proto-validators. However, such lazy unmarshaling with validation is not Go-specific, and so I don't think it makes sense to put such validation data into struct tags in the generated code. Indeed, go-proto-validators does not put validation data in struct tags.

IMO, it's better to provide a global option, for example go_tags with value like "bson" or "bson,xml".
It could perhaps look something like this:
syntax="proto3";

package user;
option go_package = "user";
option go_tags = "bson";

message User {
    string id   = 1;
    string name = 2;
}

Which generates:
type User struct {
    Id string `protobuf:"varint,1,opt,name=id" json:"id,omitempty" bson:"id,omitempty"`
    Name string `protobuf:"varint,1,opt,name=name" json:"name,omitempty" bson:"name,omitempty"`
}

protoc-gen-go generates json by default, but sometime we need others tags for marshal/unmarshal. As a result, I have to write another struct the same as generated except tags.

@meling I don't thinking it's a good idea to insert more info to every field option. protoc-gen-go bind struct with json, but we use struct in many cases, such as bson and xml and some custom cases. The global option mentioned above is to solve the problem, which is not only in my case.
To database/storage, for example gorm, use tags to define index/unique_index is a crazy action. We use protobuf with many languages, option for every field is useless for python/java. These options make the .proto unreadable  for python/java developers. What's more, option for every field makes the .proto file hard to read and modify, for there are many service logics. Simple is better.

Why is it crazy to define index fields using standard protobuf options syntax? I didn’t design the  option syntax for protobuf, but I learn to adapt. Just because something is unreadable to some people doesn’t mean it is crazy. What is crazy is to do “manual” conversion between a protobuf struct and a database struct when they are otherwise identical, because that is just error prone... and slow.
Of course your approach is simple, but it doesn’t solve the problems of at least half of the people asking for this feature...

I could not have said it better myself 😀 +1 Hein Meling
On Wed, 27 Dec 2017, 08:45 Hein Meling<EMAIL_ADDRESS>wrote:

Why is it crazy to define index fields using standard protobuf options
syntax? I didn’t design the option syntax for protobuf, but I learn to
adapt. Just because something is unreadable to some people doesn’t mean it
is crazy. What is crazy is to do “manual” conversion between a protobuf
struct and a database struct when they are otherwise identical, because
that is just error prone... and slow.
Of course your approach is simple, but it doesn’t solve the problems of at
least half of the people asking for this feature...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/golang/protobuf/issues/52#issuecomment-354069368, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABvsLfngLDTEv0R834oLqVAF01xUdQP4ks5tEfWZgaJpZM4Fbmpd
.

A file-level option for tags is off the table. It requires protoc-gen-go to have a semantic understand of all the tag formats that users may want, thus coupling a variety of struct tag formats into the proto repo. Individual field options would be the way to go since they are more extensible and does not bake package-specific logic into the generator.

@dsnet I could not agree more.

I abandoned my efforts on this since I saw the posts. So now I keep separate models (with JSON mapping to do the boilerplate) for protobuf generated structs ("Remote" structs, as I suffixed them now). A bit more code and another layer of indirection, but it works and has been in production for 4 moths.

@willks From your first comment it seems like you need to do validation. I wonder if you could provide some more details about your use case, e.g. an example. In my earlier reasoning I found that adding validation data to struct tags may not be the best approach, since ideally you would want to have such validation data (e.g. upper/lower bound on an integer value) encoded such that the protoc compiler can generate validation code for all supported languages, not just Go. If validation is the main purpose, perhaps you could make the case to add such features to the top-level protobuf compilers (I have no idea what the politics would be or if something like this has already been proposed...)
However, avoiding boilerplate code to translate proto messages that map identically to a database/storage model, that may need some additional go tags for the storage model, I do think making use of a go_tags option makes sense. My interpretation of @dsnet is that the project is open to accepting something along these lines... I have been thinking about writing up a patch myself, but haven't found the time, and the project where I need this is currently on hold.
I would encourage more people to submit examples of how they imagine that they would use such custom options to generate custom Go tags.
To start this off, let me point to our Autograder project where we want to use this: ag_service.proto.  We tried using gogoproto.moretags, but there are some compatibility issues with the grpc-web frontend codebase in typescript that remains unresolved.

RE validation: I would like to note one of the reasons why the required/optional distinction was eliminated was because of a considered position within (and without?) Google that validation should not be a part of the protobuf process itself, but rather consumers of a protobuf should be validating protobufs themselves.
Boilerplate validation is convenient for quickly throwing validation into something, but in the end, the inflexibility results in difficult to maintain, if not unmaintainable code. (The second one wishes to do anything other than just reject the protobuf outright, a complete rewrite must be performed, and oftentimes, removing the baked-in validation from the protobuf is an impossible, due to infrastructure being built upon the assumption.)

for me the major benefit of this feature would be model binding: i have a REST service implemented in go (with gin-gonic), the body is encoded in JSON and can be mapped to a go-struct using the json-annotations.
now this REST server is actually just a gateway to embedded devices which use ProtocolBuffers for the de-/encoding of messages. without any possibility to manipulate the annotations generated by protoc i have to bind the full message 1:1 to the (JSON) REST service.
implementing, e.g., a message with some field holding internal data (only used by the gateway and the device but not exposed by REST) is not possible without either manipulating the generated code OR duplicating each and every message this gateway sends through > which defeats the whole point of using ProtocolBuffers.
more likely use-case: the API is defined externally. someone suddenly decides that CamelCase is a good option instead of underscores > now instead of simply modifying the JSON annotations i'd have to either re-generate and thus re-program all devices or introduce boilerplate (mapping)

@meling You are right, this is not something that protobufs/gRPC needs to be concerned about. Protobufs/gRPC is the transport mechanism, it's how we convert it to the relevant domain model that we need to be concerned about. Hope that makes sense :) I swapped to Java for my server side since my last post, consolidating of the logic based on generics now. My validation is made on the "mapped" objects in Java to return the relevant errors.
My clients are both iOS/Android.  It has been addressed so I take back my previous post :) Thanks for replying.

Here's a formal proposal.
Background
The protoc generators for Go protobufs have historically generated proto messages as structs with fields since 2009. The major benefit of struct with exported fields is that it produces more idiomatic Go code and is generally nicer to use for users. However, at the same time, users were never given control of struct field tags, which are arguably an important aspect of how Go structs can interact with the rest of the ecosystem (e.g., encoding/xml, gopkg.in/mgo.v2/bson, etc).
Proposal
I propose we add support for a go_tag proto option that enables the user to specify the struct tag value for a field.
Example:
syntax = "proto3";

package example.gotag;

message Foo {
    string Field1 = 1 [go_tag = "xml:\",chardata\""]
    oneof Field2 {
        option go_tag = "xml:\",chardata\""
        string Field2a = 2;
        string Field2b = 3;
    }
}

The specifics:

This option only affects FieldOptions and OneOfOptions and is ignored when the option is specified on any other option dimension.

Only generators that emit messages as structs with exported fields need to respect go_tag. Those that don't can ignore it.

The syntax of go_tag is any arbitrary string. It is recommended (but not required) that the string follows the tag convention used by nearly every Go package. The value of this tag is concatenated to any tags already generated by default. For protoc-gen-go, we already generate protobuf and jsontags, which we should phase out over time (see points 4 and 5).

For protoc-gen-go, phase out generation of protobuf struct tags.
The information held in the protobuf and protobuf_xxx tags are implementation details that should stop leaking out to the public API. The work to make Message behaviorally complete (#364) will provide the proper API to supply the proto type information that was originally encoded in the protobuf tags.

For protoc-gen-go, phase out generation of json struct tags.
The json tags were an half-hearted attempt at satisfying the JSON<->Protobuf mapping. However, the encoding/json package is fundamentally insufficient to properly satisfy the mapping for all features of protobufs (e.g.,  well-known types, required fields, etc). Rather than provide the illusion that we properly satisfy the specification (when we don't), we should drop support for the mapping entirely. If users desires for the encoding/json package to be equivalent to the JSON<->Protobuf mapping (then see #256). If users desire to directly control how encoding/json interacts with generated messages, then they can use the go_tag option to control the tags.

Use cases
This feature enables full control of struct tags by the user in a simple way.

This supersedes the jsontag and moretags options in gogo/protobuf.

Design specifics
What happens if the go_tag contains the json tag?
For the near future, users should avoid doing this since it will conflict the json tag already generated by default. It is up to each generator implementation whether to explicitly ensure that the json tag is not set to avoid a conflict.
What happens if go_tag is specified on fields belonging to a oneof?
A generator may ignore it if the fields of a one-of are not represented as a struct (as is the case for protoc-gen-go). However, if a generator does represent a oneof as a flat struct with nullable fields, they may choose to place the tag on the corresponding field.
What happens if go_tag is specified on an extension?
The generator should error in this case. It is impossible for extensions to be added to a struct at proto compile time since extended fields can be added by any fields.
What about generating "opaque" messages?
There have been some proposals to possibly change the layout to support other generated APIs that are more conducive to lazy unmarshaling, where only a small set of fields are accessed. Such APIs would avoid being struct structs with fields, but instead enable access to fields via setter/getter methods. In such a case, struct tags have no semantic meaning.
At this point, it is probably reasonable to assume that the specific API generated by protoc-gen-go will always be struct with fields. Support for different APIs should be performed by an entirely different generator implementation. Such generators that do not emit messages as a structs with fields should just ignore the go_tag option.
\cc @neild @awalterschulze

Total bikeshed: go_struct_tag.

Loving this proposal.
This is one of the harder ones because of the json tag already being generated and the backwards compatibility problems that it might cause.  And then also adding the confusion of jsonpb vs encoding/json.
I am optimistic that this proposal helps in both regards.
A possible solution for number 5 would be to create protoc-gen-gov2 that does not generate any json tags and break backwards compatibility.
Then we can add a preprocessing step to protoc-gen-go that ads all the json tags as new go_tag options.  That way the generator library code only needs to maintain the newest version.
If we semi controversially live at head then we can also provide a tool to upgrade .proto files to include the jsontags inside go_tag options, for users who want to keep this feature.
But thats just some ideas for number 5.
Overall I think this proposal looks awesome.

@dsnet  Thank you very much for taking the time to phrase this awesome proposal. I completely agree with you that it would probably be better to drop the auto generated json tags.
However, to me, the readability of tags as protobuf options does not seem to be very good. This may lead to errors. Additionally it is actually quite cumbersome to write tags this way – which I would say does not align very well with the go spirit.
Therefore, another approach could be to keep the go struct tag information completely separate from the proto definition – i.e. in a adjacent .go file. This information could then be used to post process the .pb.go files and apply the go tag definitions to the generated structs. This would avoid putting non transport related information into the proto files while still allowing us to define custom go struct tags. We could even go a step further and define custom functions that are merged onto the generated structs.
mytype.proto, mytype-tags.go -> mytype.pb.go (with tags applied from mytype-tags.go)

For now, we wrote a small utility that allows us to use comments to specify tags. We then post process the .pb.go files and apply the comments as go struct tags using the go abstract syntax tree.
A minor inconvenience is that the protoc grpc plugin will not keep same line comments, so the tag comments must be placed above the field definition in the proto file.
The plugin can be found here: https://github.com/dkfbasel/protobuf

Here is another simple solution

@meling @dsnet I don't think it is a good idea to use a option like go_tag. As this option is golang only. Even the struct tag is golang only. Trying make a cross language solution is a waste of time.

@dsnet the go_tag is useless for other language. And actually, this repo can only generate golang code.

Yes, this repo contains the logic to generate .pb.go files. However, a .proto file is ingested by generators that produce source code for other languages. Magical comments is never going to fly compared to a proto option like go_tag or something else.

@meling you solution is awesom. Thank you. 👍

Thanks @puellanivis for your clarifying comments. I thought about interface implementation assertion too, shortly after posting my comment, but I wonder if that is a more costly test to perform in the critical path of the unmarshal function.
Anyway, I agree that it does not make sense to do validation in the marshal function. I totally see your points about compatibility, but there is also security/trust concerns with the data one receives. The question is how to balance these two concerns. Seems to me that the best approach (for now at least) is to do validation in each language implementation. I think it would be rare that one would need to implement multiple validation functions in different languages.
@dsnet @puellanivis does it make sense to write this up as a proposal in a separate issue, since this is really orthogonal to the go_tag issue?

Ok I am sorry, but getting all these messages about a validation language makes it really hard not to mention my own, especially now, since one has already been mentioned ;)
Here is a playground for Relapse, an encoding agnostic validation language
https://katydid.github.io/play/
It is super fast on marshaled protobufs, so you can do validation on input, right before unmarshaling
https://github.com/katydid/katydid
Currently only Go and Haskell are supported, but the testsuite is cross language.

I've managed to create a protoc plugin for this here. Currently it does not support oneof but I will add that in the coming days. I plan to finalise the proto API of specifying tags. Then I'll work on adding tests and internal restructure without effecting the proto API. Any thoughts on the following style,
message Example {
    string with_new_tags = 1 [(tagger.tags) = "graphql:\"withNewTags,optional\"" ];
    string with_new_multiple = 2 [(tagger.tags) = "graphql:\"withNewTags,optional\" xml:\"multi,omitempty\"" ];

    string replace_default = 3 [(tagger.tags) = "json:\"replacePrevious\""] ;
}

It achieves this by parsing generated proto files using go/ast package and then updating the struct tags and rewriting to the disk. The only downside is it has to run after protoc-gen-go has run.

@dsnet Thank you for the official proposal! This really helps in the enterprise world a lot!

I am looking for this also.

这个功能太多人需要了，咋就不开发一个呢！！

一个api模型， 一个数据库模型， 一个api转数据库， 一个数据库转api ........

#897

Duplicate of #46

This PR https://github.com/golang/protobuf/pull/969 is somewhat related to this.

Did anything ever come of this @dsnet? Sorry for the at but I figured since it was your proposal it might be nice to surface :-)

I am sorry if this is a duplicate as I am not sure how I run into this project:
https://github.com/srikrsna/protoc-gen-gotag
It solved the problem for me.  It might not be the correct solution, but the result is what I was looking for.

My workaround was to use sed in gen.go:
For example
//go:generate protoc --go_out=. myfile.proto
//go:generate sed -i "s/\\(MyField .*\"\\)`/\\1 datastore:\"noindex\"`/" myfile.pb.go

That turns
MyField    string    `protobuf:blah..."`
into
MyField    string    `protobuf:blah...." datastore:"noindex"`

Seriously though guys, @dsymonds please implement some support for this in protoc-gen-go already.

It is open source project.  One can make a PR, if really needs it.

It is open source project. One can make a PR, if really needs it.

There were a few attempts in helping to provide this in the form of PRs here or repo forks with comments in this and similar duplicate issues. It just does not seem to be interesting or important for the maintainers.
Most who needed the feature either rolled out their own tiny forks and scripts, or switched to gogo/protobuf.

solution N:
use sed in bash after protoc
sed 's/json:("[^"]+,omitempty")/json:\1 bson:\1/g'

json:"" tags are deprecated(?), so I wouldn’t rely upon them.

Thank you renld!
I just added -i and -E and got it working.
sed -i -E 's/json:("[^"]+,omitempty")/json:\1 bson:\1/' main.pb.go

@qianlnk I am having a problem making this work.... does it support proto3?
syntax = "proto3";
package contracts;

import "Common.proto";
//import "google/protobuf/timestamp.proto";

message ZZZ {
  string id = 1;  // `db:"id"`
  string clientId = 2;
  string datetime = 3;
}

Converting with:
protoc -I=xyz.eu/contracts/proto --go_out=plugins=grpc+retag:xyz.eu/contracts/ xyz.eu/contracts/proto/*.proto
Result:
type ZZZ struct {
	Id       string  `protobuf:"bytes,1,opt,name=id" json:"id,omitempty"`
	ClientId string  `protobuf:"bytes,2,opt,name=clientId" json:"clientId,omitempty"`
	Datetime string  `protobuf:"bytes,5,opt,name=datetime" json:"datetime,omitempty"`
}

The db custom tag on id is missing....
Like everyone using proto, I need this in order to work with (any) orm:)
Thanks for your time

That's not true.  Gorm will automatically use the Id field as your primary key.

 A data structure defined in a .proto file can be passed between different implementations or programming languages without loss of information. We should not add features to the Go protobuf implementation which are not accessible to other implementations.
A way to interrogate whether a feature of Go protobufs is appropriate or not is to conduct a thought experiment: If we have a Go service that reads and writes protobuf messages using that feature, can we practically rewrite it in another language without changing the message definitions or affecting clients of the service?
Imagine, for example, a feature which disables UTF-8 validation of string fields. This would be useful for some Go users, since Go strings are not required to contain UTF-8 data. However, Java strings are; the Java protobuf implementation has no way to represent non-UTF-8 string data. We could not rewrite a Go service using non-UTF-8 strings in Java without changing the field definition (from string to bytes) or making changes to the Java protobuf implementation.
In contrast, imagine a feature which permits setting the Go type used for a protobuf bytes field to either string or []byte. This would only affect the internal representation of data in the Go code. If we rewrote the service in another language, the annotation would no longer be useful, but harmless.
So, do custom struct tags constrain interoperability or not? At first glance, it might seem that they do not. As in the case of using string or []byte to represent a bytes-valued field, the struct tag appears only in the Go code. If we consider the use cases for these tags, however, it is not that simple.
One use case is message validation: A tag will be set on some fields indicating what data it may contain, and some package will inspect those tags when checking a message for validity. There is no good way to write that validator in some other language, since it depends on Go-specific annotations. Fortunately, we have a better, portable alternative: We can use protobuf options to annotate fields and inspect those options using the protoreflect package. (A version of this approach is described in "A new Go API for Protocol Buffers".) Not only will these annotations be accessible from other languages, using protoreflect will permit the validator to operate on other Go representations of protobuf messages, such as the one provided by the dynamicpb package.
The other often-cited use case is to permit the use of alternative serialization implementations, such as the "encoding/xml" package. It seems clear that this runs directly counter to the goal of protobufs as a language-neutral serialization mechanism: It is very unlikely that a message serialized using "encoding/xml" and some set of custom field tags will be deserializable by any other implementation. Protocol buffers provide three official serialization formats--binary, JSON, and text--and great care is taken to ensure that data serialized by one implementation can be losslessly deserialized by any other. If additional serialization forms are desired (XML, BSON, etc.), then they should be defined as a general-purpose mapping to and from the protobuf data types and implemented in a generic fashion, probably using protobuf reflection as the JSON and text serializations do.
In general, however, if a user needs to represent a protobuf data structure in some encoding not supported by protobufs (e.g., XML), then it is usually best to first encode the protobuf to a supported form (binary, JSON, or text), and then wrap the serialized data in the desired format. Alternatively, this may point at a case where protobufs are not a good fit for the job.
In both the case of validation and custom serialization, using custom field tags prevents interoperability with other protobuf implementations, are an inferior alternative to custom field options accessed via protobuf reflection, or both.
For these reasons, we do not believe we should add support for custom field tags to the Go protobuf generator.
I will leave this issue open for a few more days for final comment.

I am not contesting neild's post.
How does the logic above apply to JSON support in the library? ;)  Why JSON gets support and XML does not?  Unsigned 64 bit in JSON is serialized as string.  That's a very specific support for target format...
We have to understand why people want this feature in the 1st place.  The #1 motivation is to be able to tag structures for use by another library like DB layer (https://github.com/upper/db is just one example).
IMO, Protocol Buffers team needs to focus on helping developers to make it easy to assign protocol buffer generated structures to types that have required tag data.  Unfortunately, it is not possible today to do this:
type DBRecord struct {}
var record DBRecord = pbRecord

The problem is with additional fields that PB generates.  Those extra fields make it impossible to do an assignment of variables and requires manual field by field copy. I know that, because I had to code a lot of functions to copy data from PB and into PB.
Specification is clear here:

Type Identity
Type Assignability

It would be great if PB team could help us with copy data graph operations.  I think there are a number of approaches that can be taken here.

Those extra fields make it impossible to do an assignment of variables and requires manual field by field copy.

While I understand the desire to do this, I don't see a way for this feature to be provided without going against protobuf's goal of being backwards and forwards compatible. In particular, the ability to add new fields without breaking current usages. In order for this assignment (or a cast) to work, it requires that the memory layout of DBRecord be identical to the memory layout of the generated protobuf message type. Fundamentally, this adds a constraint that .proto source file be atomically updated with the equivalent Go type definition, which goes against one of the primary goals of protobufs.

We can go on and on with this feature, but if authors are very insistent on "my way or highway", there is not much we can do here.
This is one of the most requested features desperately missed from the original proto-gen-go. Missed so much that people switched to gogoprotobuf just to have it. If implemented, it would allow proto-gen-go generated code to be useful in more than one scenario that original authors thought of, and just be a good "code neighbour" by allowing the feature that is not in direct use of the library, but potentially make life much easier to devs who intend to "one more little thing" with generated code.
After 5 years of begging through this bug being open... This is actually quite sad.

if authors are very insistent on "my way or highway", there is not much we can do here.

To my recollection we have never said "my way or highway". We acknowledge the benefit of this feature, but we have always pointed at technical reasons as to the detriments of this feature. Fundamentally, we prioritize adherence to the overall protobuf ecosystem and this feature goes against that priority. We understand that this is not the preference of some users, but engineering is about evaluating different options, understanding the benefits and detriments of all options and making a choice.

After 5 years of begging through this bug being open... This is actually quite sad.

This statement makes it sound like we don't care about users. I should note that the fact that we spend significant amount of time engaging with people (even if we may disagree on technical points) should hopefully show that we care.

I can go back starting from a very first response on this thread, and continue to comb this (or other) thread(s) to support of what I have said, but it is not a productive use of my or your time.
You reopened this thread to gather users opinion? I think it is quite clear by now that we (users) wholeheartedly voted for it. I think there is quite an obvious way to "show that we care" other than continue to discuss it.
Thank you for your time.

Thank you @dsnet. You've done a great job supporting the project over the years.
As a community we aim to provide feedback on real world implementations and sought-after features. I don't have much to add outside echoing @Kulak where in most/all projects I've found myself copying data from/to PB.
There is a natural flow into the DB layer and the Go ecosystem makes heavy use of tags within that layer. Capturing this in a single generated object would be a huge boost in productivity for the Go community.
I may not agree with your final stand, but respect the adherence to the overall protobuf ecosystem. Keep up the great job.

I understand the engineering decision, but it is nonetheless sad. Some projects don’t care about compatibility with other languages. And doing manual copying between structs is error prone, and boring... I’m not hopeful that someone will pickup and fix gogo, so that it becomes compatible with this library, at least not in a releasable form in the next five years... unless someone put a full time team on it for the next year or so... it will probably be easier to start from scratch.
But I guess we can just hack this ourselves... I might do that when I get some spare time. Not likely to happen soon, so if anyone beats me to it, let me know.

Candidly, we’ve concluded the cost of abandoning protos at the DB layer is lower than continuing to work with them due to lack of struct tags.

@ydnar , we are about to face the same dilemma, and I am curious what did you choose as a replacement?

Candidly, we’ve concluded the cost of abandoning protos at the DB layer is lower than continuing to work with them due to lack of struct tags.

@ydnar , we are about to face the same dilemma, and I am curious what did you choose as a replacement?

We took the generated structs from protoc, deleted the protobuf-specific code, leaving struct tags as necessary. We already had a translation later between our db protos and API protos, so that layer didn’t need much additional work.

I guess the underlying problem is that protobuf messages are considered to be DTOs, and not primary entities.
While this is a common pattern in other languages, it is diametral to Go, imho.
Go walks quite a mile to have your primary entities to be used as DTOs as well -- one of its beauties. But I guess if one wants to use protobuf messages, it seems a DT layer is mandatory, as well as adding a translation of DTOs to primary entities and vice versa.
One thing I can not wrap my head around, however, is that we are talking of protoc-gen-go: This is language specific, so it would not hurt other languages to have language specific features in the code generation -- after all, the according markup could be ignored in other languages by default.

I'm going to re-open this issue, but it does not mean that we're going to support this. I'd like to see more discussion on the feasibility of this. Some thoughts:

I can see the usefulness of this feature for Go-heavy projects, but protobufs were designed a language-agnostic way to interchange data. Baking more Go-specific details in the proto files seems to be antithetical to the philosophy of protobufs.
This features makes the assumptions that generated Go protobufs are always structs with fields. What does it mean if we want the ability to generate opaque protobuf types that are interacted via methods. Performance testing has shown that techniques like lazy unmarshaling can bring significant speed benefits, but is only feasible as an opaque type.
The proper way to support this seems to be using protobuf options, but I don't see any proposal here suggesting a syntax for how that would work.

this is an old issue but I still see tags in pb.go files so maybe a sane approach at a .proto3 level could be to add a CRD generator and then allow to use it as a base for stuff like knative objects used by controllers or maybe php ORM yml custom schemas.

I imagine there should be another issue for this discussion, and I am happy to open one, but this seemed like the place for the relevant discussion. That's especially true given the locking of #1142.
I'm sort of stumped by the outcome of https://github.com/golang/protobuf/issues/1142 given the language from @neild regarding type control. I second the question in https://github.com/golang/protobuf/issues/52#issuecomment-642148289 though do understand that this is something of a slippery slope and may lead towards protocol buffer files which are too closely tied to an implementation..
What I'm really stuck on is nullability. I don't follow the principled argument against adding options to interact with the behavior of protoc-gen-go w.r.t. to nullability. Running back through the though experiment, it seems that such an option does not violate the language-switching principle.
For what it's worth, the performance wins of avoiding allocations has lead cockroachdb/cockroach to use gogo/protobuf despite the fact that it is not maintained and will not obviously ever have support for the new library which is so much better. I suppose adding something like object pooling via something like what maybe is being discussed in https://github.com/golang/protobuf/issues/1192 could be helpful but tracking object lifetimes is quite hard.
It feels like this has been a prolonged philosophical debate but I'm not sure I feel like the philosophy got lost in the w/ struct tags, which I can get behind. That argument however has extended to cover other more agnostic portions of the generated code. Value types are a major feature of go to provide not just access locality but to dramatically reduce the number of allocated objects making it more reasonable for performance sensitive code despite having a garbage collector. The lack of ability to utilize this key property of the language without forking the code generator feels confusing and at odds with a commitment to support go as a first-class user of protocol buffers.

FWIW I do feel that controlling the type of the generated structs for messages is valuable but I don't have a clear vision on how to expose that protoc in a clean way just yet.

I've submitted https://github.com/golang/protobuf/issues/1225 as a concrete and focused issue related to the above comment.

I'd like to add that this feature would be extremely useful for validation purposes.

I have written a tool named protoc-gen-go-tag to this feature!

Need this feature,A lot of request and response are not standard json lowercase ,so we need this feature to custom json tag.


Request ressources on brainhack_cloud for Github Actions runner for macapype package validation

Title
Github Actions runner for macapype package validation
Is this a new project?
yes
People involved
@davidmeunier79 @BastienCagna @oliviercoulon @kepkee
Project Description
Macapype is a python package for segmentation of non-human primate anatomical MRI images; It runs on github, with unittests and PEP8 check on PR; For pre-relsease / release, we would like to validate full pipeline runs with some example data. However, the github server are limited in memory size availble for Github Actions, and for some stedp of the validation (namely using FSL fnirt) requires up to 30 Gb of RAM. Hence my request to  add brainhack_cloud ressources to run on release validation with github actions.
Link to project repository/sources
https://github.com/Macatools/macapype
https://macatools.github.io/macapype/
What resources do you require?
Virtual Machines
Anything else?
No response

All done and setup @davidmeunier79 - you should have an email with the account details :)

This project is now closed and all resources on the Oracle Cloud will be deleted within the next few days.


Mojaloop as a payment method

Context
Currently, we are looking to add support for Mojaloop payments to support our use case for cross-border, remittance payments in the US, Mexico corridor. This requires:

abstracting away some of the ILP related code we have in Rafiki to be more method-agnostic
adding Mojaloop payment method in Open Payments, Rafiki.
adding our own Mojaloop connector similar to the ILP connector we currently have

Mojaloop Identity Lookup within Rafiki:

Mojaloop quote within Rafiki:

Mojaloop transfer within Rafiki:

Todos

[x] Create payment-method directory, move ILP related files into own directory (closed by https://github.com/interledger/rafiki/pull/1961)
[ ] Update Rafiki to use new Open Payments spec with Incoming Payment & Quote payment method (https://github.com/interledger/open-payments/pull/284)
[x] #1967
[ ] #1999
[ ] #1989
[ ] #1991
[ ] #1995
[ ] #1996
[ ] #1997
[ ] #1998
[ ] #2000
[ ] (Optional) rename peering mutations from createPeer... to createIlpPeer...

Closing in favour of larger Rafiki architecture overhaul

Closing in favour of larger Rafiki architecture overhaul


Dashboard resources always homed in EastUS?

It seems all "Microsoft.Portal/dashboards" resources are homed in EastUS, with no option to home them in any other region. Is this correct?

@ChrisSommers Thank you for reaching out to us. Can you please provide a link to the document which you are following so that I can engage the correct team to help you.

@ChrisSommers Could you please provide more background on the issue that you're experiencing. This seems to be a technical issue which requires more information and attention from the right audience. I suggest posting your query in the MSDN forum with more details.
Closing this issue for now. If you've any concerns regarding MS docs or you want to suggest some changes/enhancement to the docs, feel free to tag me in your reply and we can continue the discussion.


#7 Backport to API 14+

Uses Transitions-Everywhere library for backward compatibility.

@Mayankmkh thank you for your submission, I've been working on the same issue 👍
Do you have an idea do backport without having to create a dependency?
Since this is a library we wanna have as little impact as possible, and importing a whole another library into this one doesn't seem the right way. Depending on the License it may even be possible to copy the actual class (once again, we must respect the creator's license).

Hi @JoaquimLey I found no other way to backport this library without using these dependencies. As the vector-compat library is under apache license v2.0, so it is possible to use code from that. Transitions-everywhere doesn't have license file so the developer needs to be contacted.
I'll be happy to know any other way to do so.

@JoaquimLey Another way could be to use AnimationDrawable instead of animated vector drawable for fab drawable morphing.

The framework does provide us the tools required to backport, of course some animations will be lost, including the VectorDrawable.
This is just a case of seeing the provided methods and work our way around it. I've pushed this branch with an initial draft/ideas to backport, it can be a good foundation.

Like I said, it's not optimal to be importing 3rd party libraries to this one, this will greatly increase the method count and general overhead. If these libraries are able, we sure can too. Are you interested in pursuing a more "clean" solution?
Thanks in advance for your interest 👍

@JoaquimLey Thanks for your advice. As you suggested I would definitely try to come up with a better approach to minimise the library footprint and refrain from using 3rd party libs.

Looking forward for your solution @Mayankmkh

@JoaquimLey Sorry, it is currently not possible for me to work for 2 weeks due to time issues. I would like to contribute after that. I hope you understand.

No problem, I'll be looking for it 👍

Hi @JoaquimLey I have removed third party libraries and tried to backport the animations to API 14. Please review the commits.

Rebase with current developement please @Mayankmkh

@JoaquimLey Thanks for your comments. I will rebase it with development branch and update you soon

Do you want me to fix the conflict?

I have already fixed the conflicts.


future: cache editables in memory during the current run

I've noticed that editable VCS dependencies get fetched multiple times during the same run, but they could and should be cached during the current run.
Apart from improving performance, this makes also sure that they do not change during the run.
IIRC I've changed that in the former future branch, before the rewrite.
I could imagine adding a MemoryDependencyCache, which behaves like DependencyCache, but would not read/write anything.
What do you think?

While this change is probably good anyway, I'd really like to cache the editables also for a certain period, e.g. 5 minutes.
When building the requirements for my project, I call pip-compile for multiple requirements.in files (dev, testing, production), and they should all use the cached result from the first run.

This was implemented in PR #476 to fix a new bug introduced in 1.8.1. Closing.


Ordered Sequence Question

Question
Does vegeta support an ordered sequence in a load test? For example I have 3 requests that need to be made in a very specific order (1 must finish before 2 begins, 2 must finish before 3 begins). I have a need to load test this sequence of calls but am not seeing a way to do it currently or maybe do not know what the terminology would be to look for. Any help in answering this is greatly appreciated!

Vegeta was designed originally for testing HTTP APIs without any concept of a user session or flow. Is this what you're trying to do?

Ah, yes that would be what I am trying to do. Guess it won't work for what I am trying to do. Thanks for the quick reply!

Take a look at https://k6.io


Tab titles change according to sound

fixes #269
Tab titles now change according to whichever sound you navigate to!
Thanks again @hamirmahal for outlining the bug and solution!

Tide just keeps getting better day by day! 💪


Will it support react in the future?

Hi, it depends on our commercial projects, we have no plans to migrate to React at the moment.


Do not init process group if already initialized

What does this PR do?

Simple tweak to skip initialization of the torch process group if one is already initialized.
Before submitting

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[x] Did you read the contributor guideline,
Pull Request section?
[ ] Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
[ ] Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
[ ] Did you write any new necessary tests?

Who can review?
@OlivierDehaene

@OlivierDehaene can you let me know if we can proceed here or if you think this PR should be rejected? Thanks.


KNX sensor unit_of_measurement isn't working

Home Assistant release (hass --version):
0.34.0.dev0
Python release (python3 --version):
Python 3.4.2
Component/platform:
KNX Sensor
Description of problem:
The unit_of_measurement of the sensor stays empty, in case of the KNX sensor this should be filled by the code depending of config variable "type".
Expected:
type: temperature, should result in TEMP_CELSIUS being used.
Problem-relevant configuration.yaml entries and steps to reproduce:
sensor:
  - platform: knx
    name: Hobbyroom Temp
    type: temperature
    address: 3/3/1

Additional info:
Possible fix (works for me) is moving property unit_of_measurement from KNXSensorBaseClass() to KNXSensorFloatClass()
See attached patch.
patch.txt

Instead a patch please make a PR :+1:

Seems a lot of work for just moving 4 lines and deleting one, but will do.  ;-)


VSC open browser with Webclient Base URL

Describe the bug
I am using sandbox on my PC. I set port sharing. So I have access to my snadbox on my PC from company network. Developers can connect to my sandbox through VSC and they can do changes.
Developer can publish new extension version to my sandbox. Developer can download symbols from my sandbox. Everything works perfectly. Problem is that when developer use F5 for publishing and debugging VSC open browser where is different URL than developer set in launch.json. URL gets from my sanbox with container name and port. So developers have to manually change URL from containername:port to myPCname:port and after that everything works.
Example:
Container name: TEST
Container is on PC: PC-STATION1
Port mappings: 7049 --> 7249
80 --> 82
Publishing and debugging on PC-STATION
publishing works and VSC open web browser with http://TEST/BC. It works
Publishing and debugging on STATION in network
publishing works and VSC open web browser with http://TEST/BC. It doesn't works it have to be changed to http://PC-STATION:82/BC and after that everything works.
I can't change Webclient Base URL because it moves problem from one PC to another.
To Reproduce
Steps and to reproduce the behavior:

Create new container with sandbox (last version of BC 17) on PC
Set port mapping (you can set it during container creation when you use BCContainerHelper)
Use different PC in same network to test it
Run VSC
Create Hello Word project
Use setup similar to "Publish to PC-STATION1" in my launch.json
Download symbols
Use F5 for publish project
Project is published without errors
Browser is opened with wrong URL

{
    "version": "0.2.0",
    "configurations": [
        {
            "type": "al",
            "request": "launch",
            "name": "Publish to My SandBox 17",
            "server": "http://TEST",
            "tenant": "default",
            "serverInstance": "BC",
            "authentication": "UserPassword",
            "schemaUpdateMode": "Synchronize",
            "breakOnError": true,
            "launchBrowser": true,
            "longRunningSqlStatementsThreshold": 500,
            "numberOfSqlStatements" : 10,
            "enableLongRunningSqlStatements": true,
            "enableSqlInformationDebugger": true,
            "dependencyPublishingOption": "Ignore",
        },
        {
            "type": "al",
            "request": "launch",
            "name": "Publish to PC-STATION1",
            "server": "http://pc-station1:82",
            "serverInstance": "BC",
            "authentication": "UserPassword",
            "schemaUpdateMode": "Synchronize",
            "port": 7249,
            "breakOnError": true,
            "launchBrowser": true,
            "longRunningSqlStatementsThreshold": 500,
            "numberOfSqlStatements" : 10,
            "enableLongRunningSqlStatements": true,
            "enableSqlInformationDebugger": true,
            "dependencyPublishingOption": "Ignore",
        },
   ]
}

Expected behavior
VSC should follows parameter "server" in launch.json when open browser. If there is some reason for this behaviour it should be some optional parameter that force setup from web client base URL.
Screenshots
5. Versions:

AL Language: v6.1.362735
Business Central: 17.0.17126.18832-CZ

This looks like a bug. We'll add it to our backlog for a more in-depth evaluation.


[#98] Add contacted date

Fixes #98
Add contacted date to each person, which can be added or edited using the add or edit command via the 'c/` prefix respectively.

Codecov Report

Merging #133 (c79e7eb) into master (9f4c593) will increase coverage by 1.02%.
The diff coverage is 79.20%.

@@             Coverage Diff              @@
##             master     #133      +/-   ##
============================================
+ Coverage     74.20%   75.23%   +1.02%
- Complexity      539      566      +27
============================================
  Files            87       88       +1
  Lines          1636     1700      +64
  Branches        171      183      +12
============================================
+ Hits           1214     1279      +65
+ Misses          364      363       -1
  Partials         58       58

Impacted Files
Coverage Δ

.../java/seedu/address/logic/commands/AddCommand.java
100.00% <ø> (ø)

...ain/java/seedu/address/model/StateAddressBook.java
90.62% <ø> (ø)

src/main/java/seedu/address/model/person/Memo.java
91.66% <ø> (ø)

.../java/seedu/address/model/util/SampleDataUtil.java
20.00% <0.00%> (ø)

src/main/java/seedu/address/ui/PersonCard.java
0.00% <0.00%> (ø)

...a/seedu/address/logic/parser/AddCommandParser.java
92.00% <75.00%> (-3.46%)
:arrow_down:

.../main/java/seedu/address/model/person/Address.java
80.00% <75.00%> (ø)

...rc/main/java/seedu/address/model/person/Email.java
80.00% <75.00%> (ø)

...c/main/java/seedu/address/model/person/Person.java
92.45% <75.00%> (-1.30%)
:arrow_down:

...rc/main/java/seedu/address/model/person/Phone.java
80.00% <75.00%> (ø)

... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9f4c593...c79e7eb. Read the comment docs.


qemu-guest-agent fails to install with latest 20.04

It seems that sincer I changed the ISO to Ubuntu 20.04.5, my build fails and the problem seems to be qemu-guest-agent. When doing the subiquiti install and getting to it, it just fails with an exit code 100 and returns a prompt. Nothing really helpful in the logs.
I tried with 22.04.1 and had the same issue until I found this post: https://askubuntu.com/questions/1427461/server-22-04-1-subiquity-autoinstall-system-install-command-fails-after-changi
After adding the network part with the correct NIC naming for Proxmox (ens18 in my case), it works perfect.
Unfortunately, the same fix does not work for 20.04.5 I am not sure what has been changed there but maybe others can try with this ISO and confirm or not if you are having the same. I also opened a thread here: https://askubuntu.com/questions/1440007/qemu-guest-agent-fails-at-autoinstall
Thanks,
Andrei

Let's wait if we get more info from Canonical or Ubuntu Community here, I can't reproduce the issue on my end, unfortunately

I did an upgrade of Proxmox today and will try again as the qemu tools were updated as well. Will try again tonight and let you know the result.
Thanks for the help!

@andrei-matei do you have any update?

@ChristianLempa
Thank you for the wait. After the upgrade of Proxmox I tried again and it seems I got the same result.
As others don't have the issue, I guess it's something related to my env but cannot figure out why. Anyways, I user the 20.04.4 image and do an apt-get upgrade but would really want to understand what't the matter with .5 iteration :)
Right now I don't have access to the env to perform tests as my VPN is down but will try again next week. Please leave it open a bit more.
Thanks,
Andrei

@andrei-matei i have done a workaround that worked for me for ubuntu 20.04.5. i answered this question who also had problems with qemu-guest-agent. my workaround is to install qemu-guest-agent as part of the late-commands block instead of inside the packages block as an example here:
#cloud-config
autoinstall:
  version: 1
  locale: en_GB
  late-commands:
    - curtin in-target -- apt-get update
    - curtin in-target -- apt-get install qemu-guest-agent
  keyboard:
    layout: us
  ssh:
    install-server: true
    allow-pw: true
    disable_root: true
    ssh_quiet_keygen: true
    allow_public_ssh_keys: true
  packages:
    - sudo
  storage:
    layout:
      name: direct
    swap:
      size: 0
  user-data:
    package_upgrade: false
    timezone: Asia/Manila
    users:
      - name: <user>
        groups: [adm, sudo, docker]
        lock-passwd: false
        sudo: ALL=(ALL) NOPASSWD:ALL
        shell: /bin/bash
        # passwd: "password1"
        # - or -
        ssh_authorized_keys:
          - <ssh_key>

let me know if this updated user-data worked for you too


fix(test): fixed username shadowing in test scram setup

Issue
https://github.com/tulios/kafkajs/issues/703
In addition to fixing the related issue, I've also included user feedback in the execution which can help to detect bugs such as this one.

Thanks for looking into the issue. How about instead of relying on environment variables, we move to taking the username and password as parameters that are passed in when you execute the script? ./createScramCredentials foo bar
We can still have default values, same as now, but not using environment variables would remove the risk of someone running into the same issues as you are now.

That's a good idea, kinda moves in a way so it can be implemented as part of a CLI in the future.
I'll take a look at updating ./createScramCredentials later today;

Hi @Nevon , I've added some flags, basic docs and behaviors;
Just want to check with you if that aligned to what you were expecting if so I'll also update the docs at least to indicate that those new commands are available

That's quite a bit more advanced than what I had in mind, but sure, it seems to work great.

Yeah, I tried to stay as lean as possible so as to handle only the username and password but while also doing some checks and falling back to default values.
I've updated the docs now to reference the new controls and it seems ready to me.


Add Plausible Script

Thanks for this @ZakGriffith !
We definitely need to set up prettier in the project. Opening an issue for it.
It seems that your IDE applies some prettier rules out of the box. Could you disable it for this PR so we can merge? (And use single quotes for now)


Conversion result is always 0 when subunit difference is large enough

I'm trying to operate with custom "Ether" cryptocurrency using the library, but conversion to/from other currencies doesn't work as expected: the result is always 0. The problem is that "Ether" subunit value is 18.
Since in Money\Converter::convert() we do
$ratio = $ratio / pow(10, $subunitDifference);
$counterValue = $money->multiply($ratio, $roundingMode);

the result will always be 0 if $ratio is too low or $subunitDifference is too large, because first argument of Money\Money::multiply() will be rounded to 8 decimal places inside.

I kind of get your problem. Can you however emphasize with an example?

use Money\Converter;
use Money\Currencies;
use Money\Currencies\AggregateCurrencies;
use Money\Currencies\ISOCurrencies;
use Money\Currency;
use Money\Exception\UnknownCurrencyException;
use Money\Exchange\FixedExchange;
use Money\Money;

final class EthereumCurrencies implements Currencies
{
    const CODE = 'ETH';

    public function contains(Currency $currency)
    {
        return self::CODE === $currency->getCode();
    }

    public function subunitFor(Currency $currency)
    {
        if ($currency->getCode() !== self::CODE) {
            throw new UnknownCurrencyException(
                $currency->getCode() . ' is not ethereum and is not supported by this currency repository'
            );
        }

        return 18;
    }

    public function getIterator()
    {
        return new \ArrayIterator([new Currency(self::CODE)]);
    }
}

$currencies = new AggregateCurrencies([
    new EthereumCurrencies(),
    new ISOCurrencies(),
]);

$exchange = new FixedExchange([
    'ETH' => [
        'USD' => 330.84,
    ],
]);

$converter = new Converter($currencies, $exchange);

$eth = Money::ETH('1');
$usd = $converter->convert($eth, new Currency('USD'));
echo $usd->getAmount() . "\n";         // expected: 0, actual: 0

$eth = Money::ETH('10');
$usd = $converter->convert($eth, new Currency('USD'));
echo $usd->getAmount() . "\n";         // expected: 0, actual: 0

$eth = Money::ETH('100000000000000000');  // 10^17
$usd = $converter->convert($eth, new Currency('USD'));
echo $usd->getAmount() . "\n";         // expected: 3308, actual: 0

$eth = Money::ETH('1000000000000000000'); // 10^18
$usd = $converter->convert($eth, new Currency('USD'));
echo $usd->getAmount() . "\n";         // expected: 33084, actual: 0

@sagikazarmark This is a harder problem. The problem is on this line. That results in a float. Then, in the next line, $money->multiply is called. Because a floating point is given as multiplier, it is converted to string first. So, Number::fromFloat is called, which on its turn calls sprintf to convert the float to a real number. And sprintf is not aware of big integers.
So, this problem has two possible solutions. One: fromFloat should return big integer aware results. Two: we change the calculation that results in a float instead of string.

Three. We add a calculation method to Money in order to use a Calculator for the calculation.

I think the pow calculation can be removed, and replaced by a simple string manipulations. I'll try that first.


Updating adaptivecards.io homepage's roadmap links to the "roadmap" tab on productboard

Description
Updating the website's roadmap links to the new roadmap tab in product board
How Verified
Using the test website deployment
Microsoft Reviewers: Open in CodeFlow

@nesalang , @almedina-ms could you help sign off on these? Thanks!


Masking-API-Client-Extension is broken for Masking Engine > <IP_ADDRESS> versions.

Problem
Masking-API-Client-Extension is broken for Masking Engine > <IP_ADDRESS> versions.

Fixed.


Enable the user to set an existing access token

Coverage increased (+0.4%) to 74.368% when pulling 2d8ec2ed7c34ece4cb61d273690c1c52f61cbd7f on set-access-token into 72f6a1de98eeec701bf5773c3bb60d753df5624c on master.

Coverage decreased (-1.5%) to 72.464% when pulling 38a6aeb5bfaadd7037a1f6509cf4bb7caea541a0 on set-access-token into 72f6a1de98eeec701bf5773c3bb60d753df5624c on master.

Coverage decreased (-0.4%) to 73.626% when pulling b6ae116cb953a6dcbf947fcccb35399531938911 on set-access-token into 72f6a1de98eeec701bf5773c3bb60d753df5624c on master.


Embed

Embed the payload.so into the Go binary.

I had the same idea , I already added the needed file to embed and perform autorelease , feel free to take a look and eventually make a new pull request


Pandas 0.22 upgrade

Makes zipline compatible with pandas 0.19 and 0.22, but lacks support for pipeline categoricals.
Part of https://github.com/quantopian/zipline/issues/2132.

Coverage increased (+0.01%) to 86.455% when pulling 0b9c6beef71bbbb894c2e4f6b2e3bf01163a4bc7 on pandas_upgrade into d4de30860b15e18aa5814167f8bceb98a71a15b5 on master.

@richafrank this looks good to me. We should probably open a tracking issue to document the known failures with categoricals in old pandas.
It also might make sense to do a release with these changes and the trading_calendars refactor before we merge the breaking changes from https://github.com/quantopian/zipline/pull/2192.

🎉 👍 🥇

Should we release a new version with the updated pandas dependency?

@twiecki Great idea 😄 . zipline 1.3.0 has this change.


BUG Fix test that relies on implicit ID order breaking postgres

Merging up from 3.5 to 4.0 is awful.
I've added another commit to fix linting issues.

Yay thanks.

Cool - related 3.x fix: #8132


Issue with 33-challenge-sealed-classes folder name

When I try to clone the repository I get the following error in the terminal
error: invalid path '33-challenge-sealed-classes /.idea/.gitignore'
fatal: unable to checkout working tree
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'
It looks to me like there is a space character at the end of the directory name "33-challenge-sealed-classes ".
If I download the zip file I can see an underscore character at the end of the directory name, and in the URL in the browser I see a %20, indicating a space character.  I can continue by using the downloaded zip file, but cloning the repository cause my error.

Hi @paulio84 , thanks for pointing this out!
I have updated the directory name so it doesn't have the space anymore — hopefully has fixed the error!


switching from mobeel fat aar plugin to kezong fat aar plugin

mobeel plugin does not work with the recent gradle version)
note: this needs to be tested (with newer gradle build tools version) and then commented out code should be removed

It may not work for now.


Reproducing results

Hi, very interesting work! Currently, I am trying to reproduce to the task1a FCNN results where you reported 76.9 % accuracy with all the augmentations. I am a bit confused about the evaluation_setup files.

I see that fold1_train_all.csv has 69825 files and trainall_full.csv has 100345 files. Which one of these two should be used for training to reproduce the results (76.9% model) along with spectrum correction and reverb+drc  augmentation?
Could you tell me which one of the three rir_label i.e 2003, mardy and reverb, was used in your ICASSP paper for reverb+drc augmentation? Was energy normalization used?

I appreciate your help in clarifying the confusion.

Hi,
Thanks for your interests! For your questions:
1 We used all the data augmentation techniques that mentioned in our report to get the final system. We realized that the current csv files look a bit confusing, and we just made some modifications to clean the codes and make it easier to understand. You can refer to the current fold1_train_all.csv file (https://github.com/MihawkHu/DCASE2020_task1/blob/master/task1a/10class/evaluation_setup/fold1_train_all.csv). It indicates all data we used to train the 10-class model. Specifically, we have Original data + Mix audios + Random noise + Pitch shift + Speed change + Spectrum correction + Reverberation with dynamic range compression (mardy + reverb + 2003). For the 3-class model, you can refer to the csv file of https://github.com/MihawkHu/DCASE2020_task1/blob/master/task1a/3class/evaluation_setup/fold1_train_all_3class.csv.
2 We used all the three types of reverberation data (RIR from mardy, reverb and 2003). As pointed in the fold1_train_all.csv file, we used 3 types of reverberation data, as each device A audio has 3 different reverberation transformations. You can check https://github.com/MihawkHu/DCASE2020_task1/tree/master/task1a/data_augmentation/spectrum_correction for more details about how to generating them.

Thanks.
I assume you have used .m scripts in reverb_drc/m-scripts/ instead of _en.m scripts ( as given in gen_RevDevelopment.sh). Did you observe any difference in results with energy normalization in reverb augmentation? I am just curious since these results are not mentioned in the report or the paper.

Hi,
In our experiments, we used _en.m as scripts. With energy normalization, we observed an improvement but honestly we did not investigated too much on it.

Thanks a lot.


gRPC: Adding new API for GetAccountByNumber

The GetAccountByNumber API would be beneficial in scenarios where user need to fetch account details by account number instead of address.
This API is similar to GetValidatorByNumber and can follow the same way to implement.

/bounty $10

/attempt #505

/attempt #505

/claim #505

hey @Ja7ad, you'll want to include /claim #505 in the body of your PR (ex. right after Fixed #505) 👻


Link Rot: a.co

The following links are returning non-successful (non 2**, 3**) status codes. This ticket serves as a reminder of the problem and gives an issue to link the "ignore" and "fix" commits.
Problematic Link(s): (Limit to one domain)

a.co

Issue Summary:

[x] The entire domain seems to be having problems or is unreachable, expected to resolve on its own.
[ ] A link has been moved.
[ ] A link has legitimately been removed.
[ ] This is likely a routine aging of a short-link (or similar).
[x] Other, please note.

This only seems to be happening from build machines.

Closed in #87.


Fix non-editable installation

Fixes #33

Codecov Report

Merging #34 (ab99f51) into master (5e0d4ca) will decrease coverage by 90.38%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master     #34       +/-   ##
==========================================
- Coverage   90.38%   0.00%   -90.39%
==========================================
  Files          79      71        -8
  Lines        2986    2522      -464
==========================================
- Hits         2699       0     -2699
- Misses        287    2522     +2235

Impacted Files
Coverage Δ

laia/__init__.py
0.00% <0.00%> (-95.13%)
:arrow_down:

laia/nn/__init__.py
0.00% <0.00%> (-100.00%)
:arrow_down:

laia/common/types.py
0.00% <0.00%> (-100.00%)
:arrow_down:

laia/utils/checks.py
0.00% <0.00%> (-100.00%)
:arrow_down:

laia/data/__init__.py
0.00% <0.00%> (-100.00%)
:arrow_down:

laia/utils/__init__.py
0.00% <0.00%> (-100.00%)
:arrow_down:

laia/engine/__init__.py
0.00% <0.00%> (-100.00%)
:arrow_down:

laia/losses/__init__.py
0.00% <0.00%> (-100.00%)
:arrow_down:

laia/losses/ctc_loss.py
0.00% <0.00%> (-100.00%)
:arrow_down:

laia/callbacks/decode.py
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 66 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5e0d4ca...ab99f51. Read the comment docs.

Codecov Report

Merging #34 (ab99f51) into master (5e0d4ca) will decrease coverage by 90.38%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master     #34       +/-   ##
==========================================
- Coverage   90.38%   0.00%   -90.39%
==========================================
  Files          79      71        -8
  Lines        2986    2522      -464
==========================================
- Hits         2699       0     -2699
- Misses        287    2522     +2235

Impacted Files
Coverage Δ

laia/__init__.py
0.00% <0.00%> (-95.13%)
:arrow_down:

laia/nn/__init__.py
0.00% <0.00%> (-100.00%)
:arrow_down:

laia/common/types.py
0.00% <0.00%> (-100.00%)
:arrow_down:

laia/utils/checks.py
0.00% <0.00%> (-100.00%)
:arrow_down:

laia/data/__init__.py
0.00% <0.00%> (-100.00%)
:arrow_down:

laia/utils/__init__.py
0.00% <0.00%> (-100.00%)
:arrow_down:

laia/engine/__init__.py
0.00% <0.00%> (-100.00%)
:arrow_down:

laia/losses/__init__.py
0.00% <0.00%> (-100.00%)
:arrow_down:

laia/losses/ctc_loss.py
0.00% <0.00%> (-100.00%)
:arrow_down:

laia/callbacks/decode.py
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 66 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5e0d4ca...ab99f51. Read the comment docs.


server-metrics don't handle requests to the metricsService correctly.

I tried out the server-metrics package today and its BlazeServerExample.  Nice work, it looks really promising.
An oddity that I noticed was that the active_requests would increment but not decrement when the "/metrics" endpoint is invoked. It seems that the code in Metrics HttpMiddleware would increment but because the prefix was not found, it would never call funciton generalMetrics.

I think you're right.  Would you be interested in taking a crack at fixing this?  I think the problem is right around here.

@ChristopherDavenport said he'd try this after dinner.  If so, awesome.  If not, please kick back to me.

Behavior I'd expect:

GET /http4s/metrics does not increment counters (it’s not wrapped in the middleware)
GET /http4s/metrics does not leave a dangling active request
GET /http4s/dfdajkfldajfklddasfadskl does not leave a dangling active request. (This is one where a wrapped service returns None).
GET /http4s/djfkldajfkldadfadfjkasdf increments 4xx. (debatable, but consistent with error handler: it anticipates what the backend is going to do, and can’t know what fallback composed services does)
GET /http4s/ping increments 200 and does not leaving dangling active request

Quirky case that @rsoeldner found today:

GET /http4s/metricsadfjdasfjadklfadsfjkladsfjk returns 200.  This is because the router does simple prefix matching without regard to segments and MetricsService matches all GET requests, not just Root.  I think which one, or both, of these to change is a separate issue.


Anomaly detection - AUC for quantized model with INT8 input/output

Hi,
Regarding the anomaly detection, what is the expected AUC for the quantized model with INT8 input/output (ad01_int8.tflite or model_ToyCar_quant_fullint_micro_intio.tflite)? I am getting an AUC of 0.354 for the machine ID 0, which is very low when compared to the AUC for the quantized model, but with float input/output (model_ToyCar_quant_fullint_micro.tflite).

Hi @MiguelCosta94
It is around 0.85 as documented in https://github.com/mlcommons/tiny/tree/master/v0.5
The difference to float is actually little, so there should be something other problem in your setup.

Hi @cskiraly
It was a problem in my setup. Thanks for the reply.


Overwrite context menu with menu that input DH

@alexsb with implementation of this, remove "Select a DP" on top bar?

I think this is replaced by #17. Closing.


Add approval service listener

Adds the Approval Request Listener including a bin file for development environments.
Kafka Published Messages Are as follows: ( at create of the request )
    :payload  => {
      :request_id => '1'
    }

When the request is approved
    :payload  => {
      :request_id => '1'
      :decision =>"approved",
      :reason"=>"System approved"
    }

DRAFT: Still needs tests and the Threaded listener has nothing to update until https://github.com/ManageIQ/catalog-api/pull/167 is merged.

@lindgrenj6 please review


Ghostscript: Update URL and hash for ghostscript-9.18.tar.bz2 after upstream URL change

Things done

[ ] Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] OS X
[ ] Linux

[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Fits CONTRIBUTING.md.

Upstream changed their package filing; in particular, http://downloads.ghostscript.com/public/ghostscript-9.18.tar.bz2 does not exist anymore, which breaks this build for me.
They have a replacement (hosted both on github and their own site in a different directory) but it hashes differently. I have not been able to find a copy of the file with the original hash, so I'm also changing the hash to match. I have spot a few copies on third-party mirrors, and they were consistent with the new version despite allegedly dating back to 2015.

The hashes are the same, only the format is different:
nix-hash --type sha256 --to-base32 83daf5bbbb5afbc32cab944a1afa7ceca046dbf0c3712cd5f2bd21a13e484da1

18ad90za28dxybajqwf3y3dld87cgkx1ljllmcnc7ysspfxzbnl3

Pushed to master, sans the encoding change. Thank you.


Missing generated classes in build/apt/debug

When my project is set to build in debug mode, I cannot import the generated *_Table, *_Adapter, etc. classes. Looking at build/apt/, I only see these classes built in release and not in debug.
When I set my project to build in release though, I'm able to import those classes.
Any ideas how to fix this?

Almost same here: suddenly dbflow stopped generating any *_Table, *_Adapter, etc. sources though it was working well 1 month before. Build type change is not helping.
I'm using version control system and I'm sure gradle config was not changed

I have almost the same isssue. I  have about 20 tables, and it only generates 4 and sometimes 5 of the _Table classes.

@brwskitime I have found one workaround. I have moved all models (only models) to separate module within project and anything now works fine for me.
Except DBFlow I'm using Parceler library which is generating sources during compile time like DBFlow and before I had issues with it too. But now everything is working fine.
I guess this issue is not with DBFlow but with library that is generating sources.

I believe @re3lex just mentioned this, but discovered a fix. I had an abstract parent model to hold the primary key, and then all the real models/tables in a sub-package. Turns out when I moved my abstract class down to the same package as the rest, the problem went away.
Short story, keep all of your models in the same package!

@re3lex & @brwskitime I have all of my models (subclasses & superclasses) within the same module and still running into this problem.

@brwskitime looks like this is not my case but another one.
Thanks for highlighting it.

@arriolac try to move models into separate saying Models module hwere only models are put.
add into Models module gradle build only libraries your models depend on.
I hope this can help you (That helped me)

Didn't make any change but my errors went away after i updated to gradle 2.10 and build tools to 2.0.0-alpha5.

Still running into this issue actually. I believe the issue is related to gradle's incremental build. See related issue: https://github.com/google/dagger/issues/298

Any update on this issue?


Add more pages to the app.

Hi. Im new to AutoRoute, I cant figure out how to make more pages with this boilerplate.
So I made something liek this.
@AdaptiveAutoRouter(
  replaceInRouteName: 'Page,Route',
  routes: <AutoRoute>[
    //RedirectRoute(path: '*', redirectTo: '/'),
    AutoRoute(page: AppStartPage, initial: true),
    AutoRoute(
      path: '/home',
      page: HomePage,
      children: <AutoRoute>[
        AutoRoute<FieldInfoPage>(
          name: 'FieldInfoRoute',
          path: 'fieldInfo',
          page: FieldInfoPage,
        ),
      ],
    ),
    signInRouter,
    signUpRouter,
  ],
)

But the problem here is that after flutters "hot reload" it triggers the AppStartPage, and opens the HomePage again.
How do I make it so it doesn't open HomePage everytime it reloads?

I got it. After hours of seraching. The App Widget needs to be StatefulWidget for the hot-reload to work properly.
class App extends StatefulWidget {
  const App({super.key});

  @override
  State<App> createState() => _AppState();
}

class _AppState extends State<App> {
  final _appRouter = AppRouter();

  @override
  Widget build(BuildContext context) {
    return MaterialApp.router(
      theme: ThemeData(
        appBarTheme: const AppBarTheme(color: Color.blue),
        colorScheme: ColorScheme.fromSwatch(accentColor: Color.blue),
        // fontFamily: 'Roboto',
      ),
      routerDelegate: AutoRouterDelegate(
        _appRouter,
        navigatorObservers: () => [AppRouteObserver()],
      ),
      routeInformationProvider: _appRouter.routeInfoProvider(),
      routeInformationParser: _appRouter.defaultRouteParser(),
      localizationsDelegates: const [
        AppLocalizations.delegate,
        GlobalMaterialLocalizations.delegate,
      ],
      useInheritedMediaQuery: true,
      supportedLocales: AppLocalizations.supportedLocales,
      builder: (BuildContext context, Widget? child) {
        final data = MediaQuery.of(context);

        return MediaQuery(
          data: data.copyWith(
            textScaleFactor: 1,
          ),
          child: child!,
        );
      },
    );
  }
}