cmd/go: add mod tidy -check

Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (go version)?
go version go1.11rc1 darwin/amd64
Does this issue reproduce with the latest release?
N/A
What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOOS="darwin"

What did you do?

Remove lines from go.mod and go.sum files
Run go mod tidy to add them back

What did you expect to see?
go mod tidy should return non-zero as it made changes. This way it could be integrated to tools like pre-commit hooks to make sure that we don't commit untidied go.mod and go.sum files into version control
What did you see instead?
go mod tidy returned zero

Non-zero exit codes are usually for errors, so having it behave like this by default would be counter-intuitive.
Perhaps it could have something similar to gofmt, which has the -l and -d flags to tell if any changes were made.

If the files are successfully added why wouldn't you want to see a 0 status code?  This means that the following script would fail:
#!/usr/bin/env bash
set -e
go mod tidy  # <-- non 0 return code
echo 'we never get here'

That makes sense. How about maybe make it optional? e.g. when it's executed with a flag it returns non-zero when it made any changes.

@gopherbot, please add label modules

We could think about adding a flag that means "don't do the update, but fail if one is needed".
Not sure exactly what to call it but I put -check in the description. Needs more thought.

@rsc regarding your comment in #26850

You could use 'go list all -mod=readonly' to check that at least for the current GOOS/GOARCH no changes are needed.

go list all -mod=readonly causes changes for me to go.sum without erroring.
Is this a bug?

When this is implemented, we should run this on the TryBots (#31640).
Ideally go mod tidy -check would not access the network, since it will commonly be run in CI environments where network access is restricted. We might want to do so to provide diagnostics though. GOPROXY=off is still a good signal that we shouldn't access the network.

This is a feature that we would like to see for our CI environment as well (we want to verify that go.mod and go.sum are in the proper state for all submissions for all possible build constraints).
I echo the sentiment expressed by @jayconrod that it would be ideal if this check could be performed without network access -- is it feasible for the -check mode to verify only that no change needs to be made without making network queries? My intuition is that the information provided by the source code, go.mod and go.sum should be sufficient to make this determination, but I'm not completely sure about this.

It's would be useful to reduce git conflict in lint scenario

Networkless operation is a nice to have. Dry-run flags and intuitive exit codes are perhaps a more immediate concern.

How about maybe make it optional? e.g. when it's executed with a flag it should return non-zero when it made any changes.

This sounds to me like the solution that is most similar to how other tools solve this problem. golint uses this pattern already, and so does other tools like git diff. I'd prefer this over a -check command that alters the other behaviors of the tool because by altering the behavior of the tool there is more to learn and understand.
It'd be good to surface an exit code mode consistently across all the Go tool commands that we routinely as checks in CI pipelines.
Today running checks for go mod, go fmt, and golint all use unique and unintuitive patterns:
OUTPUT=$(gofmt -d .)
if [[ $OUTPUT ]]; then
    exit 1
fi
golint -set_exit_status
go mod tidy
git diff --exit-code -- go.mod go.sum

We could reuse the pattern demonstrated by golint:
golint -set_exit_status
go fmt -set_exit_status
go mod tidy -set_exit_status

Or, we could take the opportunity to use a term that is used by other tools, like -exit-code:
golint -exit-code
go fmt -exit-code
go mod tidy -exit-code

@leighmcculloch that is an interesting idea, but ultimately I think it's an over-engineered one. Many Go commands, by definition, already error when the check they are designed to do fails. This includes go vet, go test, go mod verify, go get, and many others. Would all of those gain an "exit code" flag that would be a no-op? That would be confusing. Would only some commands gain the common flag? That might be worse.
I also disagree that it would help for gofmt, for example. In that case, checking the output of gofmt -l is actually useful, as it helps give the end user a more useful error. You can argue the script could run the tool twice, but that's not a good solution either.
While I agree with the sentiment of making all commands more predictable, in practice there is just one command that needs this kind of fix: go mod tidy. I think we should focus there. My next comment will try to recap the conversation so far to try to move towards a decision.

@leighmcculloch that is an interesting idea, but ultimately I think it's an over-engineered one. Many Go commands, by definition, already error when the check they are designed to do fails. This includes go vet, go test, go mod verify, go get, and many others. Would all of those gain an "exit code" flag that would be a no-op? That would be confusing. Would only some commands gain the common flag? That might be worse.
I also disagree that it would help for gofmt, for example. In that case, checking the output of gofmt -l is actually useful, as it helps give the end user a more useful error. You can argue the script could run the tool twice, but that's not a good solution either.
While I agree with the sentiment of making all commands more predictable, in practice there is just one command that needs this kind of fix: go mod tidy. I think we should focus there. My next comment will try to recap the conversation so far to try to move towards a decision.

To recap: We seem to have consensus that this should be behind a flag.
On yesterday's tools call, @jayconrod wondered if it would be helpful to print more user-friendly output, like a diff. @bcmills says we should make a decision on that at the same time we decide on the flag name. It would make little sense for an -exit-code flag to also print a diff, for example.
Personally, I think we should just name the flag -check and have it return a short error, telling the user that the module is not tidy and how to fix it. The user will most likely have to run go mod tidy to fix the problem anyway, so I don't think a diff will be very helpful. Moreover, the diff would likely include go.sum too, so the size could get out of hand quickly with large projects, making the command tricky to use in CI.
Please give a thumbs up if you agree, or a thumbs down if you don't - along with a comment explaining why :) Thanks!

To recap: We seem to have consensus that this should be behind a flag.
On yesterday's tools call, @jayconrod wondered if it would be helpful to print more user-friendly output, like a diff. @bcmills says we should make a decision on that at the same time we decide on the flag name. It would make little sense for an -exit-code flag to also print a diff, for example.
Personally, I think we should just name the flag -check and have it return a short error, telling the user that the module is not tidy and how to fix it. The user will most likely have to run go mod tidy to fix the problem anyway, so I don't think a diff will be very helpful. Moreover, the diff would likely include go.sum too, so the size could get out of hand quickly with large projects, making the command tricky to use in CI.
Please give a thumbs up if you agree, or a thumbs down if you don't - along with a comment explaining why :) Thanks!

I think it would be consistent to use -mod=readonly instead of -check. It was raised in https://github.com/golang/go/issues/26850 which defers to this issue.

I think it would be consistent to use -mod=readonly instead of -check. It was raised in https://github.com/golang/go/issues/26850 which defers to this issue.

None of the go mod subcommands accept -mod right now, and -mod=readonly will be the default for build commands in 1.16. I think it would be surprising to use -mod=readonly here; -check seems more natural to me.

None of the go mod subcommands accept -mod right now, and -mod=readonly will be the default for build commands in 1.16. I think it would be surprising to use -mod=readonly here; -check seems more natural to me.

We could also consider naming the flag -n, analogous to go build -n, and have it print a summary of the requirements that would be added and removed.

We could also consider naming the flag -n, analogous to go build -n, and have it print a summary of the requirements that would be added and removed.

Outputting something like a diff of what would change would be very useful, and would make it easier to understand why the command failed.

Outputting something like a diff of what would change would be very useful, and would make it easier to understand why the command failed.

Did you see my reasoning above against diffs? It's not uncommon for large projects to have go.sum files in the thousands of lines, for example. A diff output could get large to the point that the end user can't even see what command failed due to pagination/truncation. It's also not clear that a diff would help that much; a message like "your module is not tidy, run go mod tidy" should be plenty.

Did you see my reasoning above against diffs? It's not uncommon for large projects to have go.sum files in the thousands of lines, for example. A diff output could get large to the point that the end user can't even see what command failed due to pagination/truncation. It's also not clear that a diff would help that much; a message like "your module is not tidy, run go mod tidy" should be plenty.

It seems like go mod tidy -diff would print useful information in a variety of contexts, and the exit code would get the boolean that people want. Adding -check instead would only provide one bit of information, which is less useful.
Let's add -diff.

It seems like go mod tidy -diff would print useful information in a variety of contexts, and the exit code would get the boolean that people want. Adding -check instead would only provide one bit of information, which is less useful.
Let's add -diff.

(And if the diff is huge, all the more reason to see it!)

(And if the diff is huge, all the more reason to see it!)

The key point for me with the huge diffs is that their relevance is extremely variable depending on the context. Within automation (e.g CI), the people writing the scripts and their end-users tend to only care about the binary value provided by the exit-code. Add to that as @mvdan puts it that many of those CI systems will truncate the huge diff anyway rendering it useless despite all its good intent.
It's only when manually invoked that you tend to potentially be interested in the content of that huge diff. But in those cases, on your local workstation, you can get the same result already by running go mod tidy and using your VCS's built-in diff abilities to get the same result. Hence I do not necessarily see the value add of a go mod tidy -diff over the proposed go mod tidy -check combined with other one-line commands that give the same result.
The one thing where I could see the value add of go mod tidy -diff is if its not a literal diff but rather a summarised report over what changed and why. That would however be a much more complex diagnostic tool to write than a simple check as is the aim of this issue.

The key point for me with the huge diffs is that their relevance is extremely variable depending on the context. Within automation (e.g CI), the people writing the scripts and their end-users tend to only care about the binary value provided by the exit-code. Add to that as @mvdan puts it that many of those CI systems will truncate the huge diff anyway rendering it useless despite all its good intent.
It's only when manually invoked that you tend to potentially be interested in the content of that huge diff. But in those cases, on your local workstation, you can get the same result already by running go mod tidy and using your VCS's built-in diff abilities to get the same result. Hence I do not necessarily see the value add of a go mod tidy -diff over the proposed go mod tidy -check combined with other one-line commands that give the same result.
The one thing where I could see the value add of go mod tidy -diff is if its not a literal diff but rather a summarised report over what changed and why. That would however be a much more complex diagnostic tool to write than a simple check as is the aim of this issue.

I still don't see the benefit to -diff over -check - I share @Helcaraxan's concerns. Another potential issue is consistency; gofmt -d prints a diff but the exit code does not correlate with the printing of a non-empty diff. There have been a few issues in the past to ask gofmt -d to return a non-exit status when code would be changed by gofmt, but as far as I know all such suggestions have been rejected.

I still don't see the benefit to -diff over -check - I share @Helcaraxan's concerns. Another potential issue is consistency; gofmt -d prints a diff but the exit code does not correlate with the printing of a non-empty diff. There have been a few issues in the past to ask gofmt -d to return a non-exit status when code would be changed by gofmt, but as far as I know all such suggestions have been rejected.

All that said, I think we really need this feature so I won't insist on the details. I just wanted to make my position clear that I don't see how we're getting to "print a diff".

All that said, I think we really need this feature so I won't insist on the details. I just wanted to make my position clear that I don't see how we're getting to "print a diff".

I think what is described with -diff is great 🎉. Diffs are useful because it means the developer doesn't need to do anything other than look at CI to know what's wrong. I use CI scripts to output diffs even if the tool doesn't support them, for example:
https://github.com/stellar/go/blob/9791cbf/gomod.sh#L4-L6
If someone doesn't want the diff and just the error code they can always pipe stdout to /dev/null.
-diff would do what I want, but these could also be composed:

-check set exit code to non-zero if there's a diff
-diff outputs diff

It's also worth noting there is some inconsistency with other tools such as git diff and golint which won't return a non-zero exit code unless parameters are passed (e.g. https://github.com/golang/go/issues/27005#issuecomment-564892876) and consistency could be nice for familiarity moving between tools.
@rsc If we go with -diff would go mod tidy still update the files as go mod tidy does today? Or would it not?

I think what is described with -diff is great 🎉. Diffs are useful because it means the developer doesn't need to do anything other than look at CI to know what's wrong. I use CI scripts to output diffs even if the tool doesn't support them, for example:
https://github.com/stellar/go/blob/9791cbf/gomod.sh#L4-L6
If someone doesn't want the diff and just the error code they can always pipe stdout to /dev/null.
-diff would do what I want, but these could also be composed:

-check set exit code to non-zero if there's a diff
-diff outputs diff

It's also worth noting there is some inconsistency with other tools such as git diff and golint which won't return a non-zero exit code unless parameters are passed (e.g. https://github.com/golang/go/issues/27005#issuecomment-564892876) and consistency could be nice for familiarity moving between tools.
@rsc If we go with -diff would go mod tidy still update the files as go mod tidy does today? Or would it not?

@leighmcculloch regarding your CI story I understand and agree with the general perspective that diffs can be useful. However in my experience the usefulness correlates strongly with the information contained in the diff being relevant to the user action that needs to follow. This happens usually when there are multiple courses of action available with the developer having to use their knowledge to decide what to do.
In the case that we are discussing here the only possible user action that follows the CI failure is to run go mod tidy and push the result to the branch. There is no need for "developer knowledge" to decide what to do. It's a simple one rule action with no conditionality. Indicating to a user that's the thing to do would actually be obscured by a long diff and would instead benefit from a short and clean error message instead.

@leighmcculloch regarding your CI story I understand and agree with the general perspective that diffs can be useful. However in my experience the usefulness correlates strongly with the information contained in the diff being relevant to the user action that needs to follow. This happens usually when there are multiple courses of action available with the developer having to use their knowledge to decide what to do.
In the case that we are discussing here the only possible user action that follows the CI failure is to run go mod tidy and push the result to the branch. There is no need for "developer knowledge" to decide what to do. It's a simple one rule action with no conditionality. Indicating to a user that's the thing to do would actually be obscured by a long diff and would instead benefit from a short and clean error message instead.

In the case that we are discussing here the only possible user action that follows the CI failure is to run go mod tidy and push the result to the branch.

@Helcaraxan It matters why go mod tidy is complaining and what changes it wants to make. It sounds like you are saying that users shouldn't even bother looking at the changes being made by go mod tidy to go.mod and go.sum, that they should just blindly accept the results. If that's really what you are saying, that's not good advice. In contrast, go mod tidy -diff to show exactly what it would do is very helpful for evaluating whether you want those changes or not.
Part of the reason we rejected gofmt -d setting an exit code is that CI systems should not be breaking due to non-semantic things like code having a stray trailing space. In contrast, go mod tidy is concerned with quite important semantic issues, and having go mod tidy -diff set the same exit code as diff(1) (0 for no changes, 1 for changes) is fine with me. Then if CI systems use go mod tidy -diff as the condition, you get both the exit code and information about what is different. That's exactly what users should want.
To the extent that the arguments for go mod tidy -check seem to encourage a mindset of "go.sum and go.mod are a black box that no one understands and you just run go mod tidy a bunch and pay no attention to exactly what is in those files", I am strongly opposed to that.

In the case that we are discussing here the only possible user action that follows the CI failure is to run go mod tidy and push the result to the branch.

@Helcaraxan It matters why go mod tidy is complaining and what changes it wants to make. It sounds like you are saying that users shouldn't even bother looking at the changes being made by go mod tidy to go.mod and go.sum, that they should just blindly accept the results. If that's really what you are saying, that's not good advice. In contrast, go mod tidy -diff to show exactly what it would do is very helpful for evaluating whether you want those changes or not.
Part of the reason we rejected gofmt -d setting an exit code is that CI systems should not be breaking due to non-semantic things like code having a stray trailing space. In contrast, go mod tidy is concerned with quite important semantic issues, and having go mod tidy -diff set the same exit code as diff(1) (0 for no changes, 1 for changes) is fine with me. Then if CI systems use go mod tidy -diff as the condition, you get both the exit code and information about what is different. That's exactly what users should want.
To the extent that the arguments for go mod tidy -check seem to encourage a mindset of "go.sum and go.mod are a black box that no one understands and you just run go mod tidy a bunch and pay no attention to exactly what is in those files", I am strongly opposed to that.

@Helcaraxan It matters why go mod tidy is complaining and what changes it wants to make. It sounds like you are saying that users shouldn't even bother looking at the changes being made by go mod tidy to go.mod and go.sum, that they should just blindly accept the results. If that's really what you are saying, that's not good advice. In contrast, go mod tidy -diff to show exactly what it would do is very helpful for evaluating whether you want those changes or not.

I fully agree on that point. My intent was and is not to say that people should ignore the changes induced by running go mod tidy, on the contrary. I am sorry if that's how my phrasing has been interpreted. The core of my argument is about the manner in which those changes are being presented to users. And I'm arguing there that presenting a raw diff will, in my opinion and experience, have either a neutral or even a negative effect in the longer run.
If our goal is to make developers understand the induced changes then they need more than a diff. Hence my earlier suggestion about an alternative (but more complex) behaviour for the -diff flag.

The one thing where I could see the value add of go mod tidy -diff is if its not a literal diff but rather a summarised report over what changed and why.

The behaviour of go mod tidy is unintuitive for most users as it goes beyond managing direct or indirect dependencies and it considers the entire dependency graph before applying MVS, not just the build list. This is something that comes up more often than not in my day-to-day experience at my place of work, on issues on OSS projects and on the Go Slack as an element that is not understood at all. With the exception being the very few who have been actively working on Go modules itself or related tooling over the past year(s).
As a result the current pragmatic realisation is that the average, or even more advanced, Go developer is not (yet) likely to understand the why of the changes just based on a raw diff. Simply presenting a diff will de facto result in people continuing to consider modules, the behaviour of go mod tidy as well as the content of go.sum (and to a lesser degree go.mod) as black boxes. It actually can disengage developers from learning to understand the modules system better as their default experience would be to be faced by a wall of text with no clue on how to make sense of it.
Put differently: Getting a raw diff is already possible and easy with existing tooling (go mod tidy && git diff --exit-code or equivalent), I'd rather have an actual go mod tidy -diff aspire to give higher level information than an external diff tool given that it's part of the language's own toolchain. If the choice is between it outputting a raw diff or not implementing any check/diff feature at all I'd probably vote for the latter.

@Helcaraxan It matters why go mod tidy is complaining and what changes it wants to make. It sounds like you are saying that users shouldn't even bother looking at the changes being made by go mod tidy to go.mod and go.sum, that they should just blindly accept the results. If that's really what you are saying, that's not good advice. In contrast, go mod tidy -diff to show exactly what it would do is very helpful for evaluating whether you want those changes or not.

I fully agree on that point. My intent was and is not to say that people should ignore the changes induced by running go mod tidy, on the contrary. I am sorry if that's how my phrasing has been interpreted. The core of my argument is about the manner in which those changes are being presented to users. And I'm arguing there that presenting a raw diff will, in my opinion and experience, have either a neutral or even a negative effect in the longer run.
If our goal is to make developers understand the induced changes then they need more than a diff. Hence my earlier suggestion about an alternative (but more complex) behaviour for the -diff flag.

The one thing where I could see the value add of go mod tidy -diff is if its not a literal diff but rather a summarised report over what changed and why.

The behaviour of go mod tidy is unintuitive for most users as it goes beyond managing direct or indirect dependencies and it considers the entire dependency graph before applying MVS, not just the build list. This is something that comes up more often than not in my day-to-day experience at my place of work, on issues on OSS projects and on the Go Slack as an element that is not understood at all. With the exception being the very few who have been actively working on Go modules itself or related tooling over the past year(s).
As a result the current pragmatic realisation is that the average, or even more advanced, Go developer is not (yet) likely to understand the why of the changes just based on a raw diff. Simply presenting a diff will de facto result in people continuing to consider modules, the behaviour of go mod tidy as well as the content of go.sum (and to a lesser degree go.mod) as black boxes. It actually can disengage developers from learning to understand the modules system better as their default experience would be to be faced by a wall of text with no clue on how to make sense of it.
Put differently: Getting a raw diff is already possible and easy with existing tooling (go mod tidy && git diff --exit-code or equivalent), I'd rather have an actual go mod tidy -diff aspire to give higher level information than an external diff tool given that it's part of the language's own toolchain. If the choice is between it outputting a raw diff or not implementing any check/diff feature at all I'd probably vote for the latter.

I wanted to offer an observation in hopes that it may be helpful to this discussion. Changes to the go.mod file are reasonably well understood, as they're usually in direct response to adding or updating a module. However, this isn't always true of go.sum changes.
Occasionally, changes to .go files in our code base have caused the go.sum file to become untidy (without changes to the go.mod file). Although I didn't investigate at the time, I would guess this is because we started relying on a new package that we hadn't previously that already existed somewhere in the module graph.
Furthermore, while valuable, the data in go.sum is difficult to verify or update directly.  Knowing that it's out of date is very useful, but I would argue that the contents of the diff matter less (at least in the common case).
So, I would argue that there is tremendous value in showing the diff of the go.mod file. But perhaps there is less value in showing the diff of the go.sum file, although it should probably still be indicated when it is out of date.
I suppose this begs the question: when go mod tidy is run, will it change the digest of a module without complaining? (i.e. if a particular version already existed in go.sum, but when calculated locally it was different, would it complain rather than updating it?)

Aside: our CI system currently has a 64 line Bash function to flag untidy modules that would largely be supplanted by go mod tidy -diff (coupled with defaulting to -mod=readonly), so this seems very useful. A good portion of that function is ensuring that the diff gets output so we know what got missed.

I wanted to offer an observation in hopes that it may be helpful to this discussion. Changes to the go.mod file are reasonably well understood, as they're usually in direct response to adding or updating a module. However, this isn't always true of go.sum changes.
Occasionally, changes to .go files in our code base have caused the go.sum file to become untidy (without changes to the go.mod file). Although I didn't investigate at the time, I would guess this is because we started relying on a new package that we hadn't previously that already existed somewhere in the module graph.
Furthermore, while valuable, the data in go.sum is difficult to verify or update directly.  Knowing that it's out of date is very useful, but I would argue that the contents of the diff matter less (at least in the common case).
So, I would argue that there is tremendous value in showing the diff of the go.mod file. But perhaps there is less value in showing the diff of the go.sum file, although it should probably still be indicated when it is out of date.
I suppose this begs the question: when go mod tidy is run, will it change the digest of a module without complaining? (i.e. if a particular version already existed in go.sum, but when calculated locally it was different, would it complain rather than updating it?)

Aside: our CI system currently has a 64 line Bash function to flag untidy modules that would largely be supplanted by go mod tidy -diff (coupled with defaulting to -mod=readonly), so this seems very useful. A good portion of that function is ensuring that the diff gets output so we know what got missed.

My 2¢: go mod tidy -diff should print a diff and exit non-zero if there are any.
I don't think have separate -check and -diff flags is necessary unless the output is qualitatively different. If the user just wants the exit code, the output can be piped to /dev/null.

If our goal is to make developers understand the induced changes then they need more than a diff. Hence my earlier suggestion about an alternative (but more complex) behaviour for the -diff flag.

@Helcaraxan This is interesting, but there may be a lot here. Some possible changes go mod tidy can make:

Formatting, sorting, removing duplicates, and other non-semantic changes in go.mod and go.sum
Add go directive
Add missing require directives
Remove unnecessary require directives
Update inconsistent require directives (when the version actually used is higher)
Canonicalize versions (replace versions like devbranch with a pseudo-version corresponding to that commit)
Add missing sums in go.sum
Delete unused sums in go.sum

Describing all those changes in a friendly way may be more in the wheelhouse of gorelease. There is a bit of overlap here though.

I suppose this begs the question: when go mod tidy is run, will it change the digest of a module without complaining? (i.e. if a particular version already existed in go.sum, but when calculated locally it was different, would it complain rather than updating it?)

@miquella go mod tidy (and all other module commands) reports a security error if the locally calculated sum of a downloaded module is different than an existing sum in go.sum. Usually that means a semver tag was moved to a different commit, but it could also mean the module code was tampered with.

My 2¢: go mod tidy -diff should print a diff and exit non-zero if there are any.
I don't think have separate -check and -diff flags is necessary unless the output is qualitatively different. If the user just wants the exit code, the output can be piped to /dev/null.

If our goal is to make developers understand the induced changes then they need more than a diff. Hence my earlier suggestion about an alternative (but more complex) behaviour for the -diff flag.

@Helcaraxan This is interesting, but there may be a lot here. Some possible changes go mod tidy can make:

Formatting, sorting, removing duplicates, and other non-semantic changes in go.mod and go.sum
Add go directive
Add missing require directives
Remove unnecessary require directives
Update inconsistent require directives (when the version actually used is higher)
Canonicalize versions (replace versions like devbranch with a pseudo-version corresponding to that commit)
Add missing sums in go.sum
Delete unused sums in go.sum

Describing all those changes in a friendly way may be more in the wheelhouse of gorelease. There is a bit of overlap here though.

I suppose this begs the question: when go mod tidy is run, will it change the digest of a module without complaining? (i.e. if a particular version already existed in go.sum, but when calculated locally it was different, would it complain rather than updating it?)

@miquella go mod tidy (and all other module commands) reports a security error if the locally calculated sum of a downloaded module is different than an existing sum in go.sum. Usually that means a semver tag was moved to a different commit, but it could also mean the module code was tampered with.

@rsc If we go with -diff would go mod tidy still update the files as go mod tidy does today? Or would it not?

I think my question got missed. Is the intention with -diff that it changes status code and outputs a diff in addition to mutating the files, or when used would it not mutate the files?

@rsc If we go with -diff would go mod tidy still update the files as go mod tidy does today? Or would it not?

I think my question got missed. Is the intention with -diff that it changes status code and outputs a diff in addition to mutating the files, or when used would it not mutate the files?

I think my question got missed. Is the intention with -diff that it changes status code and outputs a diff in addition to mutating the files, or when used would it not mutate the files?

I think the intent is for -diff (or -check) not to mutate files. It would be analogous to -mod=readonly in other commands: fail if any changes are needed.

I think my question got missed. Is the intention with -diff that it changes status code and outputs a diff in addition to mutating the files, or when used would it not mutate the files?

I think the intent is for -diff (or -check) not to mutate files. It would be analogous to -mod=readonly in other commands: fail if any changes are needed.

That suggests an alternate spelling: go mod diff -tidy, with a corresponding go mod diff that could report only needed additions and cleanups (like go list -mod=mod -test -deps all).

That suggests an alternate spelling: go mod diff -tidy, with a corresponding go mod diff that could report only needed additions and cleanups (like go list -mod=mod -test -deps all).

(To be clear, I have no particular preference between go mod tidy -diff and go mod diff -tidy.)

(To be clear, I have no particular preference between go mod tidy -diff and go mod diff -tidy.)

@Helcaraxan This is interesting, but there may be a lot here. Some possible changes go mod tidy can make:

Formatting, sorting, removing duplicates, and other non-semantic changes in go.mod and go.sum
Add go directive
Add missing require directives
Remove unnecessary require directives
Update inconsistent require directives (when the version actually used is higher)
Canonicalize versions (replace versions like devbranch with a pseudo-version corresponding to that commit)
Add missing sums in go.sum
Delete unused sums in go.sum

Describing all those changes in a friendly way may be more in the wheelhouse of gorelease. There is a bit of overlap here though.

@jayconrod agreed that there's quite a lot of different things that can happen for various reasons and it would go beyond the current request of having a simple -check command.
I am a bit hesitant about considering this part of gorelease. Such diff explanations have no particular link with releasing as they are part of a general Go-based workflow. Hence why I'd argue that putting it in the go command makes sense. It boils down to adding user-visible annotations to logic that the toolchain already has.
This is also another reason why I'd continue to argue against the -diff proposal. If we implement that one now than what would it look like to add this more complex diff-behaviour? Do we add another flag that has exactly the same semantics as the -diff one but with just a different output (that seems confusing)? Do we consider changing the output of -diff (which would be a breaking change)? The best option I can think of is to use something akin to the -x flag on other commands that increases output verbosity.

@Helcaraxan This is interesting, but there may be a lot here. Some possible changes go mod tidy can make:

Formatting, sorting, removing duplicates, and other non-semantic changes in go.mod and go.sum
Add go directive
Add missing require directives
Remove unnecessary require directives
Update inconsistent require directives (when the version actually used is higher)
Canonicalize versions (replace versions like devbranch with a pseudo-version corresponding to that commit)
Add missing sums in go.sum
Delete unused sums in go.sum

Describing all those changes in a friendly way may be more in the wheelhouse of gorelease. There is a bit of overlap here though.

@jayconrod agreed that there's quite a lot of different things that can happen for various reasons and it would go beyond the current request of having a simple -check command.
I am a bit hesitant about considering this part of gorelease. Such diff explanations have no particular link with releasing as they are part of a general Go-based workflow. Hence why I'd argue that putting it in the go command makes sense. It boils down to adding user-visible annotations to logic that the toolchain already has.
This is also another reason why I'd continue to argue against the -diff proposal. If we implement that one now than what would it look like to add this more complex diff-behaviour? Do we add another flag that has exactly the same semantics as the -diff one but with just a different output (that seems confusing)? Do we consider changing the output of -diff (which would be a breaking change)? The best option I can think of is to use something akin to the -x flag on other commands that increases output verbosity.

@bcmills This has a 1.18 milestone; is it likely to happen for 1.18?  Thanks.

I think I've missed the freeze on this one, unfortunately. (I tried to knock out #27628 first because it was smaller, and instead got bogged down in existing bugs — #49267 — interfering with the tests for that.)
Moving to 1.19.

@bcmills The issue is added to the G1.19 milestone; Is it likely to happen for 1.19? Thanks

@samthanawalla, this should be a fairly straightforward feature to implement, and would be useful for CI tooling.
(This was in the NeedsDecision state pending a final decision about what the command-line flag should be named; @matloob and @rsc may have some thoughts on that.)

@rsc Should we proceed with -diff or wait for the proposal committee to review?

Given the discussion I think it is fine to proceed with -diff.


Change SourceFiles to be Fetch.Lazy

Is your feature request related to a problem? Please describe.
Source files can consume a large amount of memory; we should avoid fetching them unless needed by the client.
Describe the solution you'd like

In WorkflowVersion, annotate sourceFiles with FetchType.LAZY instead of FetchType.EAGER.
That may not be enough though; if anything access the source files they'll end up getting loaded anyways.
Force fetching of sourceFiles via a new endpoint? Or via a query parameter?
UI apparently doesn't use source files; except for the hosted entries.
May require UI corresponding UI changes

Additional context
Overlaps with #3453 and #3442 . This might end up being addressed in either or both of those instead.
┆Issue is synchronized with this Jira Story
┆Issue Type: Story
┆Sprint: Sprint 35 Jellyfish
┆Issue Number: DOCK-1355

Yeah, a lot of things are broken. Search, my tools, my workflows, individual tool and workflow pages are varying levels of broken.

Might be best to start with individual tool and workflow pages to see/debug the extent of the fixes needed since @garyluu may eventually be tinkering with my tools and my workflows due to https://ucsc-cgl.atlassian.net/browse/DOCK-1353

Played around with this a bit.  Funny, we do use sourcefiles for a variety of places...but not for the file contents.  it's to determine the descriptor type so that we can use the TRS.

Yeah, I think we also use it to get the verified source for each version. I'm not sure what's going on the with my-tools page since absolutely nothing loads.

For review, I realize that we went a different direction with sourcefiles by adding a new endpoint
But this means that we will need to actually make source file lazy in 1.10?
@NatalieEO please confirm

Yes that's correct. I'll take the review label off of this.


unable to load .dylib in flutter app on ios platform

Hi,
I am working on a flutter application using an android emulator.
I have used the following dart code to load the ".dylib" file.
var libraryPath = path.join(
Directory.current.path, 'primitives_library', 'TestLibrary.dylib');
final dylib = DynamicLibrary.open(libraryPath);
but when I build the project it gives the following error.
DynamicLibrary.open error: Invalid argument(s): Failed to load dynamic library (126)
Please suggest any solution to load the ".dylib" library using an ios emulator.
Please suggest how to create a dylib library to use in the flutter app on the ios platform.

The dylibs need to be compiled to different target architectures and OSes.
And they need to be bundled the right way in Flutter.
If you have C/C++ source code, then please take a look at flutter create --template=plugin_ffi --platform=android,ios <name>.
If you have precompiled libraries, then you need to modify the Gradle/CocoaPods/CMake files generated in the plugin_ffi template.
Currently packages cannot be Dart standalone and work for Flutter. https://github.com/dart-lang/sdk/issues/50565 tracks our effort to solve this.


ERROR using ftINIT. MipGap too high, trying with a different run.

Hello,
I am trying to create context-specific GEM models for my transcriptomic data using the Cobra toolbox and gurobi solver in Matlab. I have created models before using the same Human-GEM guide. Now it's showing me some error due to high MipGap and I feel it is because of the solver.
command for creating model:
model1 = ftINIT(prepData, data_struct.tissues{1}, [], [], data_struct, {}, getHumanGEMINITSteps('1+0'), false, true);

The error shown after running this is:
ftINIT: Running step 1
MILP detected.
MipGap too high, trying with a different run. MipGap = Inf New MipGap Limit = 0.003
MILP detected.
Error using dispEM
Failed to find good enough solution within the time frame. MIPGap: Inf

Error in ftINIT (line 281)
        dispEM(['Failed to find good enough solution within the time frame. MIPGap: ' num2str(mipGap)]);

Any help would be appreciated.
Thank you.

@ShwetaNagre Did you get past this issue?

Hi @johan-gson,
Yes, it was working afterward with the same code.

Ok, I'm closing this now.

Hi @ShwetaNagre could I ask how you got past this issue since I've now run into the same problem myself while trying to run ftINIT on windows

Ok, no worries, good that you got it to work! Good luck with your project!
On Sat, Apr 1, 2023 at 2:17 PM manas-kohli @.***> wrote:

Hi johan, I did update gurobi recently but actually realised I just made a
stupid mistake and had two versions of RAVEN floating around and so that
produced the error. Sometimes it's the simple things but I reckon if others
get this error, it probably can be due to conflicting versions of
RAVEN/Gurobi. Thanks for the quick response though, I do appreciate it
—
Reply to this email directly, view it on GitHub
https://github.com/SysBioChalmers/Human-GEM/issues/511#issuecomment-1493065861,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AHX2AK6FD7LVKUSLY5TGWCLW7BWJ7ANCNFSM6AAAAAAWE6YHOM
.
You are receiving this because you modified the open/close state.Message
ID: @.***>

Sorry for the late reply @manas-kohli, good that you got this cleared.
This error was indeed due to the versions of RAVEN/Gurobi. I too updated my RAVEN and then it worked.
Hey @johan-gson can these RAVEN version specifications be mentioned on the HUMAN-GEM guide as well? So, that it would be more helpful to others. Cause when I created some models in Feb'23 it worked well with it and just showed the problem with an updated version of RAVEN.
Thank you.

@ShwetaNagre Sorry, forgot to answer this. Could you clarify - is it with the newer version of RAVEN you get problems?

I think @edkerk mentioned that RAVEN is going to have an "auto-update" function that will make sure RAVEN is running the latest version, so perhaps this part doesn't need to be added to the guide. A note on Gurobi on the other hand might be useful.

It should work fine with the new version (10) of Gurobi after a change in one of the parameters, so there should be no need for that either.

It should work fine with the new version (10) of Gurobi after a change in one of the parameters, so there should be no need for that either.

Right, but atm there is nothing specified on the Gurobi version during installation. My suggestion would be to at least add in the word "latest version of the [Gurobi Optimizer]".

@mihai-sysbio No auto-update, checkInstallation just reports if there is a newer version and nudges the user to update.

@johan-gson Sorry to bring this up again,  could you tell me what change did you make in "after a change in one of the parameters, so there should be no need for that either."?

I have ran into the same problem with Gurobi 10.0.1 and RAVEN 2.8.1, Matlab R2019b, and i am confused with the other's solutions.
ftINIT: Running step 1 MILP detected. MipGap too high, trying with a different run. MipGap = Inf New MipGap Limit = 0.003 MILP detected. Error using dispEM (line 49) Failed to find good enough solution within the time frame. MIPGap: Inf Error in ftINIT (line 281)         dispEM(['Failed to find good enough solution within the time         frame. MIPGap: ' num2str(mipGap)]);
Thank you very much in advance!!

@MengJ-bioDyn If you send me your code and the data I could test to run it and see if I can make it work. It can be multiple things. One thing is to make sure that the genes are the same - if you created the prepData with ENSEMBL, it has to be ENSEMBL etc. Also important to remove the versions of the ENSEMBL genes, i.e., if ENSG0000000001111111.2 then remove .2 from the gene for all genes. I think I should have fixed the Gurobi thing in the code, so if there is still a problem with that it is a new bug.

My email address is<EMAIL_ADDRESS>if you are willing to send your data and code.

Hi Johan-gson,
Thank you so much!
Here is my data, in TPM unit. My code is attached as well.
Each column after "ASPC1_DTP_avg" is for a cell_line under a condition. Our
aim is to build customized GEMs for each cell line for both DTP and
Parental conditions. We are eager to apply the cool methods you develop!
Somehow my data is too big. I compressed it, let me know if you have any
problems with it! I can also share it via google drive.
Thank you very much again!!
Best regards,
Meng
On Fri, Jun 16, 2023 at 6:29 PM johan-gson @.***> wrote:

My email address is @.*** if you are willing to send your
data and code.
—
Reply to this email directly, view it on GitHub
https://github.com/SysBioChalmers/Human-GEM/issues/511#issuecomment-1595567882,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADAI4EBLOSMMWGKS4XRBLFLXLUB7ZANCNFSM6AAAAAAWE6YHOM
.
You are receiving this because you were mentioned.Message ID:
@.***>

Hi Meng,
Not sure where the data is, could you by any chance share it via google drive?

Sure, no problem!
Here is the link:
https://drive.google.com/file/d/1xBj6Rq94Pd8tRFGig7PdmaKi3Zk8-2dD/view?usp=sharing
Thank you so much again!
Best regards,
Meng
On Fri, Jun 16, 2023 at 7:46 PM johan-gson @.***> wrote:

Hi Meng,
Not sure where the data is, could you by any chance share it via google
drive?
—
Reply to this email directly, view it on GitHub
https://github.com/SysBioChalmers/Human-GEM/issues/511#issuecomment-1595588148,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADAI4EB7UZSYDY7MCHC4VVTXLULCHANCNFSM6AAAAAAWE6YHOM
.
You are receiving this because you were mentioned.Message ID:
@.***>

ok, will have a look!

Hi, do you also have the code, I didn't see it? It looks like the data is in log (TPM + 1) and then centered, since you have negative values? Is that so?

Thank you! Right, the data is log(TPM +1). Can you see the code now?
I tried Gurobi 10.0.0, 10.0.1 and 10.0.2, all have the same problem... I
tried to install 9.5.2, but the current trial license I got doesn't support
9.5.2.
I saw other people have similar problems on the git issues, but somehow
they seemed to be able to solve it. That's why I'm very confused.
Again, I greatly appreciate your help!!
On Fri, Jun 16, 2023 at 8:10 PM johan-gson @.***> wrote:

Hi, do you also have the code, I didn't see it? It looks like the data is
in log (TPM + 1) and then centered, since you have negative values? Is that
so?
—
Reply to this email directly, view it on GitHub
https://github.com/SysBioChalmers/Human-GEM/issues/511#issuecomment-1595594093,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADAI4EECRAA6KX7F6H7ETM3XLUNYRANCNFSM6AAAAAAWE6YHOM
.
You are receiving this because you were mentioned.Message ID:
@.***>

Hi, I still don't see any code, did you attach a text file or something, I didn't see it? It is not a huge problem, I can use some old code I have. But, there is a problem with the data, there is just no way that is log(TPM + 1) since it contains negative numbers (log(TPM + 1) can never become negative). It doesn't look centered either? Is it log(TPM + x), where the pseudocount x is something smaller than 1? For example this row: ENSG00000000005,ENSG00000000005.5,TNMD,-3.444843513,-3.444843513,-3.444843513,-3.444843513,-3.444843513,-3.444843513,-3.388346749,-3.444843513
This is something you need to figure out, how the data was transformed. If you produced the data in your lab, that should be easy to do! I'll be happy to help once you figured that out so we can convert it back to TPM.

Hi!
Sorry for the late reply! I was out of town for the last two days. I
attached a .m file in the previous emails, but somehow it didn't show up on
your side. I pasted it at the end of this email. I double checked the data
process pipeline, the data is log2(TPM), not centered.
Inspired by what you mentioned, I checked to use 2.^(data) (convert to TPM)
as input for ftINIT, and so glad it worked!!
The command ftINIT(prepDataHumanGEM, new_data_struct.tissues{1}, [], [],
new_data_struct, {}, getHumanGEMINITSteps('1+0'), false, true) finished
without errors. So the issue is not caused by Gurobi 10.0.1 and Raven
2.8.1.  I'll update this to github.
Thank you so much! I deeply appreciate your time and your help!!
The tools you and your lab developed are very useful, and they are great
contributions to the whole scientific community!
Thank you and best regards,
Meng
Below is my code:
load('Human-GEM.mat');
model = ihuman;
% changeCobraSolver('glpk')
% setRavenSolver('cobra')
sol = solveLP(model)
% check MouseGEM biomass rxn lb & ub
rxns = [model.rxns];
biomassID_mouse = find(strcmp(rxns, 'MAR00021'))
model.rxns{biomassID_mouse}
[model.lb(biomassID_mouse) model.ub(biomassID_mouse)]
essentialTasksFilePath =
'./data/metabolicTasks/metabolicTasks_Essential.txt';
rxnsFilePath = './model/reactions.tsv';
convertGenes = false;
initial_run = false;
if initial_run
prepDataHumanGEM = prepHumanModelForftINIT(ihuman, convertGenes,

essentialTasksFilePath, rxnsFilePath)
save('prepData.mat', 'prepDataHumanGEM')

else
print('load previously generated prep model');

load('prepData.mat')

end
rnaseq_data = readtable('./data/PANC_TPM_withgenename_avg_14d.csv');
tissueN = 4;
data_struct.tissues = rnaseq_data.Properties.VariableNames(4:end)';  %
sample (tissue) names
data_struct.genes = rnaseq_data.gene_id;  % gene names
data_struct.levels = table2array(rnaseq_data(:, 4:end));  % gene TPM values
data_struct.threshold = 1;
%%  now this works!
level_test = data_struct.levels;
level_unlog2 = 2.^(level_test);
new_data_struct = data_struct;
new_data_struct.levels = level_unlog2;
new_data_struct.threshold = 1;
model1 = ftINIT(prepDataHumanGEM, new_data_struct.tissues{1}, [], [],
new_data_struct, {}, getHumanGEMINITSteps('1+0'), false, true);
model1.id = new_data_struct.tissues{1};
On Sat, Jun 17, 2023 at 7:17 AM johan-gson @.***> wrote:

Hi, I still don't see any code, did you attach a text file or something, I
didn't see it? It is not a huge problem, I can use some old code I have.
But, there is a problem with the data, there is just no way that is log(TPM

since it contains negative numbers (log(TPM + 1) can never become
negative). It doesn't look centered either? Is it log(TPM + x), where the
pseudocount x is something smaller than 1? For example this row:
ENSG00000000005,ENSG00000000005.5,TNMD,-3.444843513,-3.444843513,-3.444843513,-3.444843513,-3.444843513,-3.444843513,-3.388346749,-3.444843513
This is something you need to figure out, how the data was transformed. If
you produced the data in your lab, that should be easy to do! I'll be happy
to help once you figured that out so we can convert it back to TPM.

—
Reply to this email directly, view it on GitHub
https://github.com/SysBioChalmers/Human-GEM/issues/511#issuecomment-1595772217,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADAI4EAXXAWE7AWWDQBAVWTXLW37PANCNFSM6AAAAAAWE6YHOM
.
You are receiving this because you were mentioned.Message ID:
@.***>

Hi,
Glad that you could make it work! Just make sure that the transformation is
right, it does sound a bit fishy to me - In RNA-Seq you usually have a lot
of zeros, and those can not be log transformed like you described, will
give NaN or something like that. if you imputed those with a small value
you are fine, but otherwise there is something fishy going on. Just because
it could run doesn't mean that it is correctly run.
Good luck with your project and glad that I could help!
Johan
On Mon, Jun 19, 2023 at 7:10 PM Meng @.***> wrote:

Hi!
Sorry for the late reply! I was out of town for the last two days. I
attached a .m file in the previous emails, but somehow it didn't show up
on
your side. I pasted it at the end of this email. I double checked the data
process pipeline, the data is log2(TPM), not centered.
Inspired by what you mentioned, I checked to use 2.^(data) (convert to
TPM)
as input for ftINIT, and so glad it worked!!
The command ftINIT(prepDataHumanGEM, new_data_struct.tissues{1}, [], [],
new_data_struct, {}, getHumanGEMINITSteps('1+0'), false, true) finished
without errors. So the issue is not caused by Gurobi 10.0.1 and Raven
2.8.1. I'll update this to github.
Thank you so much! I deeply appreciate your time and your help!!
The tools you and your lab developed are very useful, and they are great
contributions to the whole scientific community!
Thank you and best regards,
Meng
Below is my code:
load('Human-GEM.mat');
model = ihuman;
% changeCobraSolver('glpk')
% setRavenSolver('cobra')
sol = solveLP(model)
% check MouseGEM biomass rxn lb & ub
rxns = [model.rxns];
biomassID_mouse = find(strcmp(rxns, 'MAR00021'))
model.rxns{biomassID_mouse}
[model.lb(biomassID_mouse) model.ub(biomassID_mouse)]
essentialTasksFilePath =
'./data/metabolicTasks/metabolicTasks_Essential.txt';
rxnsFilePath = './model/reactions.tsv';
convertGenes = false;
initial_run = false;
if initial_run
prepDataHumanGEM = prepHumanModelForftINIT(ihuman, convertGenes,
essentialTasksFilePath, rxnsFilePath)
save('prepData.mat', 'prepDataHumanGEM')
else
print('load previously generated prep model');
load('prepData.mat')
end
rnaseq_data = readtable('./data/PANC_TPM_withgenename_avg_14d.csv');
tissueN = 4;
data_struct.tissues = rnaseq_data.Properties.VariableNames(4:end)'; %
sample (tissue) names
data_struct.genes = rnaseq_data.gene_id; % gene names
data_struct.levels = table2array(rnaseq_data(:, 4:end)); % gene TPM values
data_struct.threshold = 1;
%% now this works!
level_test = data_struct.levels;
level_unlog2 = 2.^(level_test);
new_data_struct = data_struct;
new_data_struct.levels = level_unlog2;
new_data_struct.threshold = 1;
model1 = ftINIT(prepDataHumanGEM, new_data_struct.tissues{1}, [], [],
new_data_struct, {}, getHumanGEMINITSteps('1+0'), false, true);
model1.id = new_data_struct.tissues{1};
On Sat, Jun 17, 2023 at 7:17 AM johan-gson @.***> wrote:

Hi, I still don't see any code, did you attach a text file or something,
I
didn't see it? It is not a huge problem, I can use some old code I have.
But, there is a problem with the data, there is just no way that is
log(TPM

since it contains negative numbers (log(TPM + 1) can never become
negative). It doesn't look centered either? Is it log(TPM + x), where
the
pseudocount x is something smaller than 1? For example this row:

ENSG00000000005,ENSG00000000005.5,TNMD,-3.444843513,-3.444843513,-3.444843513,-3.444843513,-3.444843513,-3.444843513,-3.388346749,-3.444843513

This is something you need to figure out, how the data was transformed.
If
you produced the data in your lab, that should be easy to do! I'll be
happy
to help once you figured that out so we can convert it back to TPM.
—
Reply to this email directly, view it on GitHub
<
https://github.com/SysBioChalmers/Human-GEM/issues/511#issuecomment-1595772217>,

or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/ADAI4EAXXAWE7AWWDQBAVWTXLW37PANCNFSM6AAAAAAWE6YHOM>

.
You are receiving this because you were mentioned.Message ID:
@.***>

—
Reply to this email directly, view it on GitHub
https://github.com/SysBioChalmers/Human-GEM/issues/511#issuecomment-1597864803,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AHX2AK63W6VVJO5MS6ZSCSLXMDL7VANCNFSM6AAAAAAWE6YHOM
.
You are receiving this because you were mentioned.Message ID:
@.***>

Hi there, I am here to report the same error:
Error using dispEM
Failed to find good enough solution within the time frame. MIPGap: Inf

Error in ftINIT (line 281)
        dispEM(['Failed to find good enough solution within the time frame. MIPGap: ' num2str(mipGap)]);

I have went through all the comments, have the most recent RAVEN version (2.8.4) and Gurobi version 10.0.2.
My gene names are all in order, i.e. they are also ENSEMBL like so:

                GeneName                       AT_1      AT_2      AT_3      AT_4
    ______________________    ______    ______    ______    ______

    {'ENSMUSG00000000001'}    81.164    87.099    76.508    86.454
    {'ENSMUSG00000000003'}         0         0         0         0
    {'ENSMUSG00000000028'}    6.3361    17.716    6.8409    3.0719
    {'ENSMUSG00000000037'}    3.8911    9.5444    8.2382    4.5037
    {'ENSMUSG00000000049'}    4.2952    7.6122    1.6615    1.3399

Would appreciate if anyone has any feedback on this! Many thanks

Hi @rsango6, I think the solution to you problem is to convert the genes to gene symbols. While Human-GEM is in ensembl format, I think mouse-GEM, which I assume you are using, requires gene symbols. So just test to convert the genes and see if it works! Otherwise, let me know and we can dig further!

Hi @johan-gson, thanks for your speedy reply.
I forgot that Mouse-GEM has gene symbols instead! I have tried to do the following;
data_struct =

  struct with fields:

        genes: {29260×1 cell}
        tissues: {1×52 cell}
        levels: [29260×52 double]
        threshold: 1

and the genes are:

>> data_struct.genes(1:5)

ans =

  5×1 cell array

    {'Gnai3'}
    {'Pbsn' }
    {'Cdc45'}
    {'Scml2'}
    {'Apoh' }

but I still get the same error as before. Could it be that the this cell array that contains gene symbols, has to be transformed in some different structure before is fed to ftINIT?

Hmm, could you check in prepData.minModel, if the grRules uses gene symbols as well, just to make sure? And the data is in TPM?

Hmm, could you check in prepData.minModel, if the grRules uses gene symbols as well, just to make sure? And the data is in TPM?

would it be good to have some checks for gene formats?

Hmm, could you check in prepData.minModel, if the grRules uses gene symbols as well, just to make sure? And the data is in TPM?

Hello,
Yes, the data is in TPM, as obtained from Salmon quantification tool.
I have checked both prepData.refModel.grRules and prepData.minModel.grRules.
In prepData.refModel.grRules:
>> prepData.refModel.grRules(1:10)

ans =

  10×1 cell array

    {'Adh1 or Adh4 or Adh5 or Adh6b or Adh7 or Adhfe1'    }
    {'Akr1a1'                                             }
    {'Acss2'                                              }
    {'Acss1 or Acss3'                                     }
    {'Acss2'                                              }
    {'Acss2'                                              }
    {'Ldha or Ldhb'                                       }
    {'Ldha or Ldhal6b or Ldhb or Ldhc'                    }
    {'Aldh1a3 or Aldh3a1 or Aldh3b1 or Aldh3b2 or Aldh3b3'}
    {'Aldh1b1 or Aldh2 or Aldh3a2 or Aldh7a1 or Aldh9a1'  }

As for prepData.minModel.grRules, this is an just an empty array with no grRules. Could this be the reason for the error?

Yes, this doesn't sound good, that is most likely the error. Did you by any chance set convertGenes to true? That is not a good idea for Mouse-GEM, this should be false. If not, could you set a breakpoint in the code and see where they are cleared?

Yes, this doesn't sound good, that is most likely the error. Did you by any chance set convertGenes to true? That is not a good idea for Mouse-GEM, this should be false. If not, could you set a breakpoint in the code and see where they are cleared?

Hi,
No, I have left convertGenes as is, so this flag is false.
I also tried just running prepHumanModelForftINIT as shown in tutorial like so:
prepDataHuman = prepHumanModelForftINIT(ihuman, false, ...
    '/path/to/Human-GEM/data/metabolicTasks/metabolicTasks_Essential.txt', ...
    'path/to/Human-GEM/model/reactions.tsv');

Here I also get the same error:
>> prepDataHuman.minModel

ans =

  struct with fields:

                     id: 'Human-GEM'
            description: 'Generic genome-scale metabolic model of Homo sapiens'
                   rxns: {7079×1 cell}
                   mets: {2798×1 cell}
                      S: [2798×7079 double]
                     lb: [7079×1 double]
                     ub: [7079×1 double]
                    rev: [7079×1 double]
                      c: [7079×1 double]
                      b: [2798×1 double]
                  comps: {9×1 cell}
              compNames: {9×1 cell}
               rxnNames: {7079×1 cell}
                grRules: {7079×1 cell}
             rxnGeneMat: [7079×0 double]
             subSystems: {7079×1 cell}
                eccodes: {7079×1 cell}
               rxnNotes: {7079×1 cell}
                  genes: {}
         geneShortNames: {}

Here the prepDataHuman.minModel.grRules looks like its populated with values, but it's not:
>> prepDataHuman.minModel.grRules(1:5)

ans =

  5×1 cell array

    {0×0 char}
    {0×0 char}
    {0×0 char}
    {0×0 char}
    {0×0 char}

I then looked into how prepData is generated, and realized that at step 4 "Second simplification (~1 hour)" Parallel Computing Toolbox wasn't installed. I installed it, ran prepHumanModelForftINIT and got the same results, so it's not even that. It's weird, because I would be expecting the code to yell because of issues if nothing is detected in prepData.minModel.genes and prepData.minModel.grRules.
I hope I haven't made this super confusing!

Another update from my side:
I decided to manually impute missing grRules values inside prepData.minModel like so:
common_rxns = intersect(MouseGEM.rxns, prepData.minModel.rxns)
grRules = MouseGEM.grRules(ismember(MouseGEM.rxns, common_rxns))
prepData.minModel.grRules = grRules

and running:
model1 = ftINIT(prepData, data_struct.tissues{1}, [], [], data_struct, {}, getHumanGEMINITSteps('1+0'), false, true)

produces again the same error. maybe it also needs prepData.minModel.genes structure to be populated with values.

Hi, sorry, I misremembered, the GrRules should be empty in the minModel. Would it be possible for you to send me your code and data, and I can debug and see what is happening? You can send it to<EMAIL_ADDRESS>I can probably figure this out quickly, although there are no guarantees. One final thing to check: could you check if the b field in the mouse model has two columns? This is a bug that has been haunting mouse-GEM for a long time :).

See https://github.com/SysBioChalmers/Mouse-GEM/issues/22

In minModel, all linearly dependent reactions are merged, which makes the model smaller and thereby faster to solve. There is no good way to merge grRules, so the field is cleared - this is not used in ftINIT anyways.

Hi, sorry, I misremembered, the GrRules should be empty in the minModel. Would it be possible for you to send me your code and data, and I can debug and see what is happening? You can send it to<EMAIL_ADDRESS>I can probably figure this out quickly, although there are no guarantees. One final thing to check: could you check if the b field in the mouse model has two columns? This is a bug that has been haunting mouse-GEM for a long time :).

I have removed the second column in MouseGEM.b but to no avail. I am sending you my stuff to your email, thanks a lot for helping me out :)

Just noting here, however this gets solved, please follow up with a new issue to update either code (ftINIT or wherever else), or perhaps the guide. At least, I think the guide should be updated with some self-debugging code, so that people can self-diagnose some of the more common issues.

Thanks for the data, I will have a look. I don't see any obvious mistakes by just looking at your code, it looks good. Let me get back once I have debugged this.
@haowang-bioinfo Could you take a look at if the b field still has two columns in the latest Mouse-GEM? We have tried to fix this several times now, and it still comes back and bites us :)

@johan-gson yes I just checked the latest Mouse-GEM, which still has two columns in the b field - will fix it

@haowang-bioinfo The error must be in the script converting from human to mouse - no point fixing it in the model. But you probably realize that :). Would be good to add a test case that specifically tests this before releases, this problem seems to have a magic ability to come back.

Hi @rsango6, I tested it on my computer, and it worked, which is a bit confusing. I am using Gurobi 9.5, maybe I have not tested exactly the version you are running. One thing I see in the data is that the TPM doesn't sum to 10^6, so I would rescale it, but it is not what is causing the failure. What versions of Mouse-GEM and Human-GEM are you using?

I'm using the latest mouse gem

And, I didn't do any manipulations of the model except fixing the .b field, exactly as you did. I did not run the lines manipulating the grRules, that should not be done.

Hi @rsango6, I tested it on my computer, and it worked, which is a bit confusing. I am using Gurobi 9.5, maybe I have not tested exactly the version you are running. One thing I see in the data is that the TPM doesn't sum to 10^6, so I would rescale it, but it is not what is causing the failure. What versions of Mouse-GEM and Human-GEM are you using?

I am using:
Human-GEM v1.16.0
Mouse-GEM v1.6.0
RAVEN v2.8.4
MATLAB v2019b
and now have switched from Gurobi v10.02 to Gurobi v9.5.2. I thought that maybe the reason could be that TPM are loaded not as numeric values, rather as characters. Though still getting the same error. It does tell me at start of the analysis with ftINIT call there is a WARNING: Exchange metabolites are still present in the model. Use simplifyModel if this is not intended. Could that be the reason why? However, I had to use addBoundaryMets() on the mouseGEM otherwise I would get an error at a previous step with prepHumanModelForftINIT:
Step 3: Check tasks (~10 min)
Attempt to grow array along ambiguous dimension.

Error in closeModel (line 50)
        closedModel.b(numel(closedModel.b)+1)=0;

Error in prepINITModel (line 82)
    bModel = closeModel(cModel);

Error in prepHumanModelForftINIT (line 133)
prepDataHumanGEM = prepINITModel(m, taskStruct, spontRxnNames, convertGenes, customRxnsToIgnore, 'e');

It's rather strange that it is working for you and not in my case. By the way, TPM values are all obtained from Salmon as is, haven't manipulated with them in any way so that is also weird that there are also some discrepancies too. I need to think why that would be the case.

Ok, probably not a Gurobi thing then, which is good, could have been complicated to solve. This is strange. I also used RAVEN 2.8.4. I used the latest mouse-GEM, just downloaded it, I guess that is the same. And, the same MATLAB version. I do have an older Human-GEM though, something from devel in the middle. Does it work with the prepData I emailed you, could you test that?

Ok, probably not a Gurobi thing then, which is good, could have been complicated to solve. This is strange. I also used RAVEN 2.8.4. I used the latest mouse-GEM, just downloaded it, I guess that is the same. And, the same MATLAB version. I do have an older Human-GEM though, something from devel in the middle. Does it work with the prepData I emailed you, could you test that?

I can confirm I have now tried with your prepData object, and for both getHumanGEMINITSteps('1+0') and getHumanGEMINITSteps('1+1') and within seconds I have successful generation of the models.
Really interesting as to what has gone wrong in my case when generating prepData, seeing we are using dependencies with identical versions. Maybe something to do with the OS -> I am running the whole thing on macOS Ventura 13.4.1

Hmm, yes, weird. Can you see any difference between the prepData objects,
for example number of reactions etc. in the min model? It really should
work on Mac. Would be valuable if you could dig a bit more and debug what
is happening in prepInitData. But I would compare the prepDatas and see
what went wrong first, and then debug and see where that field(s) get their
bad values. Also strange with that warning you got, I didn't. Could be
something with the boundary metabolites. I don't have more advice,
debugging is the key, it should be doable.
On Thu, Aug 17, 2023 at 12:24 PM Roko Sango @.***>
wrote:

Ok, probably not a Gurobi thing then, which is good, could have been
complicated to solve. This is strange. I also used RAVEN 2.8.4. I used the
latest mouse-GEM, just downloaded it, I guess that is the same. And, the
same MATLAB version. I do have an older Human-GEM though, something from
devel in the middle. Does it work with the prepData I emailed you, could
you test that?
I can confirm I have now tried with your prepData object, and for both
getHumanGEMINITSteps('1+0') and getHumanGEMINITSteps('1+1') and within
seconds I have successful generation of the models.
Really interesting as to what has gone wrong in my case when generating
prepData, seeing we are using dependencies with identical versions. Maybe
something to do with the OS -> I am running the whole thing on macOS
Ventura 13.4.1
—
Reply to this email directly, view it on GitHub
https://github.com/SysBioChalmers/Human-GEM/issues/511#issuecomment-1682593823,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AHX2AKZBPFZM7JQAHLH7TWLXVZASDANCNFSM6AAAAAAWE6YHOM
.
You are receiving this because you were mentioned.Message ID:
@.***>

@haowang-bioinfo The error must be in the script converting from human to mouse - no point fixing it in the model. But you probably realize that :). Would be good to add a test case that specifically tests this before releases, this problem seems to have a magic ability to come back.

the fix is now implemented by bb4cc9fb81d3bbae249468b94734c2122030b4fa in #692

Hi,
Here are the screenshots from your prepData and mine:
[A screenshot of a computer  Description automatically generated]               [A screenshot of a computer  Description automatically generated]
minModel in each case look identical (also refModel fields look identical):
@.***                      @.***
In the first pair of screenshots, you can see differences in numbers in essentialRxns, taskStruct, essentialMetsForTasks.
Another difference is in prepData.taskReport.ok field: all of values in your cases are 0’s, in mine all of them are 1’s. Perhaps there is something wrong with the task structures that are part of my prepData? I would need to look closely into this.
Best,
Roko
From: johan-gson @.>
Date: Thursday, 17. August 2023 at 19:56
To: SysBioChalmers/Human-GEM @.>
Cc: Roko Sango @.>, Mention @.>
Subject: Re: [SysBioChalmers/Human-GEM] ERROR using ftINIT. MipGap too high, trying with a different run. (Issue #511)
Hmm, yes, weird. Can you see any difference between the prepData objects,
for example number of reactions etc. in the min model? It really should
work on Mac. Would be valuable if you could dig a bit more and debug what
is happening in prepInitData. But I would compare the prepDatas and see
what went wrong first, and then debug and see where that field(s) get their
bad values. Also strange with that warning you got, I didn't. Could be
something with the boundary metabolites. I don't have more advice,
debugging is the key, it should be doable.
On Thu, Aug 17, 2023 at 12:24 PM Roko Sango @.***>
wrote:

Ok, probably not a Gurobi thing then, which is good, could have been
complicated to solve. This is strange. I also used RAVEN 2.8.4. I used the
latest mouse-GEM, just downloaded it, I guess that is the same. And, the
same MATLAB version. I do have an older Human-GEM though, something from
devel in the middle. Does it work with the prepData I emailed you, could
you test that?
I can confirm I have now tried with your prepData object, and for both
getHumanGEMINITSteps('1+0') and getHumanGEMINITSteps('1+1') and within
seconds I have successful generation of the models.
Really interesting as to what has gone wrong in my case when generating
prepData, seeing we are using dependencies with identical versions. Maybe
something to do with the OS -> I am running the whole thing on macOS
Ventura 13.4.1
—
Reply to this email directly, view it on GitHub
https://github.com/SysBioChalmers/Human-GEM/issues/511#issuecomment-1682593823,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AHX2AKZBPFZM7JQAHLH7TWLXVZASDANCNFSM6AAAAAAWE6YHOM
.
You are receiving this because you were mentioned.Message ID:
@.***>

—
Reply to this email directly, view it on GitHubhttps://github.com/SysBioChalmers/Human-GEM/issues/511#issuecomment-1682724575, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AWOCBQKEUNPEWNJDDV7JQVTXVZLNFANCNFSM6AAAAAAWE6YHOM.
You are receiving this because you were mentioned.Message ID: @.***>

Ah, sorry, I didn't pay enough attention, the tasks did not pass in my run.
Let me look at this again, your prepdata is probably better than mine. I
will get back to you once I have checked this.
On Sun, Aug 20, 2023 at 11:12 AM Roko Sango @.***>
wrote:

Hi,
Here are the screenshots from your prepData and mine:
[A screenshot of a computer Description automatically generated] [A
screenshot of a computer Description automatically generated]
minModel in each case look identical (also refModel fields look
identical):
@.*** @.***
In the first pair of screenshots, you can see differences in numbers in
essentialRxns, taskStruct, essentialMetsForTasks.
Another difference is in prepData.taskReport.ok field: all of values in
your cases are 0’s, in mine all of them are 1’s. Perhaps there is something
wrong with the task structures that are part of my prepData? I would need
to look closely into this.
Best,
Roko
From: johan-gson @.>
Date: Thursday, 17. August 2023 at 19:56
To: SysBioChalmers/Human-GEM @.>
Cc: Roko Sango @.>, Mention @.>
Subject: Re: [SysBioChalmers/Human-GEM] ERROR using ftINIT. MipGap too
high, trying with a different run. (Issue #511)
Hmm, yes, weird. Can you see any difference between the prepData objects,
for example number of reactions etc. in the min model? It really should
work on Mac. Would be valuable if you could dig a bit more and debug what
is happening in prepInitData. But I would compare the prepDatas and see
what went wrong first, and then debug and see where that field(s) get
their
bad values. Also strange with that warning you got, I didn't. Could be
something with the boundary metabolites. I don't have more advice,
debugging is the key, it should be doable.
On Thu, Aug 17, 2023 at 12:24 PM Roko Sango @.***>
wrote:

Ok, probably not a Gurobi thing then, which is good, could have been
complicated to solve. This is strange. I also used RAVEN 2.8.4. I used
the
latest mouse-GEM, just downloaded it, I guess that is the same. And, the
same MATLAB version. I do have an older Human-GEM though, something from
devel in the middle. Does it work with the prepData I emailed you, could
you test that?
I can confirm I have now tried with your prepData object, and for both
getHumanGEMINITSteps('1+0') and getHumanGEMINITSteps('1+1') and within
seconds I have successful generation of the models.
Really interesting as to what has gone wrong in my case when generating
prepData, seeing we are using dependencies with identical versions.
Maybe
something to do with the OS -> I am running the whole thing on macOS
Ventura 13.4.1
—
Reply to this email directly, view it on GitHub
<
https://github.com/SysBioChalmers/Human-GEM/issues/511#issuecomment-1682593823>,

or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/AHX2AKZBPFZM7JQAHLH7TWLXVZASDANCNFSM6AAAAAAWE6YHOM>

.
You are receiving this because you were mentioned.Message ID:
@.***>

—
Reply to this email directly, view it on GitHub<
https://github.com/SysBioChalmers/Human-GEM/issues/511#issuecomment-1682724575>,
or unsubscribe<
https://github.com/notifications/unsubscribe-auth/AWOCBQKEUNPEWNJDDV7JQVTXVZLNFANCNFSM6AAAAAAWE6YHOM>.
You are receiving this because you were mentioned.Message ID: @.***>
—
Reply to this email directly, view it on GitHub
https://github.com/SysBioChalmers/Human-GEM/issues/511#issuecomment-1685308815,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AHX2AK2ZOB3FAHMQPV3YOATXWISO5ANCNFSM6AAAAAAWE6YHOM
.
You are receiving this because you were mentioned.Message ID:
@.***>

I have rerun it and now get the same problem as you, so at least it is
reproducible. All the tasks pass now. This will take some time for me to
solve, it is not trivial, sorry about that. I will get back to you when I
know more,
On Sun, Aug 20, 2023 at 11:38 AM Johan Gustafsson <
@.***> wrote:

Ah, sorry, I didn't pay enough attention, the tasks did not pass in my
run. Let me look at this again, your prepdata is probably better than mine.
I will get back to you once I have checked this.
On Sun, Aug 20, 2023 at 11:12 AM Roko Sango @.***>
wrote:

Hi,
Here are the screenshots from your prepData and mine:
[A screenshot of a computer Description automatically generated] [A
screenshot of a computer Description automatically generated]
minModel in each case look identical (also refModel fields look
identical):
@.*** @.***
In the first pair of screenshots, you can see differences in numbers in
essentialRxns, taskStruct, essentialMetsForTasks.
Another difference is in prepData.taskReport.ok field: all of values in
your cases are 0’s, in mine all of them are 1’s. Perhaps there is something
wrong with the task structures that are part of my prepData? I would need
to look closely into this.
Best,
Roko
From: johan-gson @.>
Date: Thursday, 17. August 2023 at 19:56
To: SysBioChalmers/Human-GEM @.>
Cc: Roko Sango @.>, Mention @.>
Subject: Re: [SysBioChalmers/Human-GEM] ERROR using ftINIT. MipGap too
high, trying with a different run. (Issue #511)
Hmm, yes, weird. Can you see any difference between the prepData objects,
for example number of reactions etc. in the min model? It really should
work on Mac. Would be valuable if you could dig a bit more and debug what
is happening in prepInitData. But I would compare the prepDatas and see
what went wrong first, and then debug and see where that field(s) get
their
bad values. Also strange with that warning you got, I didn't. Could be
something with the boundary metabolites. I don't have more advice,
debugging is the key, it should be doable.
On Thu, Aug 17, 2023 at 12:24 PM Roko Sango @.***>
wrote:

Ok, probably not a Gurobi thing then, which is good, could have been
complicated to solve. This is strange. I also used RAVEN 2.8.4. I used
the
latest mouse-GEM, just downloaded it, I guess that is the same. And,
the
same MATLAB version. I do have an older Human-GEM though, something
from
devel in the middle. Does it work with the prepData I emailed you,
could
you test that?
I can confirm I have now tried with your prepData object, and for both
getHumanGEMINITSteps('1+0') and getHumanGEMINITSteps('1+1') and within
seconds I have successful generation of the models.
Really interesting as to what has gone wrong in my case when generating
prepData, seeing we are using dependencies with identical versions.
Maybe
something to do with the OS -> I am running the whole thing on macOS
Ventura 13.4.1
—
Reply to this email directly, view it on GitHub
<
https://github.com/SysBioChalmers/Human-GEM/issues/511#issuecomment-1682593823>,

or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/AHX2AKZBPFZM7JQAHLH7TWLXVZASDANCNFSM6AAAAAAWE6YHOM>

.
You are receiving this because you were mentioned.Message ID:
@.***>

—
Reply to this email directly, view it on GitHub<
https://github.com/SysBioChalmers/Human-GEM/issues/511#issuecomment-1682724575>,
or unsubscribe<
https://github.com/notifications/unsubscribe-auth/AWOCBQKEUNPEWNJDDV7JQVTXVZLNFANCNFSM6AAAAAAWE6YHOM>.
You are receiving this because you were mentioned.Message ID:
@.***>
—
Reply to this email directly, view it on GitHub
https://github.com/SysBioChalmers/Human-GEM/issues/511#issuecomment-1685308815,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AHX2AK2ZOB3FAHMQPV3YOATXWISO5ANCNFSM6AAAAAAWE6YHOM
.
You are receiving this because you were mentioned.Message ID:
@.***>

Hi @johan-gson, did you have any luck finding a solution to this issue? Thanks!

Sorry I have not. I will try to take another look at this in the coming weeks. Sorry for the delay, I have a pretty heavy workload right now.


Add quality of service logic to admin

TL;DR
This PR implements translation from a user-specified quality of service to a queueing budget used for cost optimization.
Users can specify the quality of service for an execution (in order of decreasing specificity)

At CreateExecution request time
In the LaunchPlan spec
In the Workflow spec
As an overridable MatchableResource

System administrators can specify default quality of service specs for different tiers and default tiers for different domains.
Type

[ ] Bug Fix
[x] Feature
[ ] Plugin

Are all requirements met?

[x] Code completed
[x] Smoke tested
[x] Unit tests added
[x] Code documentation added
[ ] Any pending items have an associated Issue

Complete description
https://docs.google.com/document/d/1rYqr01x0XL6n1AdmEOUbTIGeS9Z2hJCcnG5NMy8aQ68/edit#
Tracking Issue
https://github.com/lyft/flyte/issues/343
Follow-up issue
NA

Codecov Report

Merging #107 into master will decrease coverage by 0.14%.
The diff coverage is 61.87%.

@@            Coverage Diff             @@
##           master     #107      +/-   ##
==========================================
- Coverage   62.60%   62.45%   -0.15%
==========================================
  Files         102      104       +2
  Lines        7418     7549     +131
==========================================
+ Hits         4644     4715      +71
- Misses       2228     2278      +50
- Partials      546      556      +10

Flag
Coverage Δ

#unittests
62.45% <61.87%> (-0.15%)
:arrow_down:

Impacted Files
Coverage Δ

pkg/workflowengine/impl/propeller_executor.go
52.20% <ø> (ø)

pkg/runtime/quality_of_service_provider.go
23.07% <23.07%> (ø)

pkg/runtime/configuration_provider.go
51.72% <33.33%> (-2.13%)
:arrow_down:

pkg/manager/impl/executions/quality_of_service.go
60.97% <60.97%> (ø)

pkg/manager/impl/execution_manager.go
68.67% <85.71%> (+0.03%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6d15a2b...800486e. Read the comment docs.

friendly ping @EngHabu @migueltol22

@migueltol22 @EngHabu


I'm looking for version 0.41.0

It seems the publish wasn't successful. The github action shows that step taking 0 seconds for whatever reason. The latest tags are :latest and :1.39.0 from 4 months ago.
@softprops could you manually publish a new version or try the actions run again? I can't update my deps because they depend on recently stabilized features.

sry about that. I set up the project to push on tag v0.2.6-rust-1.41.0 which should trigger a docker push to docker hub. Keep an eye out for this workflow run to complete

The new tag is working as expected. Thanks softprops.
For anyone that comes across this, either of the following in serverless.yml will update to the newer tag.
custom:
  rust:
    dockerTag: '0.2.6-rust-1.41.0'

custom:
  rust:
    dockerTag: latest


Create New Topic Content

docs/guides/evaluating-and-prototyping-with-livepeer/media-analytics-what-gets-measured-and-aggregated.mdx
Is a brand new topic.


Add Value 4 Value tipping support for substack

Is your feature request related to a problem? Please describe
As a reader I want to be able to tip a substack publisher the same way as we do it with the other "batteries"
Describe the solution you'd like
Alby could extract the lightning details from each article or the about page.

I'll pick this one up as an intro to the project.

Is there anything left to do about this issue?

No. The feature is complete.

Thanks @lukeojones ! I will close this issue.


fix(vite-ssr-plugin): remove spread object on httpResponse

:tada: This PR is included in version 7.16.1 :tada:
The release is available on:

v7.16.1
GitHub release

Your semantic-release bot :package::rocket:


Create files relative from the actual position

It would be nice if I could create files relative to the actual position in the explorer

hi, Can you explain further with an example for better understanding?

At the moment the file is created in the root of the project. But it would be cool if I can created on the position where the folder that is selected in the explorer.

I got your point! Thanks for the suggestion!

A Tips: Just right click on a folder from explorer window and choose create new file - the relative path from workspace will be auto filled.

Watch the gif (2nd part)

Thanks for this great plugin. Really like it.
Regarding this issue, it would be great if the shortcut would do that as well.

Yes, it would be great if the shortcut would do that as well. This is especially good if you, like me, want to avoid touching the mouse/trackpad when coding. 😎
@ritwickdey Any updates on this? How could we help you with this?

Also, one more thing would be useful, is to be able to create a file in the same directory of the currently open file.

I would love to also see this! As a VIM user I don't use the mouse/trackpad, so a keyboard shortcut to create files relative to current open file would be amazingly useful. =)


docs: add license and readme

This pull request includes updates to the LICENSE.md and README.md files. The changes include adding a new license and significantly revising the README to reflect the project's new direction and features.

README Overhaul:

Removed old NestJS branding, badges, and setup instructions, and replaced them with a new project description and features.
Introduced a new project introduction and features section, highlighting multi-protocol support, event-driven architecture, fault tolerance, and authentication.
Added new sections for architecture, installation, and contributing guidelines, emphasizing the use of MongoDB, RabbitMQ, Docker, and MailHog.
Updated the license section to reflect the new WTFPL license.

yolo


Change 932237 to phase 1

Motivation
932237 is currently set to phase 2, but the rule only inspects the user agent and referrer header and not anything within the request body. Changing this to phase 1 will slightly improve performance,
Proposed solution
Change the rule to phase 1 for improved performance, just like how 932239 is set to phase 1.
Alternatives
N/A
Additional context
I discoverd this issue while working on #3394

Thanks for creating this issue.

We didn't address it yet, right? @EsadCetiner do you want to push a PR for this move?

@fzipi if there's nobody that disagrees, I'm happy to open a PR.

I've open a PR here: https://github.com/coreruleset/coreruleset/pull/3570
I noticed a few more rules that inspect only the headers at phase 2 so I've changed those as well

Awesome! 🚀


GCC 7 warnings

When building with gcc 7.1.1 on archlinux  openrc 0.27.2 triggers two new warnings,
-Wformat-truncation= & -Wimplicit-fallthrough=

The -Wformat-truncation=level option detects certain and likely output truncation in calls to the snprintf  family of formatted output functions. -Wformat-truncation=1 is included in -Wall and enabled without  optimization but works best with -O2 and higher.
For example, the following function attempts to format an integer between 0 and 255 in hexadecimal,  including the 0x prefix, into a buffer of four characters. But since the function must always terminate output by the null character ('\0') such a buffer is only big enough to fit just one digit plus the prefix.  Therefore the snprintf call is diagnosed. To avoid the warning either use a bigger buffer or handle the function's return value which indicates whether or not its output has been truncated.
void f (unsigned x)
{
char d[4];
snprintf (d, sizeof d, "%#02x", x & 0xff);
…
}
warning: 'snprintf' output may be truncated before the last format character [-Wformat-truncation=]
note: 'snprintf' output between 3 and 5 bytes into a destination of size 4

-Wimplicit-fallthrough warns when a switch case falls through. This warning has five different levels. The compiler is able to parse a wide range of fallthrough comments, depending on the level. It also handles control-flow statements, such as ifs. It's possible to suppress the warning by either adding a fallthrough comment, or by using a null statement: attribute ((fallthrough)); (C, C++), or [[fallthrough]]; (C++17), or [[gnu::fallthrough]]; (C++11/C++14). This warning is enabled by -Wextra.

Output of  make "${_makeargs[@]}" is attached, make install didn't output any warnings.
Used settings :
_makeargs=(BRANDING='Arch Linux')
_makeargs+=(MKPAM=pam)
_makeargs+=(MKSELINUX=no)
_makeargs+=(MKTERMCAP=ncurses)
_makeargs+=(PKG_PREFIX="")
_makeargs+=(LIBDIR=/usr/lib)
_makeargs+=(LIBMODE=0644) # enable binary stripping by makepkg
_makeargs+=(SHLIBDIR=/usr/lib)
_makeargs+=(LIBEXECDIR=/usr/lib/openrc)
_makeargs+=(BINDIR=/usr/bin)
_makeargs+=(SBINDIR=/usr/bin)
_makeargs+=(SYSCONFDIR=/etc/openrc) # avoid conflict with initscriptsBuild options used : for

see https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=openrc for full build script .
https://gcc.gnu.org/gcc-7/changes.html

Could you provide the actual warnings you got, for those of us that don't have GCC 7 available?

It seems the attachment got lost somehow, so i've uploaded the log here :
https://app.box.com/s/429ihvq7ba2ntuvu1boz4dypvgikbp5h

Github issues don't have attachments.
libeinfo.c: In function ‘eindent’:
libeinfo.c:897:21: warning: ‘%08d’ directive output may be truncated writing between 8 and 11 bytes into a region of size 10 [-Wformat-truncation=]
  snprintf(num, 10, "%08d", amount);
                     ^~~~
libeinfo.c:897:20: note: directive argument in the range [-2147483648, 40]
  snprintf(num, 10, "%08d", amount);
                    ^~~~~~

rc-status.c: In function ‘print_service’:
rc-status.c:115:25: warning: ‘%02ld’ directive output may be truncated writing between 2 and 18 bytes into a region of size between 0 and 28 [-Wformat-truncation=]
       "%ld day(s) %02ld:%02ld:%02ld (%s)",
                         ^~~~~
rc-status.c:115:7: note: directive argument in the range [0,<PHONE_NUMBER>80911939]
       "%ld day(s) %02ld:%02ld:%02ld (%s)",
       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
rc-status.c:115:7: note: directive argument in the range [-9223372036854775808,<PHONE_NUMBER>854716339]
rc-status.c:120:20: warning: ‘%02ld’ directive output may be truncated writing between 2 and 20 bytes into a region of size between 4 and 34 [-Wformat-truncation=]
       "%02ld:%02ld:%02ld (%s)",
                    ^~~~~
rc-status.c:120:7: note: directive argument in the range [-9223372036854775808,<PHONE_NUMBER>854716339]
       "%02ld:%02ld:%02ld (%s)",
       ^~~~~~~~~~~~~~~~~~~~~~~~

These are the important messages for this issue.  There are some notes about implicit fallthrough that should probably be resolved, but that's for a different issue.

gcc 7 is now in the stable tree on my distro, so this is fixed in the following commits:
a6165168
04721ece
c7e8f113
47e4bfae
8a945194
These will be included in 0.38.


Improve seeds

Overview

Configuration

Details

Notes

Screenshots

@pwestman82 please update it with all the seeds.
Also, please write some description in the message ☝, especially when there is something we have to do after closing the pr. In this case, for example, we know that we'll have to run heroku run rails db:seed to have the seeds in production, so write it here 🙂

 Improve seeds


Leon test ticket

Leon is the best because he says so!

Thanks Leon, my issue has been resolved. Case closed!


explain public attributes in /typesystem

There seems to be confusion https://irclog.perlgeek.de/perl6/2017-09-22#i_15201661

I'm trying to wrap my head around this issue, and I can't say I have managed to do so.
First, from the conversation it's not clear whether this applies to attributes or to variables in general, since "everything is a container", and @moritz says "for all variables"
Second, not clear also what needs to be explained. If we stick to the question:

is this an lvalue method call or a variable access
I guess that would be easy by using @moritz answer. But in that case I'm not sure this would apply to typesystem, since we are not really talking about how types work, rignt?

Unless I'm missing something, this does not work:
class Foo { has $.bar };
(Foo.new.bar = "Hi").say  # Cannot modify an immutable 'Any' type object␤

So I probably need a bit of more context. @moritz ? @fluca1978 ?


Bad error message when using the pattern option without specifying a module via the --module option

The current error message reads
The main module, '', was not found. Check the --module flag.

What it should read is
You must use --module with --pattern.

The problem is, --module is optional in the sense that it is required only if --pattern is given. We should change the parser so that --module and --pattern must appear together if they appear at all.

I started to write how to do this, but it would be much nicer if we just used optparse-applicative the way it was intended.
Let's add a new executable called kore-parse with two sub-commands, kore-parse definition and kore-parse pattern. The second command will have the non-optional arguments. There is a command parser for just this case. There is a more detailed example in the tutorial.


Improve robustness of Jupyter server error handling

Right now, the code that handles errors from the spawned Jupyter server is not particularly robust. It should be improved to a) cleanly handle Jupyter server anomalies and b) properly inform the user when something goes wrong with the Jupyter server.
I began implementing this with #96, but we can make a lot more improvements.

This was tackled in #339. Ideally we would also display logs in addition to the error message.

closing since there were several enhancements already implemented