proposal: path/filepath: add a function to canonicalize paths

What version of Go are you using (go version)?

$ go version
go version go1.12.7 windows/amd64

Does this issue reproduce with the latest release?
Yes.
What operating system and processor architecture are you using (go env)?
Applies to all OSes
What did you do?

Mounted a UNC path as a drive letter.
In CMD, switched the current working directory to that drive.
Called filepath.Abs on a relative path.
Called filepath.EvalSymlinks on the result of that function.

What did you expect to see?
The same results as calling GetFinalPathNameByHandle: a UNC path.
What did you see instead?
A path using the drive letter instead of the UNC path.
Notes
This affects any attempt to canonicalize paths using the output of Git in such a situation.  Git produces some paths as absolute and some paths as relative, and uses GetFinalPathNameByHandle for canonicalizing absolute paths.  However, Go lacks a function to canonicalize paths in a standard way, so it isn't possible to produce results equivalent to a C program and still write code that works portably across systems.
Go should add a function that is explicitly defined to canonicalize paths in a way equivalent to the underlying operating system, since using filepath.Abs and filepath.EvalSymlinks doesn't work correctly on Windows.  It does work fine on Unix, but Unix paths are much simpler and easier to reason about.
It was determined in #17084 that filepath.Abs and filepath.EvalSymlinks were sufficient in this case, but that doesn't appear to be true.  I expect there are other cases in which those don't work on Windows, but I am insufficiently versed in Windows paths to know what those are.
This was originally reported to the Git LFS project in git-lfs/git-lfs#4012.

/cc @robpike @rsc per owners.

What did you expect to see?
The same results as calling GetFinalPathNameByHandle: a UNC path.

I did not try it, but, I suspect, UNC paths wouldn't work in some situations. For example, can you pass UNC path to os.Chdir?
Alex

I don't know for certain, but judging by a quick Google search, it appears to be possible in Ruby, so I assume one can do that in C-based languages.
I'm not a Windows developer, so I'm not a good person to ask about the capabilities of Windows.  I'm just a Unix developer trying to make general-purpose software not be terrible on Windows.

I don't know for certain, but judging by a quick Google search, it appears to be possible in Ruby, so I assume one can do that in C-based languages.

You are correct. I was wrong. os.Chdir does work with UNC paths.
Alex

It is also the case that filepath.EvalSymlinks fails to work when canonicalizing paths where there's a junction to a volume that lacks a drive letter (a OneDrive mount is a good example of this).  For example, if C:\Users\User\OneDrive\Vault is a junction pointing to a OneDrive mount and we try to call filepath.EvalSymlinks("C:/Users/Users/OneDrive/Vault/home.git"), that will fail with readlink C:\Users\User\OneDrive\Vault: The system cannot find the path specified.
This also works with C-based programs.

Hey,
Is there any interest in fixing this?  Right now, there is no cross-platform way to canonicalize a path in Go.  We keep running up against additional cases where the existing behavior doesn't canonicalize paths properly, leading to incompatibility with other programs on the system (notably Git).  This necessarily limits the portability of using Go as a cross-platform language.

By "function to canonicalize paths" do you mean a variation of EvalSymlinks that works on Windows? If so, note that EvalSymlinks is not recommended: #40180 (and probably can't be fixed).
Go on Windows has a variety of long-standing filesystem bugs. I suggest using x/sys/windows to call the WinAPI if that solves your problem.

I mean a function, when given a path, that returns a canonicalized version of that path.  In other words, the equivalent to realpath(3) on Unix or GetFinalPathNameByHandle on Windows, and the equivalent to Rust's std::path::canonicalize.
It isn't helpful to me to call the Windows API because (a) I'm not a Windows programmer and have no clue how to use it, (b) it isn't cross-platform, and (c) this is a function that is generally provided by the standard library.

Go has gaps on Windows; I plug them in my code. You've seen the interest this issue evoked :-p
What you need isn't hard. Create a file named yourpkg_windows.go, import "golang.org/x/sys/windows", define GetCanonicalPath() to call CreateFile("yourfile") (to get a handle) then GetFinalPathNameByHandle.
Create a file yourpkg_unix.go with a // +build directive for your unix platforms. Define GetCanonicalPath() with the solution for unix you already know.

First of all, I appreciate that you're trying to help.  However, I do feel firmly that this functionality should be in the standard library, since it is in almost every other language, and it is in POSIX.  I don't want to carry a lot of platform-specific code in a program because it's difficult to maintain and test, especially when I don't typically develop on Windows.
If Go is known to have known defects on Windows, those should be promptly fixed or clearly documented.  For many purposes, it's fine if code doesn't run or run well on Windows, but there are some cases where it does.  The documentation should clearly and prominently list any limitations with using Go on Windows so that folks can make an informed decision.  Last I checked, the filepath documentation didn't indicate such limitations, and hasn't for some time.
Normally, when I find a bug or missing feature, I would send a patch to implement that functionality.  However, Go has a CLA, and I don't sign CLAs, so any patch I might submit wouldn't be accepted.  If that changes, I'm happy to send a patch to implement this properly if nobody gets to it before me.

On Unix systems I think the proposed function is the same as filepath.EvalSymlinks.

Yes, I believe that they are identical.  filepath.EvalSymlinks is, as far as I'm aware, equivalent to realpath(3) on Unix and has the semantics I'm looking for.

This proposal should probably also deprecate EvalSymlinks, which is seriously broken on Windows.

What does "canonical" mean, precisely?

If there are multiple ways to refer to a filename, the canonical path is the absolute filename which uses no indirections and uses the canonical case (that is, the path component as written to the file system) if the system permits case folding.  On Unix, that's the one that contains no symlinks (and, on macOS, uses canonical case and composition).  On Windows, there are many ways to have indirection in a path: symlinks, junctions, SUBST, etc.  (I don't actually know all of the possible ways, since I almost never use Windows).  The canonical form uses none of those indirections and uses the canonical case.
Another way to say this is that assuming no hardlinks exist, a file on Unix should have exactly one canonical name whose components are either directories or non-symlink, non-directory (but possibly special) files.

@networkimprov, you make assertions without being specific about them. I am confused about three of the things you've said related to this issue.

You linked to #40180 which is making a general software engineering argument along the lines of "you should never actually replace all the symlinks, that's violating the abstractions that have been set up". I have some sympathy for that, but if it were true, it would apply not just to EvalSymlinks but also this issue as well. If so, then we should just close this very issue (#37113) as a terrible idea.

I see that you mentioned this issue in #40966, which is about some problems with path lengths in EvalSymlinks on Windows. We've had path length problems elsewhere on Windows. Path length issues are usually pretty straightforward to fix. Why would we want to gate a fix to #40966 on a larger design discussion on this issue?

Finally, you said, with no links at all, "This proposal should probably also deprecate EvalSymlinks, which is seriously broken on Windows." How is it broken? That comment would be a good place for an issue link.

Thanks.

I mentioned #40180 in https://github.com/golang/go/issues/37113#issuecomment-689088220 to suggest that the issue author reconsider canonicalization of paths. I didn't link it again later, but it documents a long list of problems with EvalSymlinks on Windows.
Re path length bugs, other instances of those have been left alone, see #21782 & #36375. And here's a list of Windows bugs that mention "filepath" https://github.com/golang/go/issues?q=is%3Aopen+is%3Aissue+label%3AOS-Windows+filepath

Canonicalization of paths is required to properly implement any sort of Git support in a project.  More generally, it's required to determine definitively if a path is under a directory, which has a wide variety of general-purpose applications outside of Git.  Whether other people think it is useful in their projects, path canonicalization is commonly used and is almost always provided by the standard library.  Canonicalizing paths is also recommended by CMU's secure coding guidelines; while those are for C, there's no reason to think Go is any different.
I agree that users typically don't want to see canonicalized paths and that path canonicalization cannot be used where there's a security-sensitive race condition, but that doesn't mean it lacks applications elsewhere, just that it's unsuitable for some use cases.

@networkimprov, the path length bugs are left alone only for lack of time. I don't think there's any objection to fixing them as long as it is done correctly and well.
@bk2204, I'm certainly not arguing against this functionality. I'm trying to understand why EvalSymlinks shouldn't be what provides this functionality on Windows.
(I do somewhat object to the name "canonical": if "/home/rsc" symlinks to some device path like "/u123/g3tah0uojq1/rsc", I have a hard time calling the latter the "canonical" one. And for what it's worth there are plenty of people who disagree with you about what "determine definitively if a path is under a directory" should mean. I have the bug reports to prove it. :-) But again, I'm not saying we shouldn't do this. I just think EvalSymlinks is probably the answer.)

If you change EvalSymlinks to call GetFinalPathNameByHandle on Windows, it may break some apps, so I'm pretty sure Alex wouldn't agree.

If you change EvalSymlinks to call GetFinalPathNameByHandle on Windows, it may break some apps, so I'm pretty sure Alex wouldn't agree.

It would need to keep doing what it's documented to do, namely preserve relative-ness to current directory when possible. That means calling GetFinalPathNameByHandle and fixing up the result a little. But we could still build a function around GetFinalPathNameByHandle that should handle everything Windows can throw at it.
Did you have a specific breakage in mind?

Are you proposing that the behavior differ from GetFinalPathNameByHandle (on Windows) only when the path name is relative, or when the path name is absolute as well?  The former is fine, I think, and I have no position on it; the latter would be a problem for interoperability with tools written in other languages.

@alexbrainman what do you think of changing the implementation of filepath.EvalSymlinks to just call GetFinalPathNameByHandle?

Relative paths are not thread-safe. Full stop.
EvalSymlinks should not be fixed but replaced. GetFinalPathNameByHandle always returns an absolute path. Paths with \\?\ syntax are always absolute.
GetFinalPathNameByHandleA function (fileapi.h) - Win32 apps | Microsoft Docs

The string that is returned by this function uses the "\?" syntax. For more information, see CreateFile.

The CMU guidance obviously is written without considering the arguments against EvalSymlinks that I wrote up - which is based on guidance from Microsoft, so I don't quite agree with your conclusion that resolving file system driver paths is a proper thing to do while canonicalizing. I have just hastily scanned that document from CMU and I don't immediately see they say you should resolve links. They just mention links could be present for awareness.
I have written quite a bit of software and I have never considered writing checks like that. Usually checks perhaps involve checking whether they are in some base path - for which Rel is better than EvalSymlinks. Sure you can ask for a properly cased path and/or fix slashes, but otherwise, why write string handling or comparisons at all? Paths usually come from somewhere. I mean from the working directory, from configuration or perhaps from user input in most cases. Usually those are already trusted without doing any checks. Or they are built from system settings, which are also trusted. That might be circumstantial evidence against that article but still.
Also about the Git example still I think the application should be in control of which links get resolved. Some of the links that make up a path used by Git could still be system administrator controlled which makes it wrong to resolve them and the result dependant on system configuration. Like for my system - all Git paths will be volume GUID paths on my system (and hence again all always absolute). But Git should never care about that or go that deep. It's wrong.

As it's not possible to relativize some results of GetFinalPathNameByHandle, that could be a cause of breakage if EvalSymlinks is reimplemented with it.

Relative paths are not thread-safe. Full stop.

That's not true on modern POSIX systems.  You can open a file descriptor to a directory and operate on a path relative to that file descriptor with the *at series of functions.  It may be true on Windows.

The CMU guidance obviously is written without considering the arguments against EvalSymlinks that I wrote up - which is based on guidance from Microsoft, so I don't quite agree with your conclusion that resolving file system driver paths is a proper thing to do while canonicalizing. I have just hastily scanned that document from CMU and I don't immediately see they say you should resolve links. They just mention links could be present for awareness.

I agree with some of your points and I've stated so above.  Users usually are not interested in canonical paths.  It's also wrong to use path canonicalization in a case where the possibility of changing path resolution leads to a security problem or buggy behavior.  That's a well known problem on Unix systems and Windows is no different.
It is irrelevant to me whether the path is created with a drive letter or not, so I have no position on that argument.
I agree that EvalSymlinks, as it exists today, is broken on Windows and does not do anything interesting or useful in most cases.  That's why I opened this issue.

I have written quite a bit of software and I have never considered writing checks like that. Usually path checks involve checking whether they are in some base path - for which Rel is better than EvalSymlinks. Sure you can ask for a properly cased path and/or fix slashes, but otherwise, why write string handling or comparisons at all? Paths usually come from somewhere. I mean from the working directory, from configuration or perhaps from user input in most cases. Usually those are already trusted without doing any checks. Or they are built from system settings, which are also trusted. That might be circumstantial evidence against your conclusion based on that article but still.

The question is not whether you think this feature is valuable.  You need not use it.  The question is whether Go ought to provide access in a portable way to cross-platform functionality that every operating system and every other major language provides and which is used in many projects for good and valuable reasons, and which, even if used imprudently, is required for compatibility with other software already existing for longer than Go has.  The fact that major organizations like CMU recommend this practice is evidence that this feature is important and valuable, even if you disagree.
Path canonicalization is even more important on Windows than Unix because Windows has case-folding behavior in its file system that depends on attributes related to when the file system was created.  It is therefore impossible without the kernel's help to know the proper name of a file and whether two file names actually refer to the same item on disk, and Windows otherwise lacks the concept of device and inode numbers which are normally used to perform this check on Unix.

Also about the Git example still I think the application should be in control of which links get resolved. Some of the links that make up a path used by Git could still be system administrator controlled which makes it wrong to resolve them and the result dependant on system configuration. Like for my system - most Git paths ran through EvalSymlinks on my system will (have to) be volume GUID paths (and hence again all always absolute). But Git should never care about that or go that deep. It's wrong.

I have seen first hand how Git broke when it did not canonicalize paths consistently, and I still have a broken repository on my system from that point.  I am a core contributor to Git and the primary driver of the SHA-256 transition.  I've developed in situations where symlinks and path resolution have important and subtle security implications.  I understand the problem space intimately and why Git has the behavior it does.  Please don't try to to tell me that Git's behavior is wrong here, because it is not.  Whether a path is created by the user, the system administrator, or any other actor does not change whether canonicalization is necessary.

That's not true on modern POSIX systems. You can open a file descriptor to a directory and operate on a path relative to that file descriptor with the *at series of functions. It may be true on Windows.

Obviously that is the same on any operating system if you just use Rel and Join. But EvalSymlinks makes the path relative to the current working directory and it is that fact that makes it not thread-safe. You can use Join but again you need the current working directory as argument, which might have changed in between these two calls.

The question is not whether you think this feature is valuable. You need not use it.

My argument is not against canonicalization, but against using or fixing EvalSymlinks to do it. This whole comment was about using GetFinalPathNameByHandle to implement EvalSymlinks. It can't be done properly.

Windows otherwise lacks the concept of device and inode numbers

There is the concept of object identifiers, but these are an NTFS concept. So yeah sure absolutely, the OS needs to be involved and one of the things that is wrong with path\filepath is that it doesn't involve the OS enough.

Please don't try to to tell me that Git's behavior is wrong here, because it is not.

I don't know what Git does, but I really think it is wrong to resolve all links - even those that are in a parent directory of the Git repo. The issue linked specifically mentions the ability to have files open while these links are being changed and to continue to open files afterwards as long as these links are not in any way, for any length of time, cached. I don't mean to criticize your intimate knowledge and experience with Git development, but the proper way to go is to know which links are part of repository configuration and which ones are system configuration and leave the latter ones forever unresolved.

The two objections to using GetFinalPathNameByHandle seem to be (1) needing to return absolute paths sometimes, and (2) problems with relative paths and threads changing directories.
For (1), EvalSymlinks is already defined to return an absolute path when necessary. If the result of GetFinalPathNameByHandle cannot be made relative to the current directory, then the absolute one can be returned. That's entirely within the documented behavior.
For (2), there's nothing wrong with relative paths per se provided the process is not calling os.Chdir. It is Chdir (the write operation) that is not "thread-safe", not the relative path evaluation (the read operations). If you have a program that uses Chdir, then yes, use absolute paths. Pass an absolute path to EvalSymlinks and you'll get one out. But if EvalSymlinks is passed a relative path, there is no added harm in returning one.
It's really looking to me like we should use GetFinalPathNameByHandle in EvalSymlinks. Are there other reasons we should not?

It may be in-spec to always return an absolute path. But existing apps may rely on the current behavior. It's just conjecture, but the same has torpedoed previous proposals. However I'm not personally opposed to it.

It may be in-spec to always return an absolute path.

That's not what I'm suggesting. I wrote "If the result of GetFinalPathNameByHandle cannot be made relative to the current directory, then the absolute one can be returned."
I did not write "The absolute one can be returned always."
By "made relative" I meant transformed to be relative to the current directory by filepath.Rel.

Ah, I misinterpreted that. The one issue for a relative transformation is that relative paths have a max length, so some cannot be transformed, tho an app might expect it.
@ericwj did we investigate whether filepath.Rel is healthy on Windows?

Ah, I misinterpreted that. The one issue for a relative transformation is that relative paths have a max length, so some cannot be transformed, tho an app might expect it.

That's news to me. A relative path stored as a string inside a Go program should not have a max length.
If you pass a very long path to os.Open, I thought we did the appropriate rewrite to remove the length restriction. Do we not do that for relative paths?
If the Go APIs aren't handling long relative paths correctly, we should fix that, not introduce strange inconsistencies elsewhere.

I think most(?) of the APIs which take a path can handle a length over the Windows limit for relative paths, but some can't:
#36375
#21782
#41230 (related issue)
Whether it's OK to return a relative path that's not suitable for external use (e.g. a log) should also be considered.

@bk2204 did you change any permissions on this thread? If so, could you flip them back? I asked @ericwj a Q re filepath.Rel, and he's unable to respond here now.

@alexbrainman what do you think of changing the implementation of filepath.EvalSymlinks to just call GetFinalPathNameByHandle?

We should not change what filepath.EvalSymlinks implementation.
Alex

@alexbrainman I'm sure there are cases where GetFinalPathNameByHandle would return a different value, but are there cases where it would return a worse value?

are there cases where it would return a worse value?

@ianlancetaylor I am not aware of any cases with worse values. But I did not spend any time on it, and did not consider any scenarios.
We did not implement filepath.EvalSymlinks in one day - it evolved over time. It was completely broken at the start, and then people complained about their scenarios, and we fixed their scenarios. And so on. But we are not the experts in this area, we made decisions to the best of our abilities at the time. Even Microsoft changed this area in the last 10 years - they added some new features that are used by Docker. And we started with Windows XP and now we have Windows 10 - very different OS in that regard.
Perhaps just using GetFinalPathNameByHandle would have worked as well. I do not know. But I would not change filepath.EvalSymlinks implementation now. The change will definitely brake some programs.
Alex

@alexbrainman you are arguing "this is the way it's always been" rather than arguing "this is why it is the way it is". That can be a quicker reply, which is understandable. But for what it's worth, explaining why things are the way they are and what exactly would be worse about a change is more useful, because it lets us understand the possible paths forward.

I think Alex did give the "why" -- the Windows implementation initially tried to mirror the unix one, and evolved from there, as no one questioned that decision, or the lack of a GetCanonicalPath() API.
The safest thing to do now is add a new API, and deprecate EvalSymlinks.

But for what it's worth, explaining why things are the way they are and what exactly would be worse about a change is more useful, because it lets us understand the possible paths forward.

I do not know / remember why things the way they are. I do not know what exactly would be worse. I wish I could be more useful. And I am sorry that I am not useful.
Like I said before, it has been a long process and many changes. I don't remember every decision we made along the way.
I also don't use these features myself. So it is hard for me to judge whether GetFinalPathNameByHandle would be worse that what filepath.EvalSymlinks does at this moment. I guess someone actually need to spend some time and investigate every single problem that is reported with filepath.EvalSymlinks and see, if using GetFinalPathNameByHandle is a solution to this problem. Unfortunately don't have time to spend on this.
Alex

OK, it sounds like maybe it would be best to leave EvalSymlinks alone. That leaves the git-lfs/git-lfs#4012 issue, which is that nothing can turn N:\foo into \network\host\foo. If we add something to do that while also resolving symlinks, perhaps filepath.Resolve? Then we get to define a new function, and the result is always as absolute as possible (begins with / on Unix, UNC on Windows avoiding drive letter whenever possible, evaluating out symlinks, and so on).
filepath.Resolve would be the same as Unix realpath(3), and the same as Windows GetFinalPathNameByHandle.
On Unix it would be the same as filepath.EvalSymlinks(filepath.Abs(path)).
On Windows, we have no idea what EvalSymlinks does, so it's hard to state in terms of existing API.
What do people think of adding filepath.Resolve with this definition?

Sounds good. Maybe should call it Real() or RealPath() since that term is already widely known.

That sounds like a great solution.  I'm happy with any of the three proposed names.

That leaves the git-lfs/git-lfs#4012 issue, which is that nothing can turn N:\foo into \network\host\foo

If the git-lfs/git-lfs#4012 issue fix is to call GetFinalPathNameByHandle Windows API, why cannot they just call the API?

What do people think of adding filepath.Resolve with this definition?

What would filepath.Resolve documentation say? How will filepath.Resolve differ from filepath.EvalSymlinks? For example, how would you explain to a new Go user on both Linux and Windows when she should use one function instead of the other?
Thank you.
Alex

The docs should say...
On Windows, EvalSymlinks is broken for many cases. Real/RealPath/Resolve is the only API to "evaluate" symlinks. See https://github.com/golang/go/issues/40180#issuecomment-661350111
On unix, the RealPath API is equivalent to filepath.EvalSymlinks(filepath.Abs(p))
EvalSymlinks should probably be deprecated.

If the git-lfs/git-lfs#4012 issue fix is to call GetFinalPathNameByHandle Windows API, why cannot they just call the API?

We don't want to carry a large amount of non-portable system-specific code.  We are not the Git project, with many regular full-time contributors across a wide variety of platforms.  Our team is mostly two people, neither of whom use Windows on a regular basis, and we rely on Go to make our Windows support more than best effort.  We already have platform-specific code to support NTLM specially on Windows and it's going away because it's broken and nobody wants to volunteer to fix it.
I also happen to feel that requiring every user who wants to support cross-platform path canonicalization to know about and use GetFinalPathNameByHandle explicitly and use it is not going to help Windows get better support, especially when filepath.EvalSymlinks is right there and seems to do the right thing in most cases (except it doesn't).  I only happen to know about it because I've actually looked up which function to use for this purpose at some point in the past; in general, a Unix developer won't know about it and won't know to use it.

What do people think of adding filepath.Resolve with this definition?

What would filepath.Resolve documentation say? How will filepath.Resolve differ from filepath.EvalSymlinks? For example, how would you explain to a new Go user on both Linux and Windows when she should use one function instead of the other?

filepath.Resolve is used in the case when you want to canonicalize a path, and it will always result in an absolute path.  filepath.EvalSymlinks only resolves symlinks, may produce a relative path if its input is relative, and does not handle non-symlinks on Windows.  It is not suitable for canonicalizing a path on Windows and cannot be used successfully for that purpose.  filepath.Resolve should be used unless you are sure you want the behavior of filepath.EvalSymlinks.

We don't want to carry a large amount of non-portable system-specific code.

It is not a lot of code. You can, probably, just copy all the code you need from $GOROOT directory - just grep for GetFinalPathNameByHandle.
Alex

FWIW, RealPath is not great because it says Path in it.
Real is not great because it is edit-distance-1 from another function that does something similar (Rel).
I also am not sure what makes a path "real" in the first place.
Resolve is clearer about the action being done.

Based on the discussion above, adding Resolve with the semantics described seems like a likely accept.

filepath.Resolve is used in the case when you want to canonicalize a path, and it will always result in an absolute path. filepath.EvalSymlinks only resolves symlinks, may produce a relative path if its input is relative, and does not handle non-symlinks on Windows. It is not suitable for canonicalizing a path on Windows and cannot be used successfully for that purpose. filepath.Resolve should be used unless you are sure you want the behavior of filepath.EvalSymlinks.

It looks like filepath.EvalSymlinks is broken on Windows, and we are now planning to replace it with a similar function with a different name and slightly different semantics, because it's too hard to touch EvalSymlinks on Windows, where it's already broken and doesn't always work. Nobody can enumerate where it does work or it doesn't, but it seems easier not to touch it anymore rather than fixing it.
This sounds extremely rushed to me. We're adding an API that is basically a duplicate of another existing one because we don't have time to properly fix the existing one, analyzing all issues. Also it looks like there are no volunteers to work on it.
If we can get this proposal on hold for a month or so, I can attempt a thorough analysis on the history of changes of EvalSymlinks on Windows, and attempt a CL as described by Russ in https://github.com/golang/go/issues/37113#issuecomment-701509206. This would require some time and some effort to get right, but I think it would be a better long term solution rather than adding a new API. If I fail, we can still fallback to add a new API, but at least I hope we would get a better understanding around EvalSymlinks and whether it's really inherently broken forever on Windows.

Giovanni, please give the folks above some credit for our consideration of this issue. @ericwj did a careful analysis of EvalSymlinks on Windows in #40180 (read whole thread) -- which I linked above. [BTW, he was improperly locked out of this thread by an anonymous admin for no stated reason.]
It's clear that we wouldn't add EvalSymlinks today if it didn't exist; so the best long term solution is deprecating EvalSymlinks.
We cannot fix EvalSymlinks because certain changes from relative to absolute paths could break existing Windows programs. Alex will not agree to that, and he's the Windows maintainer.

@ericwj did a careful analysis of EvalSymlinks on Windows in #40180 (read whole thread)

this proposal, as it stands today, is about adding a new API, filepath.Resolve, which still violates everything mentioned in #40180. #40180 is fundamentally a software engineering problem: @ericwj is saying that the documentation should mention that the function should be rarely used. The same would apply to filepath.Resolve if it's added.

We cannot fix EvalSymlinks because certain changes from relative to absolute paths could break existing Windows programs.

Can you point me to a real-world example? It would be useful while I write the CL. As I said, I plan to go through the history of changes of EvalSymlinks, reconstruct all rationales, make sure there are regression tests, and then attempt a fix. If something is not covered by the existing tests and history of changes, I'd be happy to evaluate it.

saying that the documentation should ...

I gather you didn't read that whole thread.

Can you point me to a real-world example?

Pathnames over 250ish characters must be absolute. EvalSymlinks doesn't respect this, and correcting it would break apps expecting a relative path.
If you want to help improve Go on Windows, there are a wide variety of other bugs to tackle. This issue has been resolved.

@networkimprov can you provide a link to a real world application or at least a working, compilable code snippet that would break if EvalSymlinks is fixed as originally proposed by Russ?

@rasky, I see two problems:

EvalSymlinks has Unix-specific behavior as far as returning absolute vs relative paths. If you have a symlink x->y and you EvalSymlinks("x"), you get "y", but if you have a symlink x->/tmp/y, you get "/tmp/y". This is correct enough for Unix but much more difficult on Windows, and on both systems sometimes you just want a canonical absolute path. Depending on the symlink is surprising, even on Unix. Note that realpath(3) always returns an absolute path. That's what Resolve will do too.

On Windows, we defined "with a drive letter" as "absolute" but even that's not correct. Drive letters are not absolute in the same way as the Unix root. Instead, Windows has introduced \\ paths that are a better definition of absolute there.
Concretely, supposing that M: and N: map to the same network drive (\\host\share\), then EvalSymlinks(`M:\`) = `M:\`  and EvalSymlinks(`N:\`) , so that EvalSymlinks(`M:\`) != EvalSymlinks(`N:\`) even though they are actually the same.
In contrast, in that case, Resolve(`M:\`) == Resolve(`N:\`) == `\\host\share` .

In both cases, EvalSymlinks has behavior that is difficult to change but is less than ideal. Adding Resolve lets us introduce a different operation that behaves like realpath(3) and GetFinalPathNameByHandle instead of trying to shoehorn those into EvalSymlinks. EvalSymlinks still does exactly what it says - it evaluates symlinks - and people who want exactly that behavior can still use it for that. But people also sometimes want "get me a fully resolved, canonical equivalent of this path". It makes sense to provide that separately.

Based on the discussion above, adding Resolve with the semantics described seems like a likely accept.

Russ,
I disagree. I don't think we should add filepath.Resolve function that behaves like GetFinalPathNameByHandle Windows API.
If someone wants to use this API, we can add golang.org/x/sys/windows.GetFinalPathNameByHandle and they can use it there.
I still don't understand what proposed filepath.Resolve function will do. It will be even harder for others, who are just learning this package, to understand the difference between filepath.Resolve and filepath.EvaluateSymlinks. I would like to see documentation for that function before this proposal is even considered.
filepath.Resolve is not a good name (like utils). It tells you nothing about what the function does. To me it is a good indication that you yourself do not know what this function does.
And what problem will this new function solve? Is filepath.Resolve is supposed to be a replacement (good version) for filepath.EvaluateSymlinks? If yes, let's have someone actually try and replace filepath.EvaluateSymlinks in the current Go tree and see if it still works. If that does not work, then we need to think even harder why we need filepath.Resolve, and not just golang.org/x/sys/windows.GetFinalPathNameByHandle.
Thank you for consideration.
Alex
PS: Sorry I did not comment earlier. But I have been busy with other staff, and missed this thread altogether.

@alexbrainman all of the questions you raised have been addressed in the comments above. Re missing the thread, you last commented here 7 days ago.

@ericwj is unable to comment on this issue.  As far as we can tell, this is a GitHub bug.  It is not due to any intentional action by the Go team.
This comment is by @ericwj, sent via e-mail:

GetFinalPathNameByHandle does not canonicalize. I just ran it, trying all flags. The result is different depending on how I configure my system, depending on which path I use to access the file and depending on which flag I provide in each situation. This means canonicalization will break if the machine configuration changes, which can also happen during Git operations, and the result may change depending on which path is used to access the same file or directory.
The simplicity of EvalSymlinks is one of its problems. Any function string Foo(string) will have some of the same fundamental, unfixable problems that I documented for EvalSymlinks. Issue #40180 only proposes docs fixes because outright no-op'ing EvalSymlinks on Windows was refused in #40104, but I think the conclusion of the analysis in #40180 is that #40104 was refused because I did not argue the case effectively yet. And this proposal is just too similar, apart from perhaps thread-safety and eventually perhaps looking more often like it works.
Even suppose something like Resolve could be perfect, Go cannot handle the return value. The unfortunate truth is that Go can only handle paths with drive letters and ‘normal’ UNC paths in the form \\server\share (without prefix). I have evaluated about half of all path/filepath API’s and the prefixes GetFinalPathNameByHandle yields break about every one of them. You cannot depend on stripping the prefixes, either.
This proposal goes wrong the same way path/filepath has, which imho is thinking *nix can be ported unmodified to a fundamentally different operating system without proper attention, expertise involved, or testing. And for letting fundamentals slide for the benefit of a bit of simplicity, offering no alternative besides P/Invoke in cases where this makes things fall apart.
I suggest writing the fundamentals, especially well for Windows, testing and reviewing meticulously, while iterating towards an API suitable for the general public on top of those fundamentals, before proposing inclusion in any standard library. If the problem is just UNC mappings, much more appropriate WNet* functions exist to translate DOS device names. I think object identifiers are the more general solution for Git LFS, in favour of path strings. Wrapped in a suitable, portable abstraction to obtain and compare them.

@rasky, I see two problems:

EvalSymlinks has Unix-specific behavior as far as returning absolute vs relative paths. If you have a symlink x->y and you EvalSymlinks("x"), you get "y", but if you have a symlink x->/tmp/y, you get "/tmp/y". This is correct enough for Unix but much more difficult on Windows, and on both systems sometimes you just want a canonical absolute path. Depending on the symlink is surprising, even on Unix. Note that realpath(3) always returns an absolute path. That's what Resolve will do too.

Yes, I agree that we cannot exactly reproduce this if we switch EvalSymlinks to GetFinalPathNameByHandle. Though I'm not fully sure that the clients of EvalSymlinks really do behave on this implementation detail (that is, unfortunately, documented). What if drop that sentence from the documentation, and just say that EvalSymlinks can either return a relative or an absolute path? Do you have an idea on whether clients really do rely on this specific behavior, even on Linux?

On Windows, we defined "with a drive letter" as "absolute" but even that's not correct. Drive letters are not absolute in the same way as the Unix root. Instead, Windows has introduced \\ paths that are a better definition of absolute there.
Concretely, supposing that M: and N: map to the same network drive (\\host\share\), then EvalSymlinks(`M:\`) = `M:\`  and EvalSymlinks(`N:\`) , so that EvalSymlinks(`M:\`) != EvalSymlinks(`N:\`) even though they are actually the same.
In contrast, in that case, Resolve(`M:\`) == Resolve(`N:\`) == `\\host\share` .

Notice that realpath on Linux doesn't follow mountpoints as well:
# mkdir a
# mkdir b
# mount --bind a b
# cd a
# touch foo
# cd ..
# realpath a/foo
/tmp/a/foo
# realpath b/foo
/tmp/b/foo

Is realpath(3) on Linux different than filepath.Abs(filepath.EvalSymlinks(path))? I might be missing something here.

In both cases, EvalSymlinks has behavior that is difficult to change but is less than ideal. Adding Resolve lets us introduce a different operation that behaves like realpath(3) and GetFinalPathNameByHandle instead of trying to shoehorn those into EvalSymlinks. EvalSymlinks still does exactly what it says - it evaluates symlinks - and people who want exactly that behavior can still use it for that. But people also sometimes want "get me a fully resolved, canonical equivalent of this path". It makes sense to provide that separately.

So my understanding is that if this proposal is accepted, we end up with:

Linux: Resolve and EvalSymlinks are basically the same function, with the only difference is that Resolve always returns an absolute path, while EvalSymlinks returns a path constructed following the symlinks, so it might be absolute or relative depending on how the symlinks were created. Neither of them see through mountpoints in any way.
Windows: EvalSymlinks (after bugfixing) would possibly go through all kind of NTFS links (with the same relative/absolute handling than Linux), and it would not follow mountpoints. Resolve instead would also resolve mountpoints.

It looks like the biggest gain here is that we somehow exposes GetFinalPathNameByHandle which is a "common" function used in other frameworks / languages to canonicalize paths. Since we're missing it, we face interoperability problems. In fact, semantically speaking, we're going to have a Resolve function which behaves differently between Windows and Linux wrt mountpoints. And BTW we're also getting into another corner: if somebody puts in the effort of adding mountpoint resolution to Resolve on Linux, we end up with interoperability problems again, because the rest of the world might still just be calling realpath(3) on Linux.
A slightly different proposal is to add filepath.SameFile(path1, path2 string). This would use GetFinalPathNameByHandle on Windows, and realpath on Linux (but can be further improved on Linux by looking at inode number, etc.). It would solve the real-world problem of many softwares without having to define what a canonical path name is, and would also relax the software engineering concern of preferring to keep around the original input path rather than a resolved one.

So my understanding is that if this proposal is accepted, we end up with:
* Linux: `Resolve` and `EvalSymlinks` are basically the same function, with the only difference is that Resolve always returns an absolute path, while `EvalSymlinks` returns a path constructed following the symlinks, so it might be absolute or relative depending on how the symlinks were created. Neither of them see through mountpoints in any way.

Yes, because bind mounts on Linux are not identical.  One directory may be read-only while the other may be read-write.  Bind mounts also, by default, don't include submounts and are therefore not necessarily equivalent.

* Windows: `EvalSymlinks` (after bugfixing) would possibly go through all kind of NTFS links (with the same relative/absolute handling than Linux), and it would not follow mountpoints. `Resolve` instead would also resolve mountpoints.

As far as I understand it, these mountpoints are functionally identical.  They are more like opaque symlinks rather than the typical Unix mountpoints with different mount options.

It looks like the biggest gain here is that we somehow exposes GetFinalPathNameByHandle which is a "common" function used in other frameworks / languages to canonicalize paths. Since we're missing it, we face interoperability problems. In fact, semantically speaking, we're going to have a Resolve function which behaves differently between Windows and Linux wrt mountpoints. And BTW we're also getting into another corner: if somebody puts in the effort of adding mountpoint resolution to Resolve on Linux, we end up with interoperability problems again, because the rest of the world might still just be calling realpath(3) on Linux.

This is exactly the behavior that's desired: canonicalizing the path in the way the operating system does that.  Languages that use the system C library don't have this problem because they use the system C library functions for this purpose and implement those functions' behavior.  I would literally define the expected behavior in terms of the system's C library, since that's what other languages (e.g., Rust) do.

A slightly different proposal is to add filepath.SameFile(path1, path2 string). This would use GetFinalPathNameByHandle on Windows, and realpath on Linux (but can be further improved on Linux by looking at inode number, etc.). It would solve the real-world problem of many softwares without having to define what a canonical path name is, and would also relax the software engineering concern of preferring to keep around the original input path rather than a resolved one.

That doesn't really assist in my use case of finding out whether one path is inside another.

Even suppose something like Resolve could be perfect, Go cannot handle the return value. The unfortunate truth is that Go can only handle paths with drive letters and ‘normal’ UNC paths in the form \server\share (without prefix). I have evaluated about half of all path/filepath API’s and the prefixes GetFinalPathNameByHandle yields break about every one of them. You cannot depend on stripping the prefixes, either.

This should mostly be fixed by my CL (https://go-review.googlesource.com/c/go/+/263538). Please test it and let me know.
It also rectifies some EvalSymlinks issues on Windows reported in your #40180 table. I have a followup CL (still not mailed) that fixes another set of issues in EvalSymlinks. NOTE: I'm not touching the semantic of the function as it is implemented now:
I'm just trying to fix implementation bugs so that we can see a clearer picture of where we are now.

There's clearly new, substantive discussion here. Moving back to Active.

all of the questions you raised have been addressed in the comments above. Re missing the thread, you last commented here 7 days ago.

@networkimprov I cannot find them. Can you, please, provide references to me? Thank you.
Alex

So my understanding is that if this proposal is accepted, we end up with:
* Linux: `Resolve` and `EvalSymlinks` are basically the same function, with the only difference is that Resolve always returns an absolute path, while `EvalSymlinks` returns a path constructed following the symlinks, so it might be absolute or relative depending on how the symlinks were created. Neither of them see through mountpoints in any way.

Yes, because bind mounts on Linux are not identical. One directory may be read-only while the other may be read-write. Bind mounts also, by default, don't include submounts and are therefore not necessarily equivalent.

To be honest, I disagree on this. The two underlying files are indeed the same and in fact os.SameFile returns true. It sounds absolutely surprising that os.SameFile(a,b) differs from filepath.Resolve(a) == filepath.Resolve(b).
Moreover, the same situation happens on Windows. You can easily have C:\FOO and \\LOCALHOST\BAR\FOO being the same file with different permissions. I'm not sure why on Windows it should be necessary to have filepath.Resolve returning true in that situation, while on Linux it should return false on bind mounts (or any other kind of mount really).
I think the only correct answer here is that the request is not to establish a cross-platform semantic, but just add a function that calls realpath on Linux and GetFinalPathNameByHandle on Windows because this is what most open source projects have ended up doing. I sympathize with the goal of interoperability, but I'm just hoping that we can find a better solution.

A slightly different proposal is to add filepath.SameFile(path1, path2 string). This would use GetFinalPathNameByHandle on Windows, and realpath on Linux (but can be further improved on Linux by looking at inode number, etc.). It would solve the real-world problem of many softwares without having to define what a canonical path name is, and would also relax the software engineering concern of preferring to keep around the original input path rather than a resolved one.

That doesn't really assist in my use case of finding out whether one path is inside another.

Looking at https://github.com/git-lfs/git-lfs/issues/4012, it looked like filepath.SameFile would be enough, to compare if the git root actually matches. From your answer, I understand that you actually need whether a specific file of the repo is within the git repo (with the git root path being read through git and thus canonicalized through GetFinalPathNameByHandle)? Can you please elaborate on the use case you are mentioning?
I have one question on this. Say that we have \\network\repo mounted as N:, just like in the referenced bug. Within the repo, there's a committed symlink N:\mylink that points to a file on a local disk C:\file. Would you expect filepath.Resolve("N:\\mylink") to return \\network\repo\mylink or C:\file? If I followed you correctly, you would need the former, but GetFinalPathNameByHandle obviously returns the latter.

I think the only correct answer here is that the request is not to establish a cross-platform semantic, but just add a function that calls realpath on Linux and GetFinalPathNameByHandle on Windows because this is what most open source projects have ended up doing. I sympathize with the goal of interoperability, but I'm just hoping that we can find a better solution.

This is what Git does and what other projects do, and it's the semantics we need.  It is the same semantics that other programming languages provide.  I am interested in Go working with other software that is also on the system in a compatible way.
You may argue that Windows and Unix should provide the same semantics for path canonicalization, and perhaps they should.  However, Microsoft was fully aware of Unix path semantics and behavior, and clearly chose deliberately to do things in an incompatible and much more complicated way.  I feel that was a mistake, but we're stuck with it now.

Looking at git-lfs/git-lfs#4012, it looked like filepath.SameFile would be enough, to compare if the git root actually matches. From your answer, I understand that you actually need whether a specific file of the repo is within the git repo (with the git root path being read through git and thus canonicalized through GetFinalPathNameByHandle)? Can you please elaborate on the use case you are mentioning?

I need to change the directory to the root of the repository to make GIT_WORK_TREE work.  I also need to be able to resolve any other Git environment variables to their canonical paths so I can set them appropriately before invoking Git, and all of those need to be canonicalized appropriately and compatibly with Git.  This is also the behavior that anyone else working with Git will need to implement.
I also need to be able to determine if a path the user has provided is within the repository and fail if it is not.  I can do that by removing the trailing file component, canonicalizing the directory, and appending the file path.  That works to tell me if the component is in the repository, whether or not the file itself is a symlink.  Repositories are not allowed to span file systems.

I have one question on this. Say that we have \\network\repo mounted as N:, just like in the referenced bug. Within the repo, there's a committed symlink N:\mylink that points to a file on a local disk C:\file. Would you expect filepath.Resolve("N:\\mylink") to return \\network\repo\mylink or C:\file? If I followed you correctly, you would need the former, but GetFinalPathNameByHandle obviously returns the latter.

I don't need to canonicalize file names, only directory names.  However, I do need the standard GetFinalPathNameByHandle semantics for that.  Other people may have different use cases and will also need the standard semantics.
I want to be clear that I'm not asking for this just for Git LFS.  I'm asking for this because these are the standard platform semantics and they should be available in a cross-platform way.  I fully realize that they are inconsistent across platforms.  I want the same behavior out of Go that I get out of languages that use the system C library.  I'm not looking for special functionality to handle my special case, I'm looking for generic tools that make Go programs compatible with non-Go programs.
It's fine that Go does not wrap the system C library, but if it chooses not to do that, it needs to be compatible with the semantics of the platform as if it did, and not provide behavior that is materially different.  Trying to sell me on a behavior that is consistent across platforms when that is not practically achievable in a useful way isn't helpful.  That's what we have now with EvalSymlinks, and everyone agrees that it's not the right decision.

I want to be clear that while there may be other proposals, such as filepath.SameFile, those are separate and independent proposals and are outside of the scope of this proposal, which is to implement a function which canonicalizes paths.  I don't dispute that they may be valuable and important proposals, but they are not this one, and as such, they should probably be discussed in their own issues, not this one.
I also want to be clear that I'm an experienced developer who understands his problem space well and am confident I know what's best for it, so it's unhelpful (and frankly, insulting) to imply that some other solution would be better for my needs or that I should adopt a different design.  If people disagree with the Git project's decisions to adopt this technique (with which I am implementing compatibility) and feel that they were unwise, those concerns should be addressed to the project's mailing list, although I suspect that in this case such a message will be about as welcome as I have found it.
As I've mentioned, my goal here is to expose a cross-platform way to canonicalize paths as the operating system defines that.  Whether the operating system designers made a provident design decision or whether the operating system has provided desirable semantics is outside the scope of this proposal.  I assume for the purposes of this proposal that the fact that the operating system is successful means that its developers were competent and implemented desirable, useful, and well-thought-out features.  The facts remain that other common, successful languages implement functionally equivalent designs, and that while similar concerns have been expressed during the design of those implementations, nobody has yet proposed a better or more universally desirable solution.  Moreover, Go attempted to provide a solution for this problem in EvalSymlinks which demonstrates that this is a problem that people need solved.  Additionally, canonicalizing paths is considered useful for security purposes by Carnegie Mellon's Software Engineering Institute, which is considered a reputable source of secure coding advice.  Finally, this functionality is implemented in the popular realpath utility, which is widely shipped for scripting across platforms (including Windows).
For these reasons, I believe that the proposal as it stands should be implemented.  I take no position on proposals for other functions, as they are outside my scope.

Brian, the Windows maintainer has objected to this proposal, raising Q's in https://github.com/golang/go/issues/37113#issuecomment-712704339. It might help to point out where Russ or Ian have previously addressed those Q's.

Looking at how this problem is solved elsewhere, if a new function is to be added, perhaps a better name would be Canonical(). Rust has canonicalize and C++'s filesystem library has canonical, which Microsoft's Standard C++ Library implements using GetFinalPathNameByHandle.
Canonical was not immediately obvious to me before I looked at this issue, and I think that's a fair concern. However, looking at modern solutions to this problem, once you know of it, the naming familiarity between languages might be a good thing.

@ericwj is unable to comment on this issue. As far as we can tell, this is a GitHub bug. It is not due to any intentional action by the Go team.

@ianlancetaylor The original author of this issue has blocked @ericwj because they felt insulted by the help they were providing. When you block a user on GitHub, they're unable to comment on issues/pull requests you're the owner of.

@saracen Thanks for the explanation.
@bk2204 Would you be willing to unblock @ericwj?  I think they have useful information to contribute for moving this proposal forward.  Thanks.

I do believe people have the right to decide who they'd like to block, assuming they are not in a government position, and that asking people to unblock others is inappropriate.  That's often an approach people use to engage in continued harassment of women and other folks who tend to be underrepresented in tech.  That's not what's happening here, and I'm sure you had only good intent to move the discussion forward in a productive way, but since we often lack all the context to know people's reasons for blocking others, it's generally best not to ask.
For the limited purpose of advancing this proposal, I have unblocked @ericwj for the moment.
I'm fine with Canonical() or Canonicalize() as a name since it's used elsewhere.  That may be more intuitive than Resolve(), but since we'll have sane documentation either way, I'm not very picky about the name.

Thanks for that. However I don't know whether it is useful for me anymore to contribute a lot, since you are very clear you will have it your way and no other.
I can't know whether you feel like a woman or a minority in tech. If you do, don't blame that on me and don't talk others into guilt for trying to be constructive and open.  I don't know anything else about you. I think you should check the news and find out who pays your salary - I believe his name is Satya. And then consider whether your stance is appropriate at all, for better progress of all of us, let alone justifiably grateful.
My opinion is that the canonicalize that was linked here in C++ is pretty much straightforward enough to port yourself and write your software with it. Not having done that at all but requiring or even demanding that we dance to your trousers legs (whether you wear them or not) and include a copy in the standard library is tremendously lazy for starters. Do the work, test it, see if it actually does what you need instead of claiming that with nothing to show for it, then come back with your own authority, reasons and consequences.
It took me 70 lines of code in C# to test both GetFinalPathNameByHandle and GetFileInformationByHandleEx to shoot down your arguments in favour of canonicalization on Windows and to test the alternative that does what you started with to ask for in the first place.

@bk2204 My sincere apologies.  I was in the wrong, and I've done you harm.  I'm sorry.
@ericwj Your comment violates the Go Community Code of Conduct (https://golang.org/conduct), and I've deleted it.

the Windows maintainer has objected to this proposa

@networkimprov I assume you refer to me as the Windows maintainer. That is not my title. Please stop inventing things. You give false impression to others.
Thank you.
Alex

Alex, sorry, I had gathered that you're recognized as the principle Windows maintainer. I didn't mean to invent a title.

Well, I'm sorry for having been slightly edgy due to the total irrelevance of that statement with which he accompanies the news that he unblocked me.
I care to repeat that Brian should argue his case from a position of hands on, mature knowledge rather than having us collect that for him and demanding his proposal be accepted unmodified and that none of us is in a better position than to test and evaluate the suitability of including the proposal of porting canonicalize unmodified into the go standard library for Windows than he himself and the company he works for. Further I can only repeat my previous arguments and fix the oversight of not mentioning the obvious - that EvalSymlinks is implemented with GetFinalPathNameByHandle and that some of the problems even mentioned in this issue will be exactly the same with canonicalize for this reason.

I'm going to close this issue and I'd like to ask the core team to lock it at their earliest convenience.  I don't feel this discussion is going in a productive way and I don't wish to pursue this issue or proposal further.  If other folks think this a similar proposal will be valuable, I'm happy for them to drive it without my involvement.

Alex, sorry,

No worries. You did not hurt my feelings or anything.
I just did not want other people to get impression that they need to convince me more than others. Any proposal here should be judged purely on its merits, not on authority of proposal author or judges.

I had gathered that you're recognized as the principle Windows maintainer.

I am Go contributor, like many others. I gathered bunch of experience over the years. But that does not make me principle Windows maintainer. I am not responsible for Windows port - Go Team is.

I didn't mean to invent a title.

No worries. No harm done.
Alex

I have written a new proposal, #42201, to replace this one.
I will now lock this issue.


token.title string equality not working

I am trying to implement below delegate method:
func tokenView(tokenView: KSTokenView, shouldAddToken token: KSToken) -> Bool {
        var shouldAddToken = true

        if token.title == "f" {
            shouldAddToken = false
        }

        print("~~ tokenTitle : \(token.title) , shouldAddToken : \(shouldAddToken)")

        return shouldAddToken
    }

Problem is - though I have entered 'f' in tokenView shouldAddToken is coming to be nil, i.e. equality check is failing :(
Here is output of print statement in console:
~~ tokenTitle : f , shouldAddToken : true
Please suggest

Issue is fixed in latest commit https://github.com/khawars/KSTokenView/commit/f3969467a56fb1d1d061904bacea0332f49efb9e
Check example AutolayoutExample.


update placement and placementdecision to v1beta1

Ref: https://github.com/stolostron/backlog/issues/19189
Signed-off-by: haoqing0110<EMAIL_ADDRESS>

/assign @qiujian16 @elgnay

/assign @zhiweiyin318

/hold

/unhold

/approve
@elgnay please take a look and tag.

/lgtm


Add new test for all_disjoint

Adding a new test for all_disjoint.
My understanding of all_disjoint is that each number in all sets passed to all_disjoint should be different. Therefore, two sets [1,2] and [4,4] should not be disjoint.
However, if this is not the case and I've understood the test wrong, then all_disjoint function might have to be updated.

https://github.com/fastai/fastai_dev/pull/209/commits/ad95642f64a4dc04cd12879e410ebc4b293c2bd7
I had to do this but I don't think this due to my change/addition of new test. Perhaps existing difference at my current fork.

I don't think this is part of what I want the documented behavior of this function to include. Thanks anyway! :)


add fn_ prefix to examples

I think the functions become available with an fn_ prefix.  I'll submit an issue to make that optional, as I'd prefer calling the function without fn_

I'd also be ok not showing the fn_ prefix version here and using fn() instead.

I'd also be ok not showing the fn_ prefix version here and using fn() instead.

that is, this?
{{ alias('homepage') }}

{{ fn('alias') }}

Or showing both:
// will be available as "fn('someMethod')" or "fn_someMethod()" in twig


Improve error reporting

Moved from https://github.com/lukechilds/hyperdex-bugtracker/issues/6

When trying to execute a trade and only having one UTXO, OR trying to sell less than 10% of a UTXO (I can't tell which cause the message is unclear :-) ),
I get a 'something unexpected happened'.

Suggestion: message should be clear on next steps.
-@jansako

Duplicate of #180


feat: integrate disable Lifecycle rule api for to remove lifecycle rule of bucket

Fixes #20

Does this API remove all lifecycle rules? if so, it should be named disableLifecycleRules rather than rule, and if not there should be documentation on which rule specifically would be removed and how to control it

Does this API remove all lifecycle rules? if so, it should be named disableLifecycleRules rather than rule, and if not there should be documentation on which rule specifically would be removed and how to control it

@JesseLovelace yes,this API remove all lifecycle rules. so you are right we should named it as disableLifecycleRules.

@JesseLovelace PTAL

@JesseLovelace PTAL

@frankyn @JesseLovelace  gentle ping.

@JesseLovelace is on point for this one. I'll bring it up in our meeting early next week (Tuesday).
Thank you for your patience @athakor

LGTM overall just fix the typos above

Will need to look into the linkage monitor failure.
As a follow-up, do you think it'd be possible to make so that setting the lifecycle rules to null and updating also works?

Will need to look into the linkage monitor failure.
As a follow-up, do you think it'd be possible to make so that setting the lifecycle rules to null and updating also works?

@JesseLovelace here we cant do anything to fix this  linkage issue,we can fix it in java-storage-nio-FakeStorageRpc. I dont think so,it'would be possible to set the lifecycle rules to null and updating works without api call.

@frankyn @JesseLovelace  here, how we can move ahead with above situation?


Implement Bot API v4.4 & Bot API v4.5

Telegram launched their v4.4 bot API:
https://core.telegram.org/bots/api#july-29-2019
List of changes:

Added support for animated stickers. New field is_animated in Sticker and StickerSet objects, animated stickers can now be used in sendSticker and InlineQueryResultCachedSticker.
Added support for default permissions in groups. New object ChatPermissions, containing actions which a member can take in a chat. New field permissions in the Chat object; new method setChatPermissions.
The field all_members_are_administrators has been removed from the documentation for the Chat object. The field is still returned in the object for backward compatibility, but new bots should use the permissions field instead.
Added support for more permissions for group and supergroup members: added the new field can_send_polls to ChatMember object, added can_change_info, can_invite_users, can_pin_messages in ChatMember object for restricted users (previously available only for administrators).
The method restrictChatMember now takes the new user permissions in a single argument of the type ChatPermissions. The old way of passing parameters will keep working for a while for backward compatibility.
Added description support for basic groups (previously available in supergroups and channel chats). You can pass a group's chat_id to setChatDescription and receive the group's description in the Chat object in the response to getChat method.
Added invite_link support for basic groups (previously available in supergroups and channel chats). You can pass a group's chat_id to exportChatInviteLink and receive the group's invite link in the Chat object in the response to getChat method.
File identifiers from the ChatPhoto object are now invalidated and can no longer be used whenever the photo is changed.
All webhook requests from the Bot API are now coming from the subnets <IP_ADDRESS>/20 and <IP_ADDRESS>/22. Most users won't need to do anything to continue receiving webhooks. If you control inbound access with a firewall, you may need to update your configuration. You can always find the list of actual IP addresses of servers used to send webhooks there: https://core.telegram.org/bots/webhooks.
As of the next Bot API update (version 4.5), nested MessageEntity objects will be allowed in message texts and captions. Please make sure that your code can correctly handle such entities.

Bot API v4.5: https://core.telegram.org/bots/api#december-31-2019
Added support for two new MessageEntity types, underline and strikethrough.
Added support for nested MessageEntity objects. Entities can now contain other entities. If two entities have common characters then one of them is fully contained inside the other.
Added support for nested entities and the new tags <u>/<ins> (for underlined text) and <s>/<strike>/<del> (for strikethrough text) in parse mode HTML.
Added a new parse mode, MarkdownV2, which supports nested entities and two new entities __ (for underlined text) and ~ (for strikethrough text). Parse mode Markdown remains unchanged for backward compatibility.
Added the field file_unique_id to the objects Animation, Audio, Document, PassportFile, PhotoSize, Sticker, Video, VideoNote, Voice, File and the fields small_file_unique_id and big_file_unique_id to the object ChatPhoto. The new fields contain a unique file identifier, which is supposed to be the same over time and for different bots, but can't be used to download or reuse the file.
Added the field custom_title to the ChatMember object.
Added the new method setChatAdministratorCustomTitle to manage the custom titles of administrators promoted by the bot.
Added the field slow_mode_delay to the Chat object.


feat: Add library to manage context configurations

Description:
see PR title
Is this change user facing?

[ ] Yes
[x] No

References (if applicable):

Sounds good Gyani I can roll back to config, I was trying to be creative but maybe Config with plural "contexts" will work just fine


refactor docs layout

Split the 'usage' page into several sub-pages, to make finding stuff easier

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (520505d) 47.12% compared to head (0efb178) 47.12%.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #123   +/-   ##
=======================================
  Coverage   47.12%   47.12%
=======================================
  Files           7        7
  Lines         469      469
=======================================
  Hits          221      221
  Misses        248      248

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


!!! Merge into the Neos 9.0 core

During the Neos Sprint 04/2023 the Neos team decided that this module will replace the workspace module in the Neos core.
Therefore I will work with other team members to adapt and merge the functionality.

[ ] Mark the package as deprecated after the 9.0 release
[ ] Implement ACL feature in the core
[ ] Move new UI to the core
[ ] Adjust features to new CR
[ ] Adjust code to PHP 8.2

see https://github.com/neos/neos-development-collection/pull/5132


ECS Operator

Build a ECS Operator
Build a ECS Hook
Using "aws_credentials"  as a connection id

Can someone help me how to test an external service? I am developing an AWS ECS operator.

@mistercrunch Following instructions to move operator and hook to ecs contrib folder. I tested the ECS Operator and already using it, but it is no testable.
How can make it pass to the code coverage?

@mistercrunch: Following what you said in other issue, I moved the ECS Operator and the ECS Hook to the contrib folder.

Can you move the commented tests out of core.py? Maybe airflow/contrib/tests? We should probably make sure that the ./run_unit_tests.sh command doesn't find and run those or the CI tests will fail.

@mistercrunch: I reverted the core.py. My integration tests only makes sense on my enviroment so I didnt pushed it anymore.
I implemented the sftp hook too. I need an operator to copy file from a sftp to S3.

Why am I having erros on code coverage?

I started reading the code and there are leftover comments and the code needs to be linted. Check out the CONTRIBUTING.md and take time to use a linter like flake8

Thanks! @mistercrunch
I am sorry for the mistakes, I am kind newbie contributing to open source projects. I will take a look.

@pedrorjbr are you still working on this? if not i'd like to take it over

@rfroetscher No, I am not! I made a fork and my team are working with a forked version of airflow. I appreciate any help. Basically we need to put the code in PEP8.

@rfroetscher @pedrorjbr Any updates on this?

I'm currently on vacation and will be until 9/12. Feel free to grab this.
On Aug 22, 2016 11:44 AM, "Vitor Baptista"<EMAIL_ADDRESS>wrote:

@rfroetscher https://github.com/rfroetscher @pedrorjbr
https://github.com/pedrorjbr Any updates on this?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/incubator-airflow/pull/652#issuecomment-241509868,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHBeGawzTfACP0bggHF6KYb_tU6PWoI9ks5qie4DgaJpZM4GkLLr
.

@vitorbaptista: No updates at all.

We're cleaning up the open PRs list, please rebase within the next few days or we'll close the PR, in which case you can always reopen the PR from your rebased branch.

+1 !! 🍰


pub AxumConnectServiceGenerator

Hi @AThilenius
It is not complicated to use AxumConnectServiceGenerator directly, I think we can pub it.
Please review it. thank you.
use std::{env, fs, path::PathBuf};

use axum_connect_build::AxumConnectServiceGenerator;

//
fn main() -> Result<(), Box<dyn std::error::Error>> {
    let descriptor_path = PathBuf::from(env::var("OUT_DIR").unwrap()).join("proto_descriptor.bin");

    prost_build::Config::new()
        .file_descriptor_set_path(&descriptor_path)
        .compile_well_known_types()
        .extern_path(".google.protobuf", "::pbjson_types")
        .service_generator(Box::new(AxumConnectServiceGenerator {})).compile_protos(
            &[
                "../xxx.proto",
            ],
            &[".."],
        )?;

    let descriptor_set = fs::read(descriptor_path)?;
    pbjson_build::Builder::new()
        .register_descriptors(&descriptor_set)?
        .build(&[".xxx",])?;

    Ok(())
}

Hi @vkill, thanks for the PR! Unfortunately I don't want to pub the service generator because on it's own the output code will be invalid. I have to do all kinds of weird monkey-patching of the resulting rust code (see lib.rs). The service generator impl exists strictly to interop with Prost.
I'm really unhappy with all of the proto stuff, and am still deciding what to do. Prost has been extremely challenging to work with and I find myself again at an impass, needing google::protobuf::any support for Connect error details from Prost which has no built-in RTTI support or even the ability to hook per-message code generation. At some point enough hacks and it starts to look like a proto code generator itself, so I'm tempted to write a 'lean and mean' proto compiler and runtime lib but not sure that I want to delay a 1.0 launch of axum-connect for it.
But regardless, I hope that all of this code goes away sometime soon or is significantly cleaned up.


An error occurs in the miss.rmiss in the raytracingshadows demo. I'm using visual studio2019

The ray tracing extension requires a newer SPIR-V environment.
You need to change it in the extensions' setting pane to e.g. spirv1.4:


[Serialization] Handle UnboundGenericTypes of generic typealiases.

Simple latent asserts-only bug, rescused from #8737. Thanks to Slava for the test case.

@swift-ci Please smoke test and merge


Golden & roundtrips on all Api types

Issue Number
#91

Overview

[x] Added roundtripAndGoldenPerType + machinery to run tests on all Api Capture- and return-types.
[x] ~WIP machinery for taking apart ADTs and running roundtripAndGolden on each component.~

[x] ~It was able to run roundtripAndGolden for the type of each field of a record~
[ ] ~I was unable to hook it up with the Api discovery~
[ ] ~I didn't figure out how to "catch" an hspec failure, such that we could then run more detailed tests on the components on the failing type (see comment for GRecursiveCheckJSON)~

Comments

It'd be nice one day to also get the roundtrip automagically for nested types, yet

@KtorZ I just discovered that
hspec-golden-aeson contains
goldenADTSpecs :: forall a. (ToADTArbitrary a, Eq a, Show a, ToJSON a, FromJSON a) => Settings -> Proxy a -> Spec

I'm checking how it works. If it is close to trivial we could add now, otherwise later. Probably in a separate pr regardless.

@Anviking
->  TODO: Go through the servant combinators. ReqBody?
No need. We'll add them as we add API types.

Closing for now, the value wasn't clear enough. A few problems:
Lacking reduction of test cases
The pr had logic for generating test-cases for a instead of [a], but this was not generalised to other types than List.
Not testing ADT components
-- We are relying heavily on generically derived Aeson instances. This is used
-- to trace the failure of<EMAIL_ADDRESS>--
-- Example:
-- @
--    data Record { a :: A, b :: B, c :: C, ds :: [D] }
--        deriving (Generic, ToJSON, FromJSON)
-- @
--
-- If we hand-write faulty Aeson instances for 'D', we'd like to discover that
-- easily when checking roundtrips and goldens on 'Record'.

Dangers of regeneration
If we believe we're making a purely additively change to a type, but screw something up, the tests won't help us. They are re-generated. (actually, no, but not re-generating them is also a problem)
For the future

In its current form this ensure that we haven't missed any roundtrip or golden tests in the Api-type. It serves like a backup, but shouldn't replace manually listed test cases.

With some more careful thought, we might be able to do something similar — but valuable —in the future.


Remove description prefetch for Geo Areas

Prefetching helps save queries, however in some cases
it provides an incorrect description for Geographical
Areas. To remedy this prefetching has been removed from
Geographical Area views.

Codecov Report

Merging #213 (6fb59d1) into master (0a1e70e) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #213   +/-   ##
=======================================
  Coverage   61.68%   61.68%
=======================================
  Files         176      176
  Lines       10560    10560
  Branches      788      788
=======================================
  Hits         6514     6514
  Misses       3866     3866
  Partials      180      180

Impacted Files
Coverage Δ

geo_areas/views.py
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0a1e70e...6fb59d1. Read the comment docs.

I imagine we'll be needing to apply similar solutions to this all over the place

I imagine we'll be needing to apply similar solutions to this all over the place

We really need more tests... :grimacing: including regression tests for this sort of thing, it's the kind of bug that's so easy to reintroduce.


Added B tree implementation

Implementation of BTree Data Structure

@bhaveshlohana
can you add 'hacktoberfest-accepted' label to this PR?


Add primitive 2D meshes

Adds RectangleMesh2D, CircleMesh2D, and CapsuleMesh2D, inheriting from the virtual base class PrimitiveMesh2D.
Allows drawing 2D shapes by using the existing (but so far perhaps underused) MeshInstance2D node:

Example project: godot-primitive-mesh-2d.zip

The two fancy shapes on the right are obtained by applying vertex shaders to primitive meshes (kind of silly, but it works).
If you want a triangle, use a Polygon2D instead.
The code is based on the code for primitive 3D meshes.

Closes https://github.com/godotengine/godot-proposals/issues/1126

If you want a triangle, use a Polygon2D instead.

Join the 3-Vertex-CircleShape-Club.

I like it, but to be honest, I'm not sure whether this really closes the linked proposal.
Is this meant as an alternative to https://github.com/godotengine/godot/pull/99210 or complementary? Since the problem with this approach is that compared to the other PR you can't have outlines and fake anti-aliasing (which are rather important IMO).

...and fake anti-aliasing (which are rather important features IMO).

The antialiased is pure placebo that does exactly nothing.
A leftover from the Godot 3 days that used canvas item triangle arrays while Godot 4 uses actual meshes.

If you want a triangle, use a Polygon2D instead.

Join the 3-Vertex-CircleShape-Club.

I was going to object, pointing out the radius property would not be "respected", but instead I just made it work.

I like it, but to be honest, I'm not sure whether this really closes the linked proposal.

The proposal asks for the ability to draw simple 2D shapes. It's not clear if those who voted on it also wanted outlines and fake anti-aliasing (or different kinds of triangles).

Is this meant as an alternative to https://github.com/godotengine/godot/pull/99210 or complementary?

It is meant to fill a feature gap (primitive 2D meshes). It's partly an alternative to that PR.

Since the problem with this approach is that compared to the other PR you can't have outlines and fake anti-aliasing (which are rather important features IMO).

I do agree those are nice features. Outlines should be achievable with this PR, e.g. with a material. I don't know much about how to achieve fake anti-aliasing, but I'll look into it.

For meshes, is it really expected that fake antialiasing is supported, when there is real 2D edge antialising available by using MSAA? If we go by parity with 3D meshes, there shouldn't be an expectation that meshes antialias themselves, IMO.
(I'm also not sure if this closes the linked issue, but that seems like a separate concern)

this is very useful for those of us directly using the rendering server who want easy to use pre made meshes :+1:

From the notes of the rendering team meeting on 19/12/2024:

AA should be built into the mesh.

Color should be set to white, then add a skirt around the mesh with color set to transparent white

The latest push adds an antialiased option to all primitive 2D meshes. Example project: godot-primitive-mesh-2d.zip

(Anti-aliasing is disabled for the procedural shapes on the right.)
Testing / reviews welcome!

mostly works great, but rect mesh anti aliasing seems broken when the subdivide counts on each axis are mismatched


Demo not working in Safari or Firefox

Trying out vibrant.js in the jsfiddle on the webpage fails, returning the follow error:
/TypeError: cmap.vboxes is undefined
Edit: just noticed the disclaimer regarding image dropping only working in Chrome. Why is this the case?

And also doesn't work with Chrome, but that gives an error like #37.

Chome on Mac Sierra gives this error when dropping an image on the demo at http://jariz.github.io/vibrant.js/:
Uncaught TypeError: Cannot read property 'map' of undefined
    at new Vibrant (http://cdn.rawgit.com/jariz/vibrant.js/master/dist/Vibrant.js:43:35)
    at FileReader.progress (coffeescript:24:19)

Same error here.. Do you have any news about this issue ?

vibrant.js is deprecated. Please try node-vibrant instead and see if this issue persists. You are welcome to submit a issue there if that's the case.


Make user group in security manager configurable.

https://github.com/azkaban/azkaban/issues/2393

@djaiswal83 Can you please kindly review this change? Thanks.


Fixed issue #85

@robes @howdyjessie
Fixes issue #85. Please merge this as Chaise RecordEdit App is breaking because of the issue.

@chiragsanghvi thanks! something so obvious like that, you don't even need to make a PR out of it. Just commit/push yourself.

@robes Ok, will do it from next time. Thanks.


Exec-ing ninja when profitable?

As it currently stands, the _program function in src/ninja/__init__.py uses subprocess.call.
When used via the ninja main, it merely waits for the exit-code and then propagates it as its own.
This is visible in that when this wrapper is Ctrl-C-ed, we see the interruption traceback in the Python interpreter.
In addition, a long-running Python process is parked without much remaining purpose.
I was wondering whether it would be a good idea to exec ninja instead of wrap it, when the platform supports it.
What do you think?

Another option would be for the ninja executable to be directly deployed as a distribution script.
This should work as long as the ninja executable doesn't require to graft dependencies (i.e. as long as it's self contained).

And you’d need to duplicate the executable, I think, since the module can’t reliably locate the scripts install location.

If we can rely on importlib.metadata (or importlib_metadata), the module can locate the script (that would mean no more support for python <2.7 & python < 3.5).

With the move to scikit-build-core, the ninja executable is directly deployed as a distribution script so it might not be worth changing the subprocess call anymore.


New plugin

Hi i would like a new plugin to be created the same as this except just using collection log slots instead of combat achievements and i would be willing to pay for it

That one might be a bit more difficult. I've looked in to how to get a collection log count before and couldn't figure out how. I doubt it's actually impossible, just hard. Let me do some research and see if I can crack it before I commit to anything.

idk if this helps or not since i dont code but there is another plugin that manages to track the collection log slots and use them kinda like a currency how they would be used for chunk men https://runelite.net/plugin-hub/show/loglocked

Hi its been a while and i dont know if you will see this but just wanted to check in and see if it was possble or not?


fix(swagger): unique operation id

Repaired operationId of /orgs/{orgID}/invites/{inviteID}/resend , swagger operation names must be unique. influxdb client API generators fail when they are not.

[x] Well-formatted commit messages
[x] Rebased/mergeable
[x] Tests pass
[x] http/swagger.yml updated (if modified Go structs or API)
[x] Signed CLA (if not already signed)

@bednar FYI

Both #18250 or #18252 provide alternative solutions, so there is no need to have a 3rd one.


Test failure peek isn't ergonomic

work on adopting ensureNoDisposablesAreLeakedInTestSuite
run a suite that leaks something
😕 you loose context and end-up in the leak util. I wanna stay in the current file instead

(this is basically the same as https://github.com/microsoft/vscode/issues/135388)

should be a separate issue but it seems that those stack traces are off by one

We should prefer to stay in the current file if there's any frame in the stacktrace that is in the current file (and within those, prefer the location that's inside the range of the test)


fix(#330): ensure buildForm is called only once per form

What kind of change does this PR introduce? Bug fix
Fixed #330

Current coverage is 91.97% (diff: 100%)

Merging #342 into master will increase coverage by 0.02%

@@             master       #342   diff @@
==========================================
  Files            16         16
  Lines           746        748     +2
  Methods         141        141
  Messages          0          0
  Branches        196        197     +1
==========================================
+ Hits            686        688     +2
  Misses           60         60
  Partials          0          0

Powered by Codecov. Last update 52a5eb9...098aa33


Console enters an infinite loop if a class overrides getattribute

From<EMAIL_ADDRESS>on 2014-03-07T11:22:28Z
Executing the following script inside Spyder makes the console go into an infinite loop:
------------------------------------------------------
class Test(object):
def getattribute(self, name):
print('Test.getattribute(self=%s, name=%s)' % (self, name))
return object.getattribute(self, name)
t = Test()
------------------------------------------------------
This works in a stand alone console, on versions 2.7 32bits and 3.3 64bits. It does not work inside Spyder 2.3.0beta2 on either Python versions.
This is what shows up on the console:
Test.getattribute(self=<main.Test object at 0x0000000006D7C978>, name=class)
...
I believe that this is related to the variable inspector. The behaviour is a bit different in a default console vs an iPython console. In a default console, it goes on an infinite loop. On an iPython console, getattribute gets called ~20 times but then returns - maybe the recursion is broken politely.
Thanks
Original issue: http://code.google.com/p/spyderlib/issues/detail?id=1751

From ccordoba12 on 2014-03-10T11:01:57Z
Our console can't handle very long print loops, so maybe that's could be happening here.
Labels: Cat-Console

From<EMAIL_ADDRESS>on 2014-03-12T11:39:20Z
Try this:
------------------------------------------------------------------------------------
import inspect
max_spam = 15 # it should be the same 15 hits over&over, try more if you like
curr_spam = 0
element_name = ['Object:', 'Filename:', "Line:", "Function:", "Context:", "Index:"]
class Test(object):
def getattribute(self, name):
global curr_spam
curframe = inspect.currentframe()
calframe = inspect.getouterframes(curframe)
if max_spam > curr_spam:
print("\nCaller", str(curr_spam).rjust(2)+":")
for hit_no in range(1, len(calframe)):
#skipping first object (this method in this file)
print("    ", end = ' ')
for element_no in range(1, len(calframe[hit_no])-2):
# skipping object identifier, context, index
print(element_name[element_no],
calframe[hit_no][element_no], end=' ')
print()
curr_spam += 1
return object.getattribute(self, name)
t = Test()
------------------------------------------------------------------------------------
I'm attaching the data I get... Now it'd be awesome if you explained me what it means in terms of Spyder, Python and IPython ;)
Attachment: data.txt

From<EMAIL_ADDRESS>on 2014-03-12T15:09:34Z
Very nice way to show what is going on. I get exactly the same results on my computer.


pub update_locked_test times out on Windows

Many pub tests run for nearly 20 seconds on Windows, and one of them is now flaking with "Test timed out after 20 seconds".
The timeout should be increased for
pub/test/update/git/update_locked_test
and perhaps for the other tests that now take almost 20 seconds.

This issue has been moved to dart-lang/pub#632.


[stdlib] Adds vector macros 1/n

Description
Adds the first batch of macros for vector. These are the least controversial.
Test plan
Features tests.

Release notes
Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.
For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.

[ ] Protocol:
[ ] Nodes (Validators and Full nodes):
[ ] Indexer:
[ ] JSON-RPC:
[ ] GraphQL:
[ ] CLI:
[ ] Rust SDK:

@tzakian @tnowacki all your comments are addressed!


updated h5py requirements

Fixes #175

This counting as decreasing test coverage is ridiculous, so I'm going to approve and merge as it passes tests and obviously you have a CLA in place.


godotenvvault proxying commands to godotenv

As a first step, get godotenvvault proxying all exposed commands from godotenv.
This way, I should be able to follow the current README in this repo, and it just works.
For example, after this issue is completed I can do the following:

Install

go get github.com/dotenv-org/godotenvvault

Use

HELLO="World"

package main

import (
    "log"
    "os"

    "github.com/joho/godotenvvault"
)

func main() {
  err := godotenvvault.Load()
  if err != nil {
    log.Fatal("Error loading .env file")
  }

  hello := os.Getenv("HELLO")

  log.Println(hello) // world
}

This issue/ticket includes all the work to have the library versioned, installable, etc.
In later tickets we'll add the .env.vault decryption logic.

This should now work. For the latest version of Go (v1.20), you can only install packages from within a module. So you need to make a file called go.mod with the following contents:
module github.com/dotenv-org/demo

go 1.20

require github.com/dotenv-org/godotenvvault v0.1.0

require github.com/joho/godotenv v1.5.1 // indirect

Then you can put your code in a file called demo.go or something and do go build demo.go. The compiler will give you some instructions about how to download the necessary dependencies, and after doing those things, go build demo.go should produce a demo executable.
For reference, the commands the compiler will tell you to run will be something like this:
go mod download github.com/dotenv-org/godotenvvault
go get<EMAIL_ADDRESS>

Awesome. Marking done.


New testserver submodule missing in release 3.3.5

I wanted to try out the new testserver in my application, but
from pyads.testserver import AdsTestServer

...fails, saying the testserver module does not exist.
The installed package contains the following files:
.
├── __init__.py
├── ads.py
├── constants.py
├── errorcodes.py
├── filetimes.py
├── pyads_ex.py
├── structs.py
├── symbol.py
└── utils.py

So it is missing the new testserver directory.

I'm a dumbass, it hasn't been merged yet... I thought that with the version bump it had...
I wish Github had a commits tree view like Bitbucket does.

Okay, I'm a double-dumbass. It was merged properly, the directory is right here: https://github.com/stlehmann/pyads/tree/3.3.5/pyads/testserver
So then there is a pypi problem. You can see the latest package is missing the new files: https://pypi.org/project/pyads/#files

Tested this locally using pip install . and I can reproduce this problem that way.
I think I fixed it with adding the submodule in setup.py:
setup(
    # ....
    packages=["pyads", "pyads.testserver"],

But I've no clue whether this is conventional.

The internet suggests it too: https://stackoverflow.com/questions/26528178/right-way-to-set-python-package-with-sub-packages
@stlehmann what do you think?

@RobertoRoos Thanks for making the PR. I merged it right away.


The input is empty. You have to specify one or more '-injars' options

spring boot 2.x, jdk 1.8, maven 3.5, proguard 7.2.2 and  proguard-maven-plugin 2.6.0.
my configuration as follows:
<build>
		...
		<plugins>

			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-war-plugin</artifactId>
				<configuration>
					<outputDirectory>${war.output.dictionary}</outputDirectory>
				</configuration>
			</plugin>
			<plugin>
				<groupId>com.github.wvengen</groupId>
				<artifactId>proguard-maven-plugin</artifactId>
				<version>2.6.0</version>
				<executions>
					<execution>
						<phase>compile</phase>
						<goals>
							<goal>proguard</goal>
						</goals>
					</execution>
				</executions>
				<configuration>
					<proguardVersion>7.2.2</proguardVersion>
					<obfuscate>true</obfuscate>
					<proguardInclude>E:/proguard.cfg</proguardInclude>
					<libs>
						<lib>${java.home}/lib/rt.jar</lib>
					</libs>
					<injar>classes</injar>
					<outputDirectory>proguard</outputDirectory>
				</configuration>
				<dependencies>
					<dependency>
						<groupId>com.guardsquare</groupId>
						<artifactId>proguard-base</artifactId>
						<version>7.2.2</version>
						<scope>runtime</scope>
					</dependency>
				</dependencies>
			</plugin>
		</plugins>

		<resources>
			<resource>
				<directory>src/main/resources</directory>
                <filtering>true</filtering>
                <includes>
                	<include>**/*.properties</include>
                	<include>**/*.yml</include><!-- 也可以指定具体的文件 application-${env}.yml -->
                </includes>
			</resource>
			<resource>
				<directory>src/main/resources</directory>
				<filtering>false</filtering>
				<excludes>
					<exclude>**/*.properties</exclude>
					<exclude>**/*.yml</exclude>
				</excludes>
			</resource>
		</resources>

	</build>

my project is packaged as war.
I want to obfuscate the war or its classes, when I execute mvn clean package in idea, the error thrown:
[proguard] ProGuard, version 7.2.2
 [proguard] Unexpected error
 [proguard] java.io.IOException: The input is empty. You have to specify one or more '-injars' options.
 [proguard] 	at proguard.ConfigurationChecker.check(ConfigurationChecker.java:62) ~[proguard-base-7.2.2.jar:7.2.2]
 [proguard] 	at proguard.ProGuard.checkConfiguration(ProGuard.java:290) ~[proguard-base-7.2.2.jar:7.2.2]
 [proguard] 	at proguard.ProGuard.execute(ProGuard.java:99) ~[proguard-base-7.2.2.jar:7.2.2]
 [proguard] 	at proguard.ProGuard.main(ProGuard.java:623) [proguard-base-7.2.2.jar:7.2.2]

what's the problem? I have set injar as classes and phase of execution as compile.

Based on your code, your injar parameter points to a directory of class files, not a JAR file. Try changing that.

@lasselindqvist but from plugin.xml of proguard-maven-plugin, it says the injar is about
Specifies the input jar name (or wars, ears, zips) of the application to be processed. You may specify a classes directory e. g. 'classes'. This way plugin will processed the classes instead of jar. You would need to bind the execution to phase 'compile' or 'process-classes' in this case.

does I misunderstand it?

I added outputDirectory of proguard-maven-plugin in my case because I want to generate the obfuscated classes in the specified folder under target directory.
After I remove this parameter, this error gone, the obfuscated classes are in classes folder and the un-obfuscated classes are in classes_proguard_base folder.
Thanks @lasselindqvist !


Unable to install: Dyndolod

I have been unable to install this modpack due to several broken downloads.
A the moment the only one that keeps failing is Dyndolod 3D. I have noticed that the last few updates have included this one being regenerated. Is this a known issue?

Greetings Mates,
I´ve got the same issue here.
I did all the Pre Installation Steps...
Don´t know how to fix this..
Can anyone Assist?
Ben

Greetings Mates,
I´ve got the same issue here.
I did all the Pre Installation Steps...
Don´t know how to fix this..
Can anyone Assist?
Ben

I have the same issue as you Ben. Did you ever find a fix for it?

@MSaf19
No not yet... I´ll try to find some support via Discord now - I´ll Update.

@MSaf19
No not yet... I´ll try to find some support via Discord now - I´ll Update.

Not sure if you solved it yet, but I solved mine my designating the download folder, as well as the installation folder (2 separated folders).

i have the same thing can some one help me :P