New Rust attribute to support embedding debugger visualizers

This RFC adds support for a new Rust attribute that will embed a debugger visualizer into a PDB/ELF.
Internals thread
Rendered

I can totally see people wanting to derive natvis files using (proc) macros.

I'm a bit wary of adding super platform-specific stuff to Cargo.toml (and this is coming from a windows user...)
I think it would be useful to look at prior art on other platforms, to see if there's some common denominator there.
It looks like natvis is also supported with GDB in vscode, although I don't know if there's any way to embed natvis files into the debug info in that case.

Idk, quite often it’s one way for windows and one way for unix systems.
What matters is being able to have a better debug experience. If that means
we have to do two things to cover all platforms that’s not too arduous.
On Sat, 6 Nov 2021 at 02:11, Diggory Blake @.***> wrote:

I'm a bit wary of adding super platform-specific stuff to Cargo.toml (and
this is coming from a windows user...)
I think it would be useful to look at prior art on other platforms, to see
if there's some common denominator there.
It looks like natvis is also supported with GDB in vscode, although I
don't know if there's any way to embed natvis files into the debug info in
that case.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/rust-lang/rfcs/pull/3191#issuecomment-962371833, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAGEJCFCMCBXSK6HOSL3HZDUKSFE5ANCNFSM5HJ5SMPQ
.

Idk, quite often it’s one way for windows and one way for unix systems. What matters is being able to have a better debug experience. If that means we have to do two things to cover all platforms that’s not too arduous.
…
On Sat, 6 Nov 2021 at 02:11, Diggory Blake @.***> wrote: I'm a bit wary of adding super platform-specific stuff to Cargo.toml (and this is coming from a windows user...) I think it would be useful to look at prior art on other platforms, to see if there's some common denominator there. It looks like natvis is also supported with GDB in vscode, although I don't know if there's any way to embed natvis files into the debug info in that case. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3191 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGEJCFCMCBXSK6HOSL3HZDUKSFE5ANCNFSM5HJ5SMPQ .

I agree with this as well. Other platforms do not have a way to embed this kind of debug information automatically. VSCode uses MIEngine which only has a subset of the Natvis framework ported to it. This applies natvis to types within the debugger but the natvis is not embedded into the debug info. This means having to find the natvis for a specific version of a crate while debugging to get the correct behavior. This would put onus on the Rust developer who is consuming said crate to find the correct version of natvis for the version of the crate they are using. I believe having this solution for Windows is a great first step since these platforms are extremely different in how they support debugger visualizations.

So, I think it is a great idea to support natvis in an ergonomic way. Even though natvis is specific to certain platforms/debuggers and so is intrinsically not cross-platform, I think it is important for this RFC to consider other platforms so that we are sure that it is possible for crates to support multiple formats for different platforms, ideally using a single mechanism or very similar mechanisms (not the same visualisation format, but the same mechanism for Cargo to discover the visualisations and notify them to rustc, maybe).
On a lower level, how much does rustc need to do? Is it literally just passing filenames to the linker? Or does it need to do some symbol (de-)mangling? Integration with debuginfo support? How exactly will natvis files be saved in the metadata?
Finally, I much prefer the auto-discovery alternative. It seems much more inline with Cargo's 'convention over configuration' philosophy - we do not list source files, for example.

I think it is important for this RFC to consider other platforms so that we are sure that it is possible for crates to support multiple formats for different platforms

Hi, @nrc. I'm one of the co-authors of this RFC. Can you be more specific about this request? The other visualization formats supported by other debuggers are quite different in how they are implemented, packaged, and used. This RFC doesn't prevent implementing support for those debugger visualizations. In fact, it proposes a filesystem schema specifically for avoiding collisions, by using .natvis file extension that are based on the root module filename for a given crate. This works for the primary crate, unit test binary, integration test binaries, etc.
If there is something more to do here, we are certainly open to that. To me, it appears that the main requirement is not to interfere with implementing support for other debug formats, and I think this proposal meets that requirement.
Because different debugger visualizers are so different in how they work, I don't see much of an opportunity to "use a single mechanism or very similar mechanism". Again, I'm open to that, but I just don't see any reusability here.

Is it literally just passing filenames to the linker?

Yes, that's pretty much all there is to it. The only complexity here lies in the requirement to propagate these filenames across the dependency graph.  That is, when you invoke the linker, you need to pass the linker options for all crates in the dependency graph, not just the current crate.
In my original prototype implementation, I did this purely in Cargo, because Cargo knows the entire dependency graph and so it could walk it and find all of the NatVis files. However, when I discussed this design with some folks on the Cargo channel on Zulip, they felt that this was not a good approach because it would not work correctly for build system that were directly driving rustc, rather than using Cargo. They suggested the approach of bundling the NatVis files into metadata fields, so that the final rustc invocation could just inspect metadata, pull out the NatVis files, and pass them to the linker. That's what we propose to do in this RFC.
We also favor auto-discovery. However, like many options in Cargo, I think the developer should still be able to override auto-discovery.

Hi, @nrc. I'm one of the co-authors of this RFC. Can you be more specific about this request? The other visualization formats supported by other debuggers are quite different in how they are implemented, packaged, and used. This RFC doesn't prevent implementing support for those debugger visualizations. In fact, it proposes a filesystem schema specifically for avoiding collisions, by using .natvis file extension that are based on the root module filename for a given crate. This works for the primary crate, unit test binary, integration test binaries, etc.

Hi, so I'm personally not familiar with how any debuggers support visualisations, so it would be useful for me to have a short survey of how that works so that I can evaluate the similarities/differences myself. I assume that is also the case for others who want to evaluate this RFC too. If they are so different, then that would be enough.
Since I don't know the other mechanisms, I'm making a complete strawman here, but suppose that GDB takes a python file as a visualisation input, then that suggests that the Cargo manifest syntax should be something like
[debug-visualisations]
natvis = [...]
gdb-py = [...]

(again, totally making up the keywords as I go along). In that case, I would suggest this RFC propose this syntax (without the gdb case, leaving that for future work), rather than having natvis as a top-level key in the TOML file.

If there is something more to do here, we are certainly open to that. To me, it appears that the main requirement is not to interfere with implementing support for other debug formats, and I think this proposal meets that requirement.

I would say that not only must we not interfere, we should leave scope for supporting other formats in a clean way.

Because different debugger visualizers are so different in how they work, I don't see much of an opportunity to "use a single mechanism or very similar mechanism". Again, I'm open to that, but I just don't see any reusability here.

I'm not sure if it makes sense, but I hope the hypothetical example above makes clear the extent of reusability I'm imagining, it's not much of an iteration over the current state of the RFC.

Yes, that's pretty much all there is to it. The only complexity here lies in the requirement to propagate these filenames across the dependency graph. That is, when you invoke the linker, you need to pass the linker options for all crates in the dependency graph, not just the current crate.

That sounds fine. I think it would be nice to specify this in the RFC in a bit of detail so that we can evaluate the scope of changes being proposed

We also favor auto-discovery. However, like many options in Cargo, I think the developer should still be able to override auto-discovery.

Do you have an example of something you are following with this design? My intuition is that natvis files should be treated like source files, for which there is no mechanism to list them (not sure about overriding auto-discovery for source files, there is a mechanism in Rust source, but I'm not aware of a mechanism in Cargo).
Are there typically many natvis files in a project, or usually just one (or one per target)? My preference would be not to keep metadata in the src directory and thus have a separate directory for such files, but I guess I would feel less strongly if it were just one file. I'm not sure of any precedent for keeping metadata in src vs in a separate directory, but it would be good to know about any if there is any.

In that case, I would suggest this RFC propose this syntax (without the gdb case, leaving that for future work), rather than having natvis as a top-level key in the TOML file.

This RFC initially suggested having the natvis key under the [package] section of the TOML file, which was one of the unresolved questions I had. Creating a new section for debugger visualizations sounds fair to me and would probably be more inclusive of future work to support other formats.

That sounds fine. I think it would be nice to specify this in the RFC in a bit of detail so that we can evaluate the scope of changes being proposed

I've explained at a fairly high level in the reference level explanation the scope of changes this RFC introduces in terms of new toml syntax and rustc flags as well as needing to make changes to the crate metadata. I will go into more detail in this section to show what specific changes need to be made and what fields need to be added to a CrateMetadata object.

Are there typically many natvis files in a project, or usually just one (or one per target)?

There is typically one Natvis file per library i.e. one file for a static lib and a separate for a shared DLL. This helps to break down where Natvis visualizations are defined for types. In the case of a Rust crate, you could imagine each crate having a single Natvis file that defines the debugger visualizations for types within that crate.

I'm not sure of any precedent for keeping metadata in src vs in a separate directory, but it would be good to know about any if there is any.

There is no reason as to why this would need to be kept in the src directory. Natvis files can be stored anywhere but they must be published as part of the crate. This way other crates that consume said crate would be able to pass the .natvis files into the linker.

@nrc I like your suggested TOML structure, of:
[debug-visualisations]
natvis = [...]
gdb-py = [...]

About the question of where to put the files, and how many files there are. For typical C++ libraries, there is often just a single NatVis file, but it certainly does happen that a library is large enough to warrant breaking NatVis into more than one file, for the benefit of developers organizing things. So my expectation is that your typical crate will have just one, or a small number, of NatVis files.
When I wrote the first draft of this spec, I thought just having a ${crate_root}/dbg/natvis/*.natvis rule would be sufficient. However, if we want to allow a crate to have different sets of NatVis files for each unit that it produces (primary library, binaries, integration test binaries, etc.), then I think placing the NatVis files into the src tree makes sense. It also makes sense just for keeping the NatVis visualizations close to the module trees that they describe.
Or, we could simply say that the scenario of having different NatVis for different units is overkill, and that the same set of NatVis files should be used for all units in a given crate. That way, we just put them into ${crate_root}/dbg/natvis/*.natvis and be done with it. I like that approach, because it is simple and easy to remember and easy to implement, and because I can't actually think of a scenario where I would want different NatVis in different units.

Hi, so I'm personally not familiar with how any debuggers support visualisations, so it would be useful for me to have a short survey of how that works so that I can evaluate the similarities/differences myself. I assume that is also the case for others who want to evaluate this RFC too. If they are so different, then that would be enough.

I think that's a reasonable request, mainly because it will illustrate just how different they are. For GDB, it's a set of Python scripts (I think? might be wrong). For others (WinDbg), it's a native, compiled DLL that gets loaded into a debugger. WinDbg also supports JavaScript-based extension, too. Some of these can be linked into target binaries, or into their debug symbols (PDBs), but others have no convenient way to package the debugger extensions in a way that is discoverable by the debugger -- you just have to know how to find it.

However, if we want to allow a crate to have different sets of NatVis files for each unit that it produces (primary library, binaries, integration test binaries, etc.), ... and because I can't actually think of a scenario where I would want different NatVis in different units.

It sounds like it is worth digging in to this requirement. Is there an equivalent thing for C++/VS projects?
I know some projects have extensive support for testing infrastructure, and that might warrant separate debug metadata. However, one could also use a separate crate in workspace for this. I can't imagine want separate visualisations for most tests?
I can imagine a crate has both a library and binary targets and the latter has some data types (with visualisations) which are not in the library. Could this be handled with a single natvis file? Relatedly, if a crate has datatypes which are present or not depending on cfgs (e.g., Tokio), how would this be handled with natvis and this proposal?

Is there an equivalent thing for C++/VS projects?

No, because in VS a single project produces a single output (EXE, DLL, LIB). There's no concept of "units".
On balance, I think we should start with the simpler model: For a given crate, the user defines a set of NatVis files. Those NatVis files are compiled into the primary crate unit (exe or rlib), and into the unit test crate unit. For benchmarks, integration tests, and bins, we don't include the NatVis files, because those compilations already reference the primary crate, which already contains the NatVis files.
I think this is best because:

It's simple for the user to reason about.
It still allows sophisticated developers to include definitions that are specific to their testing code. Yes, those definitions will be placed in the PDB file for the primary crate, which does mean that there's some test-only information in a non-test PDB. But I think that's OK -- it doesn't affect the primary crate itself, only its debugging info. It will be harmless, and will be minuscule in size, compared to the PDB itself.
It lets us use the simpler filesystem layout, which will be easy for developers to learn and use.

Relatedly, if a crate has datatypes which are present or not depending on cfgs (e.g., Tokio), how would this be handled with natvis and this proposal?

I think, for this V1, the NatVis file would contain definitions for the union of all features. As long as features are additive (and they are supposed to be), that should work fine. It means that some definitions may be ignored or useless when their corresponding features are not activated. I think that's OK.
When I originally drafted this RFC, I included two means to package NatVis into crates. The first is the simple "whole file" method, which is what is described here. The second was to allow users to include NatVis fragments directly into Rust source code, such as:
natvis!(r#"
    <Type Name="my_crate::MyFancyType">
      <DisplayString> ... </DisplayString>
      <Expand> ... </Expand>
      ...
    </Type>
"#);

Or, to provide it as an attribute on a type, so that the compiler could generate exactly the right type string, for matching descriptions to mangled types. This would free the developer from needing to write the <Type Name="..."> element.  (Also, NatVis supports generic types, to an extent. Writing the <Type> parameters for generics is a little more tedious, so handling that for the developer would be a win.) Example:
#[natvis(xml = r#"
  <DisplayString> ... </DisplayString>
  <Expand> ... </Expand>
")]
pub struct MyFancyType { ... }

Rustc would then find all of these and assemble them into a single XML document, and bundle that into the PDB.
If we allow users to embed NatVis fragments directly into source code, then we can allow #[cfg] to control them, or to construct them using proc macros or macro_rules. This has lots of advantages, but we wanted to start with the simplest V1 that we could, because we felt that it would take a much longer time to come to consensus on what the V2 would look like.
So our plan is, stabilize the V1 support, and then to submit PRs to various crates, so that we can improve the debug experience of those crates. Then, when the value of this becomes apparent, we (with the community) can concurrently be working on spec'ing V2, with direct source embedding.

I posted one comment, but otherwise this looks good to me.
I think it's completely reasonable to do this in two phases, with the first requiring separate files and the second allowing inline attributes that get extracted.

Hi @joshtriplett thanks for taking the time to review this RFC. The goal of this RFC is to see stabilization of this feature at some point in time. As such, I was wondering if the flags I have suggested in the RFC are appropriate. There's a couple of ways this could be achieved.

As described in the RFC, keep the -Z natvis={comma-separated list} flag at which point it would eventually get turned into a -C flag for stabilization.
Create a -C natvis={comma-separated list} flag with a -Z enable-natvis flag as well which would guard against using the -C natvis flag in stable Rust.

Thoughts?

@ridwanabdillahi I personally would propose starting with -C natvis, and gating it with the existing -Z unstable-options; that would make a future stabilization easier. But there may be a reason I'm not aware of for why that isn't a good idea.

I personally would propose starting with -C natvis, and gating it with the existing -Z unstable-options;

@joshtriplett that sounds good to me. I'll update the RFC to reflect as such.

I believe I've responded to all of the comments, is there anything else that needs to be addressed in this RFC?

In response to https://github.com/rust-lang/rfcs/pull/3191#issuecomment-974442018 (and just general pondering). I wonder if it is worth thinking a bit about integration with Cargo features at least (and perhaps other sources of cfg input such as the target platform). I don't want to get too much into discussion of what v2 would look like, but I have a strong preference to avoid inlining the natvis info and keeping it in separate files. Given this possibility, I want to make sure there is scope in the proposal to extend it to including different natvis depending on features, etc. We don't have to go into too much detail, obviously.
Could we imagine extending the Cargo manifest entry to include features, etc. in the same way as dependencies? I think it should be easy enough some how, and perhaps this is a good justification for keeping the explicit filename list rather than auto-discovery?

One more thought, on toolchains which don't support adding natvis into the PDB during linking, should Cargo or rustc emit all the specified natvis files into a location in the target directory so that they can be easily accessed by the user?

For this RFC I do not believe there is a compelling reason to have the Natvis files stored in the target directory. If things shall change in the future, this is something that can always be easily updated but for now I think leaving the Natvis files where they live is the proper solution.

I think this is targeting a really important problem and I would like to see it move forward. In my conversations with people using Rust, debugger visualizations (or the lack thereof) is a common topic.
While I would not want it to block this RFC, I do think it's worth thinking about whether we can support debugger visualizations for Rust types in a more portable way. I've been pondering the idea of a general "type reflection" mechanism; the idea would be that there is a const fn (hence, evaluatable at compilation time) that understands how to "interpret" a type and supply pointers to its members (which can then be further unrolled if the caller wishes). We could then use that to implement default Debug interfaces, serializes, and debugger visualizations and so forth.

So, re-reviewing this, I think it'd be reasonable to go ahead and add this option to explicitly list natvis files.
As far as I can tell, it also seems like it'd be really straightforward to add autodiscovery, and the biggest blocker there is agreeing on the path for that autodiscovery. I think it'd be perfectly fine to discover files in one or more of natvis/**/.natvis or dbg/natvis/**/*.natvis or src/**/*.natvis; can we just agree to search all of those?
I do still agree that we should defer in-Rust-attribute natvis support to a later iteration of this feature.

As far as I can tell, it also seems like it'd be really straightforward to add autodiscovery, and the biggest blocker there is agreeing on the path for that autodiscovery.

Currently the RFC lists the path for Cargo to auto-discover Natvis files as dbgvis/natvis. I will update the RFC to move this from the Future Goals section and this will be part of the first iteration of this feature. As stated in the RFC we specifically chose to avoid dbg or debug as the path since many tools (outside of Cargo and Rust) automatically create such a folder in the project root when building a project. This way there is no confusion for C or C++ developers moving from other build tools to Rust.

I think it'd be reasonable to go ahead and add this option to explicitly list natvis files.

I will update the RFC to show that with auto-discovery being part of the first iteration of this feature, the Cargo toml syntax to explicitly list .natvis files will override the auto-dicovery of natvis files in the dbgvis/natvis directory.

I had a few questions. I'm not entirely following why this specifies that the natvis files should be embedded in the crate metadata. Is there a particular reason that needs to be done? I think that adds a lot of complexity, so I think it needs a strong justification.
As an alternative, I would expect that cargo can just pass the appropriate -C natvis flags to rustc when making a binary or DLL. Cargo knows the entire dependency graph, and knows exactly which natvis files exist for the final linking.
If -C natvis then becomes a simple pass-through of /NATVIS to the linker, then it might be good (as mentioned in https://github.com/rust-lang/rfcs/pull/3191#pullrequestreview-848060513) to specify why a new -C flag is needed as opposed to -C link-arg. Personally, I'd be hesitant to use link-arg, since we try to avoid having cargo know too much about these things. For example, cargo doesn't know which linker is used or linker-flavors.
Another issue is how does -C natvis behave if not using link.exe. Will it fail? Be ignored? I think if cargo is unconditionally setting -C natvis, but the linker is set to something else, that could be a problem. I'm not a big fan of flags that get ignored, but in this case I'm not sure how to work around that.
Is there a specific reason to copy the natvis files to the target directory? Are they needed to be in one place for any particular reason?  If cargo just passes all the paths to the final rustc run, then I would think it should be fine to leave the natvis files in their original locations. rustc has limited abilities to start splatting files and directories in the output path. It would need to deal with concurrent rustc invocations running at the same time which could potentially be writing to the same files at the same time. I think it would be good to avoid that if possible.

I'm not entirely following why this specifies that the natvis files should be embedded in the crate metadata. Is there a particular reason that needs to be done?

There are several reasons for this. When I originally brought up this RFC in Zulip (about a year ago), I started with exactly this position, that Cargo could handle finding the files, and simply pass them to rustc using -Clink-args=/natvis:... when appropriate. There was significant resistance to that from Cargo and Rustc developers; they raised several points.
First, Cargo is not the only build system used with Rust. There are others, such as Blaze/Bazel, and I believe either CMake or Meson has support for directly driving Rust. That might be a minor issue to the wider community, but for the people that are working on those systems, it is beneficial to pass this information through crate metadata. That way, the information enters the dependency graph only at the leaf nodes, and the code building the dependency graph doesn't need to know how or why it flows through the dependency graph.
Second, there's a working group that is trying to figure out how to handle binary crate packages. That is, compiling crates to rlibs, and then passing around rlibs. My own team needs this, and our need for it is not subject for debate, so we are actively helping to drive that effort. If we have to pass around "rlibs + whatever other unrelated files", then this gets much harder.
We're also looking at supporting systems like GDB's debugger scripts, WinDbg's debugger scripts, etc. Passing around yet more "related files" gets real ugly real fast. Packaging things in the rlib is simple and reliable; it seems very "right", and this is exactly the direction that the Rustc devs and Cargo devs pushed me toward, when I engaged with them last year.

Another issue is how does -C natvis behave if not using link.exe.

I think it is perfectly fine for a feature of MSVC's toolchain to be silently ignored when using other toolchains.  I imagine that developers would be  unhappy if the addition of this feature broke builds for targeting -gnu toolchain. Between "my debug visualization doesn't work" and "my product can't build", I think the choice is always going to be the latter.

Is there a specific reason to copy the natvis files to the target directory?

We want to avoid placing paths to files within files. In my experience working on build systems, putting paths inside files always creates problems. Packaging the files themselves is quite simple and reliable. And the files themselves are quite small -- we're talking about hand-written XML files, which are orders of magnitude smaller than the RLIBs themselves.
The only time we need to write files is to be able to pass data to link.exe. So we just want to write out the files before invoking the linker, and not otherwise.

I would think it should be fine to leave the natvis files in their original locations.

I disagree. In my experience maintaining build systems at scale, this always causes problems. Packaging the file contents is efficient and reliable.

@ehuss Using -C link-arg would be a reversal of the conclusion from a previous Cargo meeting on the same topic, in which we said we shouldn't be directly implementing this, we should be having rustc do it and just pass through to that.

I see that the RFC has been updated for autodiscovery. I think we shouldn't just enable this in dbgvis/natvis ; we could discover *.natvis files anywhere under dbgvis. (We could also discover them under src.)

I think we shouldn't just enable this in dbgvis/natvis ; we could discover *.natvis files anywhere under dbgvis. (We could also discover them under src.)

Is there a reason we would like to turn on auto-discovery for multiple search paths? Would there be an issue with having a single directory where all natvis files for a given crate live in order to be automatically discovered? The reason for this specific directory is so that it can also be used to potentially support packaging GDB debugger scripts as well in the future. For example, under dbgvis/gdb-scripts/. This would allow for a simple mechanism that could be extensible for adding debugger visualizations for other debugger types.

@ridwanabdillahi If you only have a single natvis file and a single gdb script, you might just put them at the top level of dbgvis rather than having a one file directory.
I'm not attached to finding them in src though.

There are several reasons for this.

Sounds reasonable. I might be good to capture the reasons for embedding the files in the RFC.

silently ignored when using other toolchains

Silently ignoring is probably reasonable, though I'm not sure if there is a precedent one way or the other.  Might be good to mention that behavior.  My comment was more about using something like -C linker to use a different linker than it was about using a different toolchain like -gnu.
I see now that lld supports natvis, so that is probably one of the major possible alternatives that shouldn't be a problem.

previous Cargo meeting on the same topic

Sorry, I don't recall discussing this, nor see anything written down.  I think having rustc be responsible is right, I was just trying to suss out why the collection would be done along the way instead of once at the end.

<crate_name>-<stable_crate_id>-<file_name>-<count>.natvis

Thanks for adding some detail with how the files will be written. However, I'm not really clear what <count> is.  Is that something to address concurrent writes?  How would that work?
I'm not super confident that StableCrateId is unique across all rustc invocations, at least with cargo.  I think it is, however that is a relatively recent change (https://github.com/rust-lang/cargo/pull/9418).  I don't think there are any particular guarantees that it is globally unique across unrelated compilations.

Another possibly too low-level question: Will the standard library use this mechanism to embed the natvis files, or will it use the existing sysroot mechanism?

@ehuss

However, I'm not really clear what  is. Is that something to address concurrent writes? How would that work?

<count> is meant as a way to ensure file names are unique. It's just an arbitrary number that is appended to the filename to ensure multiple .natvis files with the same name are generated as unique, so Natvis representations are not lost in translation. I will update the RFC to directly specify this.

I'm not super confident that StableCrateId is unique across all rustc invocations.

That's interesting, I was not aware that StableCrateId was not unique across rustc invocations. Would there be a better naming mechanism to use here to ensure uniqueness is guaranteed?

Will the standard library use this mechanism to embed the natvis files, or will it use the existing sysroot mechanism?

For the time being the standard library would still use the existing mechanism for embedding Natvis. Once this feature is stabilized, I don't see why we wouldn't be able to update this as well now that the contents of Natvis files are being embedded directly into the metadata.

If you only have a single natvis file and a single gdb script, you might just put them at the top level of dbgvis rather than having a one file directory.

I totally understand the reasoning for this, but I believe enforcing the directory structure would still be fine here and leave clarity for where .natvis files live in general. Having an extra subdirectory under dbgvis specifically for Natvis files should be reasonable for using this feature.
I'll update the RFC to add support for Cargo to automatically discover sub-directories within the dbgvis/natvis/ directory as well for all .natvis files so that dbgvis/natvis/**/*.natvis is supported.

Would there be a better naming mechanism to use here to ensure uniqueness is guaranteed?

Unfortunately I don't have an easy answer here. This will be tricky to do well.  For example, it will need to consider if there are a dozen crates being built simultaneously, all with the same dep graph that has a dozen or more natvis files. Will it create duplicate copies of all of them? Will it clean them up somehow?  I don't think it should rely on file locking since that is not always available or reliable.

Perhaps we should calculate a hash of the file contents and name the file {hash}.natvis (or {file name}-{hash}.natvis if we want to keep the original file name in the path)? That would also provide an easy way to deduplicate natvis files across the crate graph.

Perhaps we should calculate a hash of the file contents and name the file {hash}.natvis (or {file name}-{hash}.natvis if we want to keep the original file name in the path)?

Both of these options sound good to me. I will go ahead and include the original file name for now and update the RFC to state as such unless there's a reason not to.

That would also provide an easy way to deduplicate natvis files across the crate graph.

It's not across the crate graph that I'm concerned with.  It's if you have rustc foo.rs and rustc foo.rs --test running concurrently. They both will need to write out the natvis files for all dependencies simultaneously. How would they avoid stepping on each other's toes?
I don't want to belabor this topic, as I think it is touching on things that are too low-level for an RFC. However, since this RFC is specifically going for an approach that involves copying all files at the end, that introduces problems that can be difficult to work around.

@ehuss and @wesleywiser , thanks for the feedback. It's appreciated.
I like the <hash> more than <count>, because it is deterministic and does not depend on file ordering.  It would be nice to preserve some sense of the origin of the file, since when I'm debugging NatVis extensions I often need to look at the list of NatVis streams that are embedded in PDBs.  It would be nice to see libfoo-<hash>.natvis, for example. That way, it's obvious that it came from a crate called libfoo, and I can ignore the content hash.  That would give me a reasonable experience when debugging this.

I do still agree that we should defer in-Rust-attribute natvis support to a later iteration of this feature.

I just saw this RFC and I have to say I strongly disagree with exposing this to Cargo in any form.
Since the information goes to rustc anyway, why should it go the long way around?
Anything that could be passed with a -C option would just as well work through a system like this:
#[debug(natvis = include_str!("Foo.natvis"))]
pub struct Foo {...}

(Ideally some validation/rewrite would be performed to ensure 1:1 matching with the type being annotated)

I just saw this RFC and I have to say I strongly disagree with exposing this to Cargo in any form. Since the information goes to rustc anyway, why should it go the long way around?

Just to be clear, you strongly disagree with exposing the inline natvis fragments to Cargo? This syntax would only be used by rustc to generate .natvis files that would be embedded in the PDB along with the Natvis files that were passed through Cargo via the -C option. I don't believe Cargo would be interpreting the in-Rust-attribute at all.

@eddyb So when I first brought this feature proposal up with the Cargo devs, mainly Josh Tripplet, the guidance I received was that doing this in Cargo was the right approach. I also don't see why this rises to the level of "strong resistance", without any reason given about why that is.
We intentionally chose the least invasive approach for V1, which is having Cargo + Rustc figure out which natvis files to package, and then doing so.  The RFC already mentions that a future version could provide NatVis descriptions either directly in Rust source files, or by having attributes similar to what you mention point at files.  I'm fine with that, but that seems like a very invasive "V1" for a feature, and would slow down the approval and stabilization process for this.
If you feel this strongly that teaching Cargo about this is not an acceptable direction, are you also willing to help push support for the approval and stabilization of the in-source-code approach?

Just to be clear, you strongly disagree with exposing the inline natvis fragments to Cargo?

I mean the separate files. The only reason Cargo seems to be involved is to pass -C flags to rustc, but if the information is in the Rust source code, then rustc can do this entirely internally.

We intentionally chose the least invasive approach for V1, which is having Cargo + Rustc figure out which natvis files to package, and then doing so.

To me, touching Cargo at all seems strictly more invasive. rustc could internally attach Natvis to PDBs without Cargo even knowing it's happening. There may be some subtler reason for putting Cargo in between the user and rustc for this, but from what I've seen so far I don't see it mentioned.

I'm not even sure I even see a strong reason for supporting non-inline Natvis (other than getting XML highlighting and whatnot from having a separate file).
As it stands, the need to have a file to pass to link.exe is no more interesting than the fact that an .a archive is made out of .o that one might have to have as files on disk in order to create the .a.
Even with a -C flag, it's safer to use the standard "make a temporary file with the right contents and pass the path to that, to the tool (link.exe in this case)" approach, after reading the contents first.

One of the earlier comments (https://github.com/rust-lang/rfcs/pull/3191#issuecomment-974442018) contained this:

Or, to provide it as an attribute on a type, so that the compiler could generate exactly the right type string, for matching descriptions to mangled types.

I think this probably needs to be a mandatory feature, otherwise we will end up with desyncs, hardcoding details of mangling, and other unnecessary points of friction. The poor integration with debuginfo just makes this problem worse, at least rustc can try to encode enough information in both places to make sure debuggers can match them up reliably.

Besides all that, long-term this only makes sense as an ad-hoc way to extend debuginfo (i.e. on top of what rustc+LLVM already generate). The information encoded clearly belongs in debuginfo, it just happens to be a poorly-integrated format that has to have all this unnecessary infrastructure to work in the first place.
At least if it's well- integrated with rustc, platforms using standard debuginfo could still get some value out of it, by rustc transforming the Natvis or embedding it in some form that e.g. gdb Python scripts could use.
And if we eventually end up with a good replacement, a transition between two inline formats (deprecating the non-standard one) would work best.

On the RFC itself: is there any reason to make a specific "insert natvis into rlib" feature instead of a generic "insert any blob with filename X into rlib"? I saw this touched on, but never evaluated as an alternative.
I would generally like the alternatives section fleshed out. Right now it's purely rationale without any alternatives at all. You do mention them in comments or the motivation, but listing them explicitly would be good

On the RFC itself: is there any reason to make a specific "insert natvis into rlib" feature instead of a generic "insert any blob with filename X into rlib"? I saw this touched on, but never evaluated as an alternative.

I think it is hard to design such a generic feature without many specific use cases and those don't really exist at the moment. I think this RFC presents a nice mid-point, which is a specific feature with an obvious way to turn it into a more generic feature in the future if those use cases arise.

Since the information goes to rustc anyway, why should it go the long way around?
Anything that could be passed with a -C option would just as well work through a system like this: ...

@eddyb I think you're assuming an inline format? There's obviously trade-offs between separate files and inline, but to me inline here seems bad - the data is likely to be large and distracting to have on many type declarations, plus if there is support for other debuggers then there are potentially many visualisations for every type. I think that would make source code unreadable. Requiring include incantations as a work around seems like just adding more noisey boilerplate to declarations.
I understand visualisations as artifacts that are threaded through the build with little interpretation from the compiler. So it seems natural that these artifacts are managed primarily by the build system rather than the compiler.

To me, touching Cargo at all seems strictly more invasive. rustc could internally attach Natvis to PDBs without Cargo even knowing it's happening. There may be some subtler reason for putting Cargo in between the user and rustc for this, but from what I've seen so far I don't see it mentioned.

AIUI, there are visualisations from dependent crates which must be included in the final binary, so if this were to be managed by rustc then we'd need to extend the rlib format to include the visualisations and rustc would need to do some juggling of that natvis to pass to the linker. The Cargo approach means no changes to rlibs and (I think) less for rustc to do to get the metadata to the linker. Simply not changing rlib handling makes this approach more attractive to me!

I'm not even sure I even see a strong reason for supporting non-inline Natvis (other than getting XML highlighting and whatnot from having a separate file).

As I wrote above, the main downside is that it fills the source code with distracting detail that very few folk will want to read,

As it stands, the need to have a file to pass to link.exe is no more interesting than the fact that an .a archive is made out of .o that one might have to have as files on disk in order to create the .a.

Even with a -C flag, it's safer to use the standard "make a temporary file with the right contents and pass the path to that, to the tool (link.exe in this case)" approach, after reading the contents first.

I'm not sure what you're proposing here? Are you saying the programmer should manually do this? Because messing around with artifact files, compiler flags, and calling the linker seems exactly the sort of thing the build system should be doing and I would be furious if I had to wrap cargo build in a script just to do this :-)

Besides all that, long-term this only makes sense as an ad-hoc way to extend debuginfo (i.e. on top of what rustc+LLVM already generate). The information encoded clearly belongs in debuginfo, it just happens to be a poorly-integrated format that has to have all this unnecessary infrastructure to work in the first place.

AFAIK no debugger includes visualisations in the debuginfo, and none have them have signalled any desire to do so. Although the mechanism for specifying visualisations varies between debuggers, they are all separate from debuginfo. Importantly debuginfo is usually compiler-generated and visualisations are usually hand-written (with tooling), so they belong to different categories of metadata.

And if we eventually end up with a good replacement, a transition between two inline formats (deprecating the non-standard one) would work best.

This sounds wonderful, but I think it is so far in the future that we won't mind having such a thing as well as support for what works today.

I think this RFC presents a nice mid-point, which is a specific feature with an obvious way to turn it into a more generic feature in the future if those use cases arise.

I just feel like it's the asm situation all over again. We'll add it unstably, and never stabilize it because then we'll have to support some debugging tool specific logic forever. Even if we add a more generic scheme, we'll have to keep supporting this one.
I mean... we already have a way to bundle arbitrary object files via build scripts and even change linker flags. Yet there is no mention of the explored alternatives. I think this is my problem with this RFC. It shows this as the one true path without creatively exploring the future extensions and alternatives. Yes, it will solve the authors' immediate problem, but it encodes the name of a debugging visualization format into our forever-to-be-supported Cargo.toml manifest keys and rustc argument list.
I'm not saying that I know this is not the right solution, but that the alternative designs are not listed and thus there's only a "do this or don't" decision to be made and that's bad.

Even with a -C flag, it's safer to use the standard "make a temporary file with the right contents and pass the path to that, to the tool (link.exe in this case)" approach, after reading the contents first.

I'm not sure what you're proposing here? Are you saying the programmer should manually do this?

No, I'm referring to what most compilers (including rustc) do internally to pass data to an external tool that requires the data to be in files. It's the only safe way to do it (AFAIK), that can't cause clobbering during parallel builds and whatnot. You can see some examples around in the codebase: https://github.com/rust-lang/rust/search?q=MaybeTempDir
rustc passing the file paths directly to link.exe seems irresponsible/unnecessary/non-standard, to me.

AIUI, there are visualisations from dependent crates which must be included in the final binary, so if this were to be managed by rustc then we'd need to extend the rlib format to include the visualisations and rustc would need to do some juggling of that natvis to pass to the linker.

That's good though, because it means the standard library can provide Natvis without cargo -Z build-std.
Though it would be nice if the debuginfo format was more robust and didn't need this workaround.

AFAIK no debugger includes visualisations in the debuginfo, and none have them have signalled any desire to do so.

I don't really understand this, it's literally in the PDB? If this is some general statement about debugger development having been more disappointing then it has to be, I can only agree.

Usually, when something is proposed that is already possible on stable in other ways (usually for libstd additions that can also live in a free crate), we ask the authors to first explore this and come back with an experience report. Since you said you would like to start adding this to popular crates, what do you think about starting this process in parallel to this RFC? Add or extend builds scripts in those crates and collect the pain points or successes in a doc.

We discussed this in today's @rust-lang/cargo meeting, and we're still heavily debating whether it makes more sense to support this via Cargo, or whether it makes sense to only have the attribute that takes a string and let rustc handle this itself. In particular, we liked the idea that a #[natvis] attribute could easily use include_str! or even (via a macro) include every *.natvis file under a directory.
Obviously, adding #[natvis] support would be harder than just passing files to the linker. But we'd like to pick the right long-term solution. If we're going to want to support #[natvis] anyway, what would be the downside (apart from delay in implementation) of only supporting #[natvis] and letting anything else be built atop that?

Also, if we go the separate-files route and have cargo handle it, we need to specify whether it's valid to generate the separate files in a build.rs script. I would propose that we specify in the RFC that you cannot do that, because if you're trying to generate it we should use #[natvis] for that rather than build scripts.

@joshtriplett

We discussed this in today's @rust-lang/cargo meeting, and we're still heavily debating whether it makes more sense to support this via Cargo, or whether it makes sense to only have the attribute that takes a string and let rustc handle this itself. In particular, we liked the idea that a #[natvis] attribute could easily use include_str! or even (via a macro) include every *.natvis file under a directory.

I believe only supporting the attribute would be fine if we extend the attribute to allow for including separate .natvis files and not limiting the attribute to inline Natvis XML fragments. This was one of the ideas I wanted to bring up today. Extending the attribute in such a way would mean there wouldn't need to be any changes in Cargo to get this feature supported which I understand was a point of friction for this RFC.

what would be the downside (apart from delay in implementation) of only supporting #[natvis] and letting anything else be built atop that?

I don't see any major downsides to this approach. This could run into the issue of including the same .natvis file multiple times from different .rs files but the list of files shouldn't be hard to deduplicate.

Also, if we go the separate-files route and have cargo handle it, we need to specify whether it's valid to generate the separate files in a build.rs script. I would propose that we specify in the RFC that you cannot do that, because if you're trying to generate it we should use #[natvis] for that rather than build scripts.

I think removing Cargo from the equation solves this problem in itself.
Another issue that has been brought up in this RFC thread was the naming of the rustc flag and it being specific to Natvis, -C natvis={...}. With the attribute only approach, a new flag would no longer be necessary. The one downside to using the attribute only approach would be in ensuring the mapping of type name to its generated mangled name. This would need to be cleaned up to ensure the mapping is accurate, so the visualizations match up to the right type. This has also been brought up in a previous comment above.

@oli-obk Thanks for reviewing this!

Since you said you would like to start adding this to popular crates, what do you think about starting this process in parallel to this RFC? Add or extend builds scripts in those crates and collect the pain points or successes in a doc.

As I mentioned above in response to your previous comment. This approach would not work as expected since it would not gather the full set of .natvis files within transitive dependencies.
The Cargo team has suggested making this a rustc specific feature. This would remove the need for any changes in Cargo and can be done through attributes in Rust. I would like to know if the compiler team agrees with and/or approves this approach.

The one downside to using the attribute only approach would be in ensuring the mapping of type name to its generated mangled name.

Wouldn't it be easier to do with an attribute on a type? We could feed the mangled name as a templated argument to the natvis file and apply it before putting it into metadata.
On the concern of hardcoding natvis flags/attribute names: we could add it as field on a #[link] attribute and reserve the right to just silently or with a lint drop support some day in the future

Wouldn't it be easier to do with an attribute on a type? We could feed the mangled name as a templated argument to the natvis file and apply it before putting it into metadata.

I'm sorry, I was speaking to the inline Natvis XML fragment syntax, but yes you're correct there are clear ways around this to ensure we always generate the correct name for a given type.

On the concern of hardcoding natvis flags/attribute names: we could add it as field on a #[link] attribute and reserve the right to just silently or with a lint drop support some day in the future

I really like this idea. I think that makes this feature that much more flexible

In my opinion, I don't think we should attempt to have the compiler generate the mangled type names and inject them into the natvis file. I think the intent behind the idea to help users with that is good, but I think it adds a fair amount of complexity for not that much value.
Over the past year, I've gone from never touching a natvis file to writing quite a few, including some quite complex ones. During that time, writing the correct name to match a type has just never been an issue. In every Windows debugger, the type name you need to match is readily available and you really need to load this in a debugger to test and tweak the visualizer anyway. I won't claim this is true of every user, but I did want to give an experience report.
Additionally, if look at the natvis files we ship with Rust, we'll see that each file contains at least one instance where the natvis requires use of a cast to a particular type. In these cases, we either cast to a well known type (but with the C++ name instead of a Rust one) or to a type name that contains types used in the matched type. In either case, the user really needs to understand how to name types in a way the debugger engine will be able to correctly interpret. Having the compiler mangle & inject the type names at the appropriate places doesn't solve this and hides important information from the user.
Finally, I would like to note that VS and VS Code have a reasonably good experience when writing .natvis files. The xml namespace is used to drive autocompletion that is sensitive to what nodes and attributes are allowed at each place in the document. You can also achieve a pretty tight feedback loop by starting a debug session in VS or WinDBG, loading the natvis file and reloading just the natvis as you make edits (with the appropriate configuration, VS will watch your natvis file and reload automatically). This experience won't really work if we pursue something that requires embedding natvis in Rust source files. Overall, I think that would be a slightly worse developer experience than is currently available even if the compiler could generate your type names for you.

Oh, that makes a lot of sense, thanks! I think from my side that resolves the last concern I had.
I'll give the doc another full read and write a post for rfcbot.
On your end, please update the doc with the change to a general link attribute. I'm envisioning something like
#[cfg_attr(target_platform="msvc",link(file="foo.natvis", arg="/NATVIS"))]
struct Foo;

Bikeshed repaints are very welcome, I just wanted to throw out a design to start the discussion, and we can continue that after the RFC, too. There is no need for the RFC to make it finalized, just to get the direction right.

On your end, please update the doc with the change to a general link attribute. I'm envisioning something like
#[cfg_attr(target_platform="msvc",link(file="foo.natvis", arg="/NATVIS"))]
struct Foo;

My only concern with the #[link] attribute is that it is still very specific to Natvis. I believe having a more general attribute such as #![debugger_visualization(natvis_file = "foo.natvis")] is flexible enough to add additional debugger visualization options such as gdb debugger python scripts. Using the link attribute would not be flexible enough to support all of those use cases.

In my opinion, I don't think we should attempt to have the compiler generate the mangled type names and inject them into the natvis file. I think the intent behind the idea to help users with that is good, but I think it adds a fair amount of complexity for not that much value.

There's been a lot of back and forth on this thread about the importance of having a way to embed fragments of Natvis into Rust source directly. Some reviewers have stated that this should be the only way to add Natvis for Rust. i.e. this comment from above https://github.com/rust-lang/rfcs/pull/3191#issuecomment-1023877206:

I'm not even sure I even see a strong reason for supporting non-inline Natvis (other than getting XML highlighting and whatnot from having a separate file).

I believe having support for this is something that would be extremely helpful to a lot of Rust developers in terms of the ease of adding Natvis visualization for a given type. I also believe being able to include a separate .natvis file included via an attribute is essential as well. If there needs to be a lot of Natvis defined to properly represent how a type should be viewed under the debugger, it would be much cleaner to have this in a separate file. As @wesleywiser specified above, in VS and VS Code, the experience of writing Natvis files with autocompletion makes this a much nicer developer experience as well as constant iteration when loaded into a debugger. I think supporting both is the best way to move forward.

I think using the #[link] attribute is not a good approach. It puts the focus on how this stuff is implemented, rather than on the semantics that are relevant to the user.  It should be:
#![debugger_extension(natvis_file = "foo.natvis")]

or something similar, at mod scope.  It should be simple and understandable.  It shouldn't need the cfg_attr to turn it off when not targeting MSVC; that's just a needless barrier to every user of this.  Also, if the GNU Win32 toolchain does pick up support for this, then we will have needlessly made it more difficult than it should be to re-enable NatVis for GNU/Win32.
About whether NatVis XML should be inline in Rust files, or packaged in separate files.  I think it's basically mandatory that users have the option of writing XML files and just having a source code pointer that points to them.  The editing experience for NatVis in Visual Studio is pretty good, but it will be awful if it is embedded in a string inside Rust.  The reason is that you lose all semantic support from the IDE.  You lose Intellisense for the XML schema, auto-format for XML editing, etc.  You also make the experience of writing NatVis for Rust much different than it is from writing NatVis for C++, and that seems strictly negative to me.
I'd like to have the option of embedding short NatVis fragments into Rust source code, but I want that to be an option, not a requirement.  Having the ability to store NatVis in a separate *.natvis file is, I think, a hard requirement.

I think we can safely start with just allowing files and revisiting this before stabilization.
Yea, the link attribute is very low level, but it doesn't prevent a user defined proc macro attribute from working exactly as you described. Our derive api is also very low level and requires you to use helper crates for convenience.

Note that embedding a string means you can still use include_str! to embed a file

On your end, please update the doc with the change to a general link attribute. I'm envisioning something like
#[cfg_attr(target_platform="msvc",link(file="foo.natvis", arg="/NATVIS"))]
struct Foo;

This seems sub-optimal in terms of user experience: it requires annotating every type with a very boilerplatey annotation and requires the author to operate at a lower level of abstraction (with regards to the tool chain) than they usually do when writing source code.
It appears general, but only from the point of view of the implementation looking up (i.e., it encodes a specific implementation mechanism and makes that generally useful). From the user's point of view this is very specific. For the user, the task is adding metadata for debugging (how does that generalise? Perhaps adding metadata for other tools or adding metadata for any use, but I don't think it generalise to adding artifacts to the link step). Maybe in the future we want to support doing that without linking or maybe it requires a pre-processing step or whatever. I believe we should find an abstraction that generalises to different implementation mechanisms, rather than trying to expose the underlying implementation mechanism.

Yea, the link attribute is very low level, but it doesn't prevent a user defined proc macro attribute from working exactly as you described by expanding to the link attribute. Our derive api is also very low level and requires you to use helper crates for convenience.

This still ties the high level task to the low-level implementation and means the compiler/build system will never get an opportunity to help because they don't know the intent behind the annotation. It also adds the friction of adding an ecosystem crate for something which is all about core tooling (assuming somebody actually writes that crate and continues to maintain it).
I think that debugging is a core programming activity and one where the experience in Rust is poor. We should be doing everything we can to improve the experience including giving it first class support in the tool chain.

If we are generalizing, shouldn't we at least RFC two different schemes for debugging visualization instead of just natvis?

If we are generalizing, shouldn't we at least RFC two different schemes for debugging visualization instead of just natvis? (In order to not paint ourselves into a corner)

I agree with the desire to avoid painting ourselves into a corner.  I think we can meet that bar simply by having an attribute syntax that 1) indicates the general class of what we are trying to do (debugger visualizations), and yet 2) ensures that a specific usage of NatVis cannot collide with other systems.
Part of the reason I'm pushing back on this, slightly, is that debugger extension frameworks are quite heterogeneous. There isn't much to share between them, aside from the highly abstract notion that they improve the experience of debugging.
We can meet the first requirement with:
#![debugger_extension(...)]

where the set of metadata items nested within debugger_extension is a namespace that this RFC controls.  Then, we can meet requirement 2 by having the RFC specify the NatVis-related key(s) that can be nested under debugger_extension, while also reserving all other non-assigned keys.
In other words, the RFC can say that this is allowed:
#![debugger_visualization(natvis_file = "foo.natvis")]

and that the following keys are reserved (and would fail to compile, since they are not assigned), but could be made to work in the future:
#![debugger_visualization(gdb_script = "foo.py")]
#![debugger_visualization(windbg_script = "foo.js")]
#![debugger_visualization(lldb_script = "foo.?")]

This way, it's always clear and obvious to the user what an attribute means -- we don't have 5 different attributes, we have just one, whose meaning is clear, and we have a clear way to increase its support in the future.
And for the relevant targets, these debug facilities can "light up", when they are available.  There is no need for a cfg_attr to control them; rustc is in a much better position to know when these technologies are relevant to a particular compilation, and we should not burden the user with this.  I mentioned the example of GNU-Win32 adding support for NatVis; there's also the possibility that NatVis itself could support ELF targets, in the future.  So let's not burden the user with making that call, and let's let the toolchain do the best job it can do to improve the debugging experience.
If a debug annotation is present in source code, but not relevant to a particular target, then the compiler can just silently ignore it.  I think that's an acceptable experience.

My recurring point stands though: this reasoning should be in the doc, so no one has to read this discussion in the futureto find out the "why"

I will go ahead and update this RFC with the reasoning providing in this thread but will also make several substantial changes as well.

Specify that this RFC will not include any changes to Cargo.
Specify that a Rust attribute will be in control of including debugger visualization files.
Include reasoning as to why a new Rust attribute is required as opposed to using the !#[link]attribute.
Describe the attribute syntax and how it will support different schemes for debugger visualizations.

I believe this is all of the points that we have come to a consensus on in this RFC thread. I'll update this RFC to reflect this. Thanks for all the valuable input in helping shape this RFC.

Author of natvis-pdbs chiming in!  natvis-pdbs is taking the approach of Alternative 2:

Uses a build script (on the final .exe packages) to scan all crates for {crate}/*.natvis and {crate}/src/*.natvis
I'd be happy to add {crate}/dbgvis/natvis/*.natvis to natvis-pdb's the search list!
I'd be happy to add [package.metadata] autodiscovery/search paths.
Root and src are awkward places for natvis files, I just couldn't bring myself to finish bikeshedding something better 😄
Uses cargo:rustc-env=LINK=/NATVIS:...
I believe cargo:rustc-link-arg=/NATVIS:... was rejected by rustc at the time of authoring
I was inclined to eventually upstream natvis-pdb's logic into cargo to moot the need for per-application build scripts.
Never got around to finishing the requisite bikeshedding of what paths to search, though, so this RFC is good.
src/lib.rs is tiny and is compatibly licensed MIT OR Apache-2.0 (+ I'm happy to add LLVM linking exceptions if that's necessary?)
#![debugger_extension(natvis_file = "foo.natvis")] won't handle the existing crates with natvis files without updates.
There's few enough that this probably isn't a big deal, but worth mentioning?
#![debugger_extension(natvis_file = "foo.natvis")] instead of autodiscovery seems brittle (breaks when renamed.)
natvis_file = "debug/*.natvis" or natvis_dir = "debug" would make me happy though.

Reasons to not generate natvis files / embed them in *.rs attributes:

A major advantage of non-generated, non-embedded loose files: they can be hot-reloaded in Visual Studio without rebuilding!
This fast iteration is incredibly useful when authoring complicated <CustomListItems ...> based visualizers.
(This is, in fact, how the HashMap visualizers were initially developed, and fixed at least once.)
Even debugger unit tests aren't enough to stop people from subtly breaking natvis files on specific debuggers IME.
HashMap visualizers on stable are broken (fixed on nightly) because they worked in CDB but not VS, due to changes to tuples.
Having attributes or comments on HashMap would not have helped avoid this, since HashMap wasn't "changed".
Aside from VS and CDB parsing natvis files differently, there are third party debuggers that also parse natvis files differently.
Trying to make them harder to break is a non-starter IMO.  They're too brittle.  Prefer making them easy to fix instead.
Aforementioned xmlns-based auto-completion / intellisense in some IDEs for *.natvis files.
I'm strongly inclined to avoid inlining hand-authored natvis files into *.rs as a result.

On generating natvis files / embedding them into *.rs anyways:

I'm currently generating *.natvis via cargo run -p xtask --bin gen for a windows error codes in a crate I'm publishing soon.
Thousands upon thousands of individual error codes... it's too much for even me to want to hand author ;)
Oddball niche tags like <HResult Name="..."><HResultValue>...</HResultValue></HResult>.
Nothing cargo or rustc could really do to help here.
bitflags! and similar could be worth generating *.natvis for
I've also considered:

build.rs scripts instead of xtasks (avoided as less friendly for sccache / rust-analyzer / ...)
cargo subcommands using syn (which could scan for proc-macro stub attributes)
proc macros doing file I/O (sketchy, but reasonable for prototyping right now)
proc macros emitting inventory attributes (+ stub bins reading from inventory and writing to *.natvis)

I'm likely to continue preferring one of the above approaches even if rust adds built-in support for something else:

Will let me support whatever weird edge case is causing me to resort to generating natvis files
Won't rely on nightly

Misc. errata:

Copying hand-authored *.natvis to target doesn't seem all that necessary.
target/**/*.pdb already embeds a copy of the natvis files after linking, not merely a path.
rustc already uses /NATVIS:... for all msvc-style targets if rust-src is available, to embed stdlib natvis files.

Nominating for compiler so it will show in the weekly t-compiler agenda
@rustbot label +I-compiler-nominated

Mentioned during T-compiler meeting (on zulip)
@Mark-Simulacrum would you kindly
@rustbot label -I-compiler-nominated
for me pls? :)

We discussed this in today's @rust-lang/cargo meeting, and we're still heavily debating whether it makes more sense to support this via Cargo, or whether it makes sense to only have the attribute that takes a string and let rustc handle this itself. In particular, we liked the idea that a #[natvis] attribute could easily use include_str! or even (via a macro) include every *.natvis file under a directory.
Obviously, adding #[natvis] support would be harder than just passing files to the linker. But we'd like to pick the right long-term solution. If we're going to want to support #[natvis] anyway, what would be the downside (apart from delay in implementation) of only supporting #[natvis] and letting anything else be built atop that?

@joshtriplett thanks for posting this input from the Cargo team! The RFC has changed quite a bit since then and is much closer to the alternative idea you presented here than it was when you wrote this comment. Would you all mind discussing this again and letting us know what you think? Assuming there are no objections, I believe it makes sense to remove T-cargo from this RFC as it no longer interacts with Cargo.

@wesleywiser @ridwanabdillahi The new version does indeed not intersect with Cargo, so I've removed the T-cargo label.
Since it adds a new attribute to the language, it does intersect with Lang, so I've added the T-lang label.

Speaking for myself and not either the Cargo or Lang teams:
The concept of this attribute seems reasonable. Note, though, that we've had a long history of file path attributes having complex semantics. "relative to the location of the attribute" could use some clarification. Consider, for instance, if you have code like #[path = "x/y/z.rs"] mod z;. Is a natvis_file inside z.rs relative to x/y/? What about the difference between abc/xyz/mod.rs vs abc/xyz.rs? In the former case is natvis_file relative to abc/xyz/, while in the latter case it's relative to abc/? I'd like to see that precisely specified somewhere. (The right way to handle that might be to add such clarifications to include_str! or similar and then state that this resolves paths identically to include_str!.)
Also, as a more minor issue: #[debugger_visualizer(natvis_file = "../foo.natvis")] seems really verbose. (debugger_visualizer also feels like awkward-er phrasing-er.) I'm wondering if that could be at least somewhat more compact. For instance, debug_vis(natvis = "...", gdb_script = "...") (I don't want to bikeshed this, I just want to provide that feedback.)

@joshtriplett About the verbosity.  First, about the top-level attribute name.  I think this is a case where we use the principle of "the more often a thing is used, the shorter the name should be". In this case, debugger visualizations will be used fairly infrequently, and a lot of people reading the code will not even know what they are.  So I think erring on the side of clarity is best, and debugger_visualizer or debugger_extension is pretty clear.  Maybe we could shorten it to debug_ext or dbg_ext?
Using _extension might be good, since it gives us a place to put almost anything relating to debuggers, not just "visualizers".
For the keys under debugger_visualizer, we need to be able to distinguish between "this is a NatVis thing and it's in a separate file" and "this is a NatVis thing and the attribute value is the NatVis description itself".  I think natvis_file and natvis_xml are the best ways to identify those two cases.
About the semantics of file locations. This is where I look to you & Lang team to tell us how to structure this.  My default would be "we use the same rules as include_str!(...)".  Either that, or we use only one key, natvis, and the RFC says explicitly that "the way to put NatVis into a separate file is to use #![debugger_visualizer(natvis = include_str!("foo.natvis"))] ".  If that already works and works well, then I think that would be a fine place to land.  The only problem I can see with that is that the code that handles NatVis, inside Rustc, might lose the original filename, which might make debugging the NatVis support itself a little difficult at times. But I'm willing to give up that for a better developer experience.

@joshtriplett

The right way to handle that might be to add such clarifications to include_str! or similar and then state that this resolves paths identically to include_str!.

The plan is for this to resolve paths in a manner that is identical to how include_str! does. I can clearly state this in the RFC to ensure there is no confusion.

Also, as a more minor issue: #[debugger_visualizer(natvis_file = "../foo.natvis")] seems really verbose.

I thought making this more verbose would ensure there is no confusion on how this attribute was meant to be used. For instance, using natvis_file as opposed to natvis in the attribute is meant to be able to distinguish that the item takes a file path and not just the contents to place within a natvis file which is called out as a future goal in this RFC. It helps to create a potential path forward in reusing the same attribute to support that use case as well.

(debugger_visualizer also feels like awkward-er phrasing-er.)

I'm fine with updating the attribute name itself as long as it is clear what it is meant to be used for. dbgvis, dbg_vis, debug_vis or dbg_ext have all men mentioned in the comments thread. dbgvis or dbgext sound good to me but if there is a better suggestion or a strong argument for one over the other that is fine with me as well.

Thanks to all for giving this RFC a review. The latest changes in the RFC include:

Specifying GDB scripts will have their contents embedded into the .debug_gdb_scritps section as opposed to referencing a filename.
Specifying the #[debugger_visualizer] attribute targets modules and can be used as an inner attribute (with the leading !) or an outer attribute specifically targeting a module(i.e. #[debugger_visualizer] mod foo;).
Removes specifying the generated name of a debugger visualizer file, this is an implementation detail and does not need to be hardcoded in the RFC. (Thanks! @michaelwoerister )
Specifying the #[debugger_visualizer] attribute resolves paths identically to how the include_str! macro resolves paths.
Removes the need for copying visualizer files to a specific directory in the target folder. Uses a temp directory instead since the contents will be embedded in the PDB/ELF anyway.

Confirming some behavior: include_str! does not take inline modules into account, only the file containing include_str!. So, for instance:
/tmp/pathtest/src$ cat main.rs
pub mod a {
    pub mod b {
        pub mod c {
            pub const C1: &str = include_str!("path");
        }
    }
}

const C2: &str = {
    pub mod a {
        pub mod b {
            pub mod c {
                pub const S: &str = include_str!("path");
            }
        }
    }
    a::b::c::S
};

fn main() {
    dbg!(a::b::c::C1, C2);

}
/tmp/pathtest/src$ cat path
src
/tmp/pathtest/src$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.00s
     Running `/tmp/pathtest/target/debug/pathtest`
[src/main.rs:21] a::b::c::C1 = "src\n"
[src/main.rs:21] C2 = "src\n"

This is the behavior I would expect.
@rfcbot reviewed

The file path specified must be
relative to the location of the attribute and is resolved in a manner that is
identical to how paths are resolved in the include_str! macro

Right now the compiler's approach to loading files is to load everything during macro expansion, after that all the correspondence between modules and filesystem locations is lost.
So this attribute will either require some special logic during macro expansion (*), or require keeping all of the filesystem structure of the crate for later.
In the past this was the reason why #[doc(include = "my_file.md")] (https://github.com/rust-lang/rust/issues/44732) was replaced with #[doc = include_str!("my_file.md")].
(*) That special logic will either load the file, or normalize the path to some "absolute" form, and keep the results in some side table.
Can the attribute be limited to crate-level instead?
In practice with cargo these files will probably be kept in some single top-level natvis directory anyway, instead of being interspersed between regular source files in module subdirectories.

@joshtriplett

Confirming some behavior: include_str! does not take inline modules into account, only the file containing include_str!

Yes, this is the behavior for the #[debugger_visualizer] attribute as well. It only takes into account the file containing the attribute when resolving paths.
@petrochenkov

Right now the compiler's approach to loading files is to load everything during macro expansion, after that all the correspondence between modules and filesystem locations is lost.

This attribute does not need to know the correspondence between modules and filesystem locations. The attribute only takes into account its own location within the source tree and resolves the path relative to this. Since attributes have the knowledge of where in the source tree they live, I do not believe this would be an issue.

Can the attribute be limited to crate-level instead?

Yes it could. I don't have a strong preference one way or the other. Is there a reason you would suggest it to be a crate-level attribute?

Oh wow, I've checked how include(_str,_bytes)! work and they actually use span_to_filename for obtaining the path.
That's incorrect in general case, and quite different from how mod items are loaded.
TBH, I'd prefer if all file loading was delegated to include_*! family macros.
Too bad you cannot expand macros deep inside attributes.

Since attributes have the knowledge of where in the source tree they live

They do not, that was my point, this knowledge is only available during macro expansion (at least if the include macros are implemented correctly).

That's incorrect in general case, and quite different from how mod items are loaded.

I see. I currently have a draft PR out for an implementation of this RFC https://github.com/rust-lang/rust/pull/91779. I'll reach out to you via zulip to better understand the extent of this problem.

I'll reach out to you via zulip to better understand the extent of this problem.

The conclusion was that it's some small scale implementation annoyance due to resolving filesystem paths from attributes being a novel mechanism previously unused in the language, but it's not a blocker.

@rfcbot fcp reviewed

These nitpicks from mainly an implementer's perspective.

Through the use of a new Rust attribute, #[debugger_visualizer], the compiler will
encode the contents of the .natvis file in the crate metadata if the target is an rlib. If the target is a dll or exe, the /NATVIS MSVC linker flag is set for each .natvis file which will embed the Natvis visualizations into the PDB.

the compiler will encode the contents of the .py file in the crate metadata if the target is an rlib

The Rust compiler will serialize the contents of the file specified via the #[debugger_visualizer] attribute and store it in the crate metadata.

Returning to the metadata encoding within rlibs discussion. I don't believe metadata is a great place to put bundled files. rlib is already an archive that bundles a number of files, and placing the natviz files into the archive itself would make much more sense to me. At the very least from the perspective of it enabling use of standard tooling to inspect the contents.
As far as the pretty-printers are concerned the toolchain might as well avoid the work of moving the data around across compilations all the time by producing an object with an concatenated .debug_gdb_scripts section in the first place. Once that's done rustc no longer needs to think about the upstream pretty printers at all, anymore.
I guess overall my impression is that this wording is over-constraining the implementation-time decisions here. None of these guarantees for where data is placed are required for an implementation of this feature, nor are these the guarantees we want to make about the structure of the rlibs, I feel. As long as rustc can guarantee that the contents of these are bundled and preserved throughout compilation, where exactly within the rlib these contents are stored shouldn't matter.

To provide Natvis files, developers create a file with the .natvis file extension.

To provide pretty printers, developers create a file with the .py file extension

Why the extension restriction? The type of the file is already specified by the attribute. If there's a reason the file extension must be exactly natvis or py, the RFC should specify what happens (error, warning, ignore the attribute?) when it is not (IIRC it does not).

This attribute will directly target modules which means the syntax #![debugger_visualizer] is also valid when placed at the module level.

AFAIK crate is a distinct kind of entity and is not a module item. This wording in turn disallows application of the #![debugger_visualizer] to the crate itself, which is something the examples do.
More generally I would like a separate paragraph listing the exact list of the kinds of entities that this attribute is applicable to (crate, module item, anything else?) and a specification of what happens when this attribute is applied to anything else (error, warning, ignore?)
Finally, hand-waving a bit, but I'm not entirely sure if allowing this attribute at a module level today is the best idea. At the very least allowing something like this now immediately blocks any potential for future improvements to the attribute's behaviour. For example, I could see rustc automatically doing the work to scope the activation of the rules within the natviz/pretty-printer to the types contained within the module in question or something.

Sorry if I missed answers to any of these in prior discussion. I did try grepping through the page, but the discussion is definitely too long for me to catch up on properly.

@nagisa Thank you so much for taking the time to provide feedback.

As far as the pretty-printers are concerned the toolchain might as well avoid the work of moving the data around across compilations all the time by producing an object with an concatenated .debug_gdb_scripts section in the first place. Once that's done rustc no longer needs to think about the upstream pretty printers at all, anymore.

This is very interesting, in terms of the RFC I will remove any specifics as to how or when rustc will package the pretty printers into the .debug_gdb_scripts section. This will allow for flexibility in terms of the implementation of this feature.

I guess overall my impression is that this wording is over-constraining the implementation-time decisions here. None of these guarantees for where data is placed are required for an implementation of this feature, nor are these the guarantees we want to make about the structure of the rlibs, I feel. As long as rustc can guarantee that the contents of these are bundled and preserved throughout compilation, where exactly within the rlib these contents are stored shouldn't matter.

With this being my first RFC I believe I was too specific in the implementation details. I'll peel back a bit in terms of the implementation-time decisions and keep it at more of a high level.

Why the extension restriction? The type of the file is already specified by the attribute. If there's a reason the file extension must be exactly natvis or py, the RFC should specify what happens (error, warning, ignore the attribute?) when it is not (IIRC it does not).

I can't speak to any extension restriction for pretty printers, but in terms of Natvis, the MSVC linker flag /NATVIS: currently accepts a file with any file extension but the debuggers (WinDbg, VS Debugger) are unable to load these files into the debugging session automatically when the file extension is not .natvis. However, WinDbg does support directly loading a natvis file with any extension with the .nvload command inside the debugger. This sadky takes away from the point of this RFC that Rust developers shouldn't need to manuallly load anything for Natvis visualizations to apply. This is only mentioned in the RFC as an example for how to create these files, there is no error or warning if the files have a different extension since neither the linker or the debuggers throw an error.

More generally I would like a separate paragraph listing the exact list of the kinds of entities that this attribute is applicable to (crate, module item, anything else?) and a specification of what happens when this attribute is applied to anything else (error, warning, ignore?)

Currently this attribute is only applicable to module items and at the crate-level. I will update this so that this attribute is only applicable at the crate-level and will throw an error if used any where else.

Finally, hand-waving a bit, but I'm not entirely sure if allowing this attribute at a module level today is the best idea. At the very least allowing something like this now immediately blocks any potential for future improvements to the attribute's behaviour. For example, I could see rustc automatically doing the work to scope the activation of the rules within the natviz/pretty-printer to the types contained within the module in question or something.

Currently the RFC mentions that this attribute can be allowed to target module items as well as being a crate-level attribute which I understand are a distinct kind of entity in terms of how they are applied. @petrochenkov also suggested making this attribute specifically a crate-level attribute which sounds like the right direction to go. I think this will resolve the concern you have mentioned here in terms of this attribute targeting module items.

@nagisa

Why the extension restriction? The type of the file is already specified by the attribute. If there's a reason the file extension must be exactly natvis or py, the RFC should specify what happens (error, warning, ignore the attribute?) when it is not (IIRC it does not).

The more I think about it, this issue is moot. Since the contents of this file is embedded in the crate metadata and written out to a temp file with the correct .natvis extension in the case of a natvis file, there should not be any issues with loading these debugger visualizer files in either of the 2 debuggers. Thank you for catching this, I'll update the RFC to remove any reference to specific file extensions.

I'm worried that moving in the direction of limiting the #[debugger_visualizer] attribute to the crate-level would only make it more difficult for use by Rust developers. In the case of a crate with multiple modules, having Natvis files next to the .rs file which defines the type that the visualizations work with would be extremely valuable. This would also help with avoiding Natvis files from becoming stale when types deep within the module system are changed/updated. Being able to add the #[debugger visualizer] attribute at the module level would extremely simplify the paths specified on the attributes themselves as opposed to forcing users to place this attribute solely at the crate-level.
This also would work well with crates that turn off entire modules depending on which features have been activated. If a cfg feature removes a module from being active, then any of the #![debugger_visualizer] attributes used within this module can be dropped which means none of the debugger visualizers would be embedded into the binary. I think this would be great that way we aren't adding debugger visualizers for types that aren't going to exist anyway.

For example, I could see rustc automatically doing the work to scope the activation of the rules within the natviz/pretty-printer to the types contained within the module in question or something.

I'm not sure I understand why this would need to happen? If a debugger visualization exists for a given type, it would be applied once that type is used. I don't see why we would want to enable/disable a visualization for a type especially since the debugger visualizations are directly embedded into a binary.

I'm not sure I understand why this would need to happen? If a debugger visualization exists for a given type, it would be applied once that type is used. I don't see why we would want to enable/disable a visualization for a type especially since the debugger visualizations are directly embedded into a binary.

Well, the thing is that there can be multiple types that share the same name, so it seems quite natural to me that if I have something along the lines of…
mod a {
    #![debugger_visualizer="a.natviz"]
    struct Mango { ... };
}

mod b {
    #![debugger_visualizer="b.natviz"]
    struct Mango { ... };
}

then I can write the natviz like
<!-- a.natviz -->
<?xml version="1.0" encoding="utf-8"?>
<AutoVisualizer xmlns="http://schemas.microsoft.com/vstudio/debugger/natvis/2010">
  <Type Name="Mango">
    .
    .
  </Type>

</AutoVisualizer>

<!-- b.natviz -->
<?xml version="1.0" encoding="utf-8"?>
<AutoVisualizer xmlns="http://schemas.microsoft.com/vstudio/debugger/natvis/2010">
  <Type Name="Mango">
    .
    .
  </Type>

</AutoVisualizer>

and not have to worry about which visualization would end up getting applied where.
This is also a concern between crates, of course, but it feels to me like “use paths for types reachable from crate root” is a quite straightforward rule to follow, and everything more than that is, honestly, sounds like a significant burden by itself – a much bigger one than having to specify slightly longer paths.
I'm not necessarily saying that some sort of automatic processing of the files is the best approach. But I would at least like to see an explicit decision that this is something we definitely do not want to see now or in the future. And if we aren't willing to make such call today, we should stay a little conservative as to where we allow the attribute to appear for the time being.
Does that help to clarify my line of thought?

Well, the thing is that there can be multiple types that share the same name, so it seems quite natural to me that if I have something along the lines of…

Thank you for clarifying your line of thought. In the case mentioned above this actually would not be an issue since the names generated for those types would actually be distinct. For the example above, the fully qualified names for these types for some crate foo would be foo::a::Mango and foo::b::Mango. The Natvis visualization would match against the fully qualified type names and therefore they would be unique and would not clash. The same would be true for types with the same name in different crates.

But I would at least like to see an explicit decision that this is something we definitely do not want to see now or in the future.

I believe for this specific attribute this is not something that we want to allow. In the future goals section, there is a mention about having support for defining Natvis visualizations or pretty printers "inline". Inline meaning an attribute or a proc-macro would take a snippet of a visualization schema and create the boilerplate needed around it and embed this in the binary as well.
This could potentially use the same attribute which would mean updating the attribute to allow for targeting types as well and using a different meta-item to switch on which target is allowed. Is this enough of an explicit decision for what is acceptable now and what can be added later?
In either case we would not want any automatic processing of the files by rustc.

How do the embedding protocols provide for their own versioning and evolution?
For example, what if GDB came out with a new major version of its scripting interface, and your library crate wanted to provide scripts for v1 and v2 respectively. How would you go about this? cargo doesn't support mutually exclusive features, nor does the build necessarily have control over what debugger the binary gets run in, so it seems like we would have to provide for embedding both and annotate with a version somehow.
Related question: These are considered debug sections and would get stripped out when the strip option is enabled, correct?

I am concern about having needed to maintain multiple separate implementation for debug, say if I use one platform and some contributors contributed another platform debugging visualizers, then I had to maintain both even though I don't have the access to it, when there are breaking changes, chances is that it might be dropped due to limited maintenance capability on a small crate.

The Compiler and Lang Teams have accepted this RFC! 🎉
Please see the tracking issue for further discussion: https://github.com/rust-lang/rust/issues/95939

Just noticed this RFC. Does it support using a custom function declared on the type to be called to produce the value? The scenario is a Rust type, that wraps a handle + group of accessor functions. E.g.
struct File {
  file_handle: isize,
}

impl Tensor {
  pub fn name(&self) -> String {
     let raw_name = winapi.GetFileName(self.file_handle);
     ...
}

@lostmsu

Just noticed this RFC. Does it support using a custom function declared on the type to be called to produce the value?

No this scenario is not supported. Currently the supported scenario is having standalone visualizer files, Natvis or pretty printers, and using the attribute proposed to include those files directly. It does not support dynamically generating the visualization descriptions.

I've just become aware of this RFC and skimmed at least all of comments here. One thing that I don't see discussed in the thread (although it is mentioned in the RFC text itself) is why, specifically, we can't use Debug impls for this. The RFC says this:

Alternative 4: miri executes the MIR of a Debug impl within a debugger
Supporting this option would mean that changes to cargo and rustc are not necessary. This would have the added benefit of taking full advantage of existing implementations of the Debug trait. Many Rust developers already implement the Debug trait which is used to format how types should be viewed, this would only ease the debugging quality of Rust when viewed under any debugger. This option also has the added benefit of not requiring any changes to a crate from a Rust developer by leveraging existing Debug impls.
The drawbacks for this option is that this has not been fully investigated to determine its viability. This could be a great potential feature to ease debugging Rust but without concrete data to push this towards a potential RFC, I would assume supporting debugging in the systems that are already heavily used by the Rust community to be a higher priority. If/when this option becomes a bit more viable, there would be nothing stopping it from becoming a true feature.

The idea of just using Debug impls seems like a massive win, right? So I must be missing something. Instead of having to explicitly maintain separate debug files, you instead get nice pretty printed displays automatically for virtually every type. The motivating example in the RFC is particularly relevant here. Consider this program:
use regex::Regex;

fn find_c_defines(input: &str) {
    let rx = Regex::new(r#"^#define\s+(\w+)\s+([0-9]+)\s*(//(.*))?"#).unwrap();
    for captures in rx.captures_iter(input) {
        println!("{:?}", captures);
    }
}

fn main() {
    find_c_defines("#define SOME_CONSTANT 42");
}

It outputs:
Captures({0: Some("#define SOME_CONSTANT 42"), 1: Some("SOME_CONSTANT"), 2: Some("42"), 3: None, 4: None})

And this works because the derived Debug impl for Captures is really not helpful at all, as explained in the RFC. So it has its own custom Debug impl to make the output much more useful. Having to redo these sorts of hand-tuned Debug impls seems pretty unfortunate and a hard sell for me personally.
Looking at the natvis example in the RFC:
<?xml version="1.0" encoding="utf-8"?>
<AutoVisualizer xmlns="http://schemas.microsoft.com/vstudio/debugger/natvis/2010">
    <Type Name="foo::FancyRect">
      <DisplayString>({x},{y}) + ({dx}, {dy})</DisplayString>
      <Expand>
        <Synthetic Name="LowerLeft">
          <DisplayString>({x}, {y})</DisplayString>
        </Synthetic>
        <Synthetic Name="UpperLeft">
          <DisplayString>({x}, {y + dy})</DisplayString>
        </Synthetic>
        <Synthetic Name="UpperRight">
          <DisplayString>({x + dx}, {y + dy})</DisplayString>
        </Synthetic>
        <Synthetic Name="LowerRight">
          <DisplayString>({x + dx}, {y})</DisplayString>
        </Synthetic>
      </Expand>
    </Type>
</AutoVisualizer>

This looks like some kind of interpreter is needed for this anyway? How does this work? And why can't Debug be called instead? Moreover, can someone show me the natvis file that will make the debugger visualization nicer for the Captures type?
So I guess my question boils down to: is supporting Debug impls here "just" a matter of implementation work? Or is there a semantic mismatch behind what you want to see in a debugger versus what a Debug impl gives you?

We've actually prototyped the MIR interpreter approach, all the way through calling Debug imple. It's compelling, it's a great experience. But it's several orders of magnitude more complexity to implement and deploy. There are some nontrivial things to figure out, such as how do you store the MIR after compilation, how do you interpret it against the state of a running program, or even against a crash dump. How do you manage the problem of multiple compiler versions, etc.
We'd like to do that work, and do it the right way (open collaboration, RFC, etc.). But meanwhile, in parallel, we felt that NatVis could provide a lot of value, relatively quickly, and with very low complexity cost and low resource cost.
We can view them as complementary, overlapping approaches. We can get immediate benefits from NatVis, while working toward the interpreter approach.
The interpreter approach also enables some really wild capabilities, such as writing debug extensions directly in source code. Not just Debug impls, but arbitrary debugger commands / extensions, using a dyn trait obj that represents the hosting debugger environment.

The RFC lays out a few different approaches to the "debugger visualizer" problem but I think they can be mostly summed up in three categories:

Use the existing functionality of the platform's debugger

Pros:

Using the native debugger functionality will probably offer a better UX for the user because it's tied into the debugger itself. As an example, using natvis in WinDbg means nested items are represented in the GUI as expandable/collapsible elements.
One of the selling points of Rust IMO is that we try hard to integrate with the underlying system and not do something opaque on top of it. A lot of higher-level languages aren't really debugable using WinDbg/gdb/lldb and you have to use whatever debugger ships with your language, which is often missing features in comparison to the platform's debugger.
We don't have to implement any kind of debugger plugin to make this work. To answer your last question here: the NatVis interpreter is embedded into Windows debuggers. The equivalent functionality is embedded into gdb/lldb.

Cons:

You have to implement debug formatting multiple times and keep them in sync. This is a significant downside but there are ways to mitigate many of the problems here. These aren't currently implemented but there are multiple, reasonable approaches we can take here which are backwards compatible with this RFC.

Call the <MyStruct as Debug>::fmt method directly

Pros:

Debug formatting only has to be implemented once!
You can write normal Rust code instead of needing to write NatVis/Python expressions to format your types.

Cons:

As is currently implemented, the debug function may not actually exist in the binary. If it's never called, the compiler can elide generating code for it and in the case of optimized programs, LLVM or the linker may remove it since it's dead code. There are ways around this but in any case, we still have the problem of debug infrastructure being stored in the binary instead of in debuginfo which hurts compile times and binary sizes.

MIR interpreter in the debugger

Pros:

It's very cool!
No impact to binary size since the MIR could be stored somewhere other than the binary.
Debug formatting only has to be implemented once!
You write normal Rust code.
Enables other, more advanced scenarios.

Cons:

Requires writing a debugger plugin for every platform's debugger we want to support. This is a non-trivial amount of work.
MIR is currently unstable (but there are efforts to create a stable-MIR) which means the debugger plugin must link to the same version of the compiler that built the code.

Having said all of that, this RFC being accepted does not mean we close the door to options 2 or 3. For Rust to integrate well with the underlying platform, we need to have a solid implementation of option 1, which this RFC provides. It would be great to go above and beyond that as well!

I see, okay. I think that makes sense. For context, I didn't even know (2) and (3) were distinct choices. So there is a lot that I don't know here.
For (1), it sounds like what you're saying is that the natvis file works because the debugger has its own interpreter. To what extent does that work exactly? What would a natvis file for a regex::Captures look like? Namely, there are no public Captures fields and its representation is not particularly straight-forward. So you really want to be able to call methods on the Captures type to get a nice debug display. Is that possible?
I think I'm just having a hard time seeing how option (1) materially improves things if folks don't put in the work to maintain these debugging files alongside their crates. Have library authors been consulted about the overall user experience here? (I'm still pretty unclear on it myself. It's hard to even know what's possible exactly.)
To be clear, I see what y'all are saying. And the progression here makes sense. And I also agree that having a way to expose the underlying debugger's functionality is also a good idea. And I love the idea of making the debugging UX better. Maybe I'm just jumping in too early here before the docs for how to effectively use this feature are written. :)

@wesleywiser wrote in https://github.com/rust-lang/rfcs/pull/3191#issuecomment-1148733043:

The RFC lays out a few different approaches to the "debugger visualizer" problem but I think they can be mostly summed up in three categories: [,,,]

Perhaps there is a fourth way? I was experimenting with code similar to this:
fn show_debug(dyn_debug: &dyn std::fmt::Debug) {
    println!("{dyn_debug:?}\n");
}

then doing p show_debug(your_value_you_want_to_look_at) in gdb. However this doesn't work with gdb, because gdb seems not to be able to cast to trait objects. While Rust code can do this:
let trait_object: &dyn std::fmt::Debug = &"example value";

you can't do this in gdb: set trait_object = &"example value" fails with Invalid cast..
I tried to work around this limitation of gdb by writing a helper function to do this for me, however I was not successful. I can't use a generic function because it might be not monomorphised for the type of the value I want to look at. I experimented with std::ptr::metadata() (RFC 2580), however I couldn't find a way to bypass the need to cast to a trait object from inside gdb. RFC 2580 doesn't offer the possibility to create a vtable directly from the trait. It was like kicking the can down the road, somewhere the value must be converted to a trait object, and boom Invalid Cast. or even segmentation violations.
I think gdb should be able to cast to a trait object. Then we might have a simple solution for debugger visualisations with Debug!
I propose to go upstream and ask for the feature of casting to trait objects.
I didn't look at Windows debuggers, but I can imagine that they already have this covered.

An impl Debug serves different semantics than special Visualizations like Natviz files. Using <MyStruct as Debug>::fmt is therefore ideal for the main visualisation of the types. Natviz files are also concerned with the visualization of the fields.
What follows are my ideas/dreams for the far future of rust debugging. Notably, they have yet to be implemented so it's unclear to me what exactly is required from a technical perspective. It also builds upon scenarios 2/3 from https://github.com/rust-lang/rfcs/pull/3191#issuecomment-1148733043
Using <MyStruct as Debug>::fmt is great to summarize the information on a given type but gives no instructions as to what fields the debugger should display.
I'd propose adding a new Trait that can specify exactly this behavior. This would allow users to write rust code once that could then be provided to debuggers. When the trait is not implemented it simply uses the default behavior, similar to how Debug impls should be handled then.
This trait would work similarly to Debug. There would be one method to implement and a mutable formatter &mut std::fmt::DebuggerVisualizer is passed to the method. The formatter would expose a builder pattern to define the visualization of the debugger. So methods like these would probably fit nicely there:

DebuggerVisualizer::nativeFields(&mut self)->&mut self - adds an entry [native Fields] under which the underlying fields of the type are shown.
DebuggerVisualizer::section(&mut self,name:String,description:String,f:F)->&mut self - adds a section with given name and description and populates it with the result from the closure that gets a &mut DebuggerVisualizer.
DebuggerVisualizer::field<T>(&mut self,name:String,t:T)->&mut self - adds a field with the type that should populate it. Notably this type has no Debug or DebuggerVisualizer bounds since the Debugger can just display it normally (but use the impls when present).

Some more thoughts about this

possibly add some way to let users know which fields are actually present? Is a [native Fields] section enough?
rustc should probably only generate one such function that can handle generics by using trampolines internally. We don't want implementations for all Generics to bloat the size.
These implementations must be compiled in to be used. Should this be done into a separate binary as to not inflate the debug build binary?

Hello! FWIW, if that could simplify things, we could easily add in LLD parsing of /NATVIS flags in the .drectve section embedded in the .objs/.rlib for that crate. Here: https://github.com/llvm/llvm-project/blob/main/lld/COFF/Driver.cpp#L414