mesonpy: replace auditwheel with an in-house vendoring mechanism

Signed-off-by: Filipe Laíns<EMAIL_ADDRESS>

@rgommers the tests are failing on 3.7 due to weirdness in the ABI flags, which fully goes away in 3.8 due to several improvements to the build system. Since SciPy only seems to be supporting 3.8+, do you think it makes sense to try to fix it, or should we just drop 3.7 for now? The benefit seems minimal to me, and we can always bring it back in the future if needed.

Anyway, after merging this, we should be ready for a release.

Since SciPy only seems to be supporting 3.8+, do you think it makes sense to try to fix it, or should we just drop 3.7 for now? The benefit of working on this seems minimal to me, and we can always bring it back in the future if needed.

Agreed, it seems fine to drop 3.7; SciPy and other scientific projects all won't have a need for that (due to NEP 29).

Have you thought about how to make this configurable? We established that there are valid reasons to do this bundling, and also valid reasons to skip it (xref #20 (comment)).

Yes. I think that definitely makes sense if we run auditwheel by default, but I am not sure if I feel the same way if we do not.
IMO, I think we should probably be running it by default, so that user installs via sdist are less prone to breaking, but that does have drawbacks.
What do you want to do?

I did a bit of testing. First, a trivial fix for this PR:
diff --git a/meson.build b/meson.build
index 0514e2b..53a6d90 100644
--- a/meson.build
+++ b/meson.build
@@ -10,6 +10,7 @@ endif
 py.install_sources(
   'mesonpy/__init__.py',
   'mesonpy/_compat.py',
+  'mesonpy/_elf.py',
   'mesonpy/_tags.py',
   'mesonpy/_util.py',
   subdir: 'mesonpy',
diff --git a/pyproject.toml b/pyproject.toml
index 7030ec2..7ffdc78 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -27,6 +27,7 @@ classifiers = [
 dependencies = [
   'meson>=0.60.0',
   'ninja',
+  'pep621',
   'tomli>=1.0.0',
   'typing-extensions>=3.7.4; python_version<"3.8"',
 ]

Then I can run python -m build --no-isolation --skip-dependency-check in my test-mesonpy branch.
The sdist gets created (two warnings; the first one is unexpected/new for me, we're making the second one a hard error in SciPy's current setup.py):
+ meson dist --no-tests --formats gztar
WARNING: Repository has uncommitted changes that will not be included in the dist tarball
WARNING: Submodule 'doc/sphinxext/doc/scipy-sphinx-theme' is not checked out and cannot be added to the dist
Created /tmp/mesonpy-aenppslx/build/meson-dist/SciPy-1.9.0.dev0.tar.gz

I'd recommend to either pick up uncommitted changes, or make it a hard error - this seems a little confusing. My changes were to pyproject.toml, so they seemed to be picked up - but when building the sdist from the wheel, it ended up switching back to setup.py because I hadn't committed my build-backend change. sdist creation is also fairly slow (I guess it does a fresh git checkout under the hood)
It then goes on to build a wheel, showing the build output (this is good, no need to add -v like for pip). The build itself succeeds, but then there's some warnings before the _elf part has a problem:

WARNING Using heuristics to map files to wheel, this may result in incorrect locations
WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/mesonpy-2qdx8tuh/install/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/scipy/__config__.py (/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/scipy/__config__.py)
WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/mesonpy-2qdx8tuh/install/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/scipy/version.py (/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/scipy/version.py)
WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/mesonpy-2qdx8tuh/install/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/scipy/special/tests/data/boost.npz (/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/scipy/special/tests/data/boost.npz)
WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/mesonpy-2qdx8tuh/install/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/scipy/special/tests/data/gsl.npz (/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/scipy/special/tests/data/gsl.npz)
WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/mesonpy-2qdx8tuh/install/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/scipy/special/tests/data/local.npz (/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/scipy/special/tests/data/local.npz)
WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/mesonpy-2qdx8tuh/install/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/scipy/linalg/cython_blas.pyx (/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/scipy/linalg/cython_blas.pyx)
WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/mesonpy-2qdx8tuh/install/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/scipy/linalg/cython_lapack.pyx (/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/scipy/linalg/cython_lapack.pyx)
WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/mesonpy-2qdx8tuh/install/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/scipy/linalg/cython_blas.pxd (/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/scipy/linalg/cython_blas.pxd)
WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/mesonpy-2qdx8tuh/install/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/scipy/linalg/cython_lapack.pxd (/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/scipy/linalg/cython_lapack.pxd)
WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/mesonpy-2qdx8tuh/install/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/scipy/linalg/_blas_subroutines.h (/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/scipy/linalg/_blas_subroutines.h)
WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/mesonpy-2qdx8tuh/install/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/scipy/linalg/_lapack_subroutines.h (/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/scipy/linalg/_lapack_subroutines.h)
WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/mesonpy-2qdx8tuh/install/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/scipy/linalg/_blas_subroutine_wrappers.f (/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/scipy/linalg/_blas_subroutine_wrappers.f)
WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/mesonpy-2qdx8tuh/install/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/scipy/linalg/_lapack_subroutine_wrappers.f (/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/scipy/linalg/_lapack_subroutine_wrappers.f)
Copying files to wheel...
Traceback (most recent call last):x86_64-linux-gnu.so
  File "/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/pep517/in_process/_in_process.py", line 280, in <module>
    main()
  File "/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/pep517/in_process/_in_process.py", line 263, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/pep517/in_process/_in_process.py", line 204, in build_wheel
    return _build_backend().build_wheel(wheel_directory, config_settings,
  File "/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/mesonpy/__init__.py", line 780, in build_wheel
    return project.wheel(out).name
  File "/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/mesonpy/__init__.py", line 720, in wheel
    wheel = _WheelBuilder(self).build(self._install_plan, self._copy_files, self._build_dir)
  File "/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/mesonpy/__init__.py", line 315, in build
    self._install_file(whl, counter, origin, destination)
  File "/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/mesonpy/__init__.py", line 285, in _install_file
    assert elf.rpath in ('', '$ORIGIN/'), f'Unexpected RPATH in {origin}: {elf.rpath}'
AssertionError: Unexpected RPATH in /tmp/mesonpy-2qdx8tuh/build/mesonpy-wheel-libs/scipy/_lib/_ccallback_c.cpython-39-x86_64-linux-gnu.so: /home/rgommers/anaconda3/envs/scipy-meson/lib

ERROR Backend subproccess exited when trying to invoke build_wheel

The warnings come from specifying install_dir, e.g. for scipy/__config__.py:
generate_config = custom_target(
  'generate-config',
  install: true,
  build_always_stale: true,
  build_by_default: true,
  output: '__config__.py',
  input: '../tools/config_utils.py',
  command: [py3, '@INPUT@', '@OUTPUT@'],
  install_dir: py3.get_install_dir() + '/scipy'
)

Unless there's a way to add the purelib/platlib info in the meson.build file (which I don't think there is?), those warnings should not be shown. Side note: if heuristics are used, it seems odd that .py is mapped to platlib.
For the error:
  File "/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/mesonpy/__init__.py", line 285, in _install_file
    assert elf.rpath in ('', '$ORIGIN/'), f'Unexpected RPATH in {origin}: {elf.rpath}'
AssertionError: Unexpected RPATH in /tmp/mesonpy-2qdx8tuh/build/mesonpy-wheel-libs/scipy/_lib/_ccallback_c.cpython-39-x86_64-linux-gnu.so: /home/rgommers/anaconda3/envs/scipy-meson/lib

The path /home/rgommers/anaconda3/envs/scipy-meson/lib is indeed where libopenblas.so et al. are located.
Finally a question: how do I build in a directory that's not /tmp? In this case I wanted to debug, but the build output had already disappeared because it was living in a tmpdir. I don't see a flag in build that lets me do this.

IMO, I think we should probably be running it by default, so that user installs via sdist are less prone to breaking, but that does have drawbacks.
What do you want to do?

I think it's hard to determine what is less fragile. So far I haven't been able to produce a wheel yet because both auditwheel and mesonpy._elf gave exceptions. Rpath rewriting is not completely trivial, so more likely to break than just skipping that step.
It's also not clear to me what the rules are for including shared libraries. Let's take a concrete example from SciPy:
$ ldd build/scipy/linalg/_flinalg.cpython-39-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007ffc21290000)
        libopenblas.so.0 => /home/rgommers/anaconda3/envs/scipy-dev/lib/libopenblas.so.0 (0x00007ff0b3f6c000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007ff0b3d79000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007ff0b3c35000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007ff0b3c14000)
        libgfortran.so.5 => /home/rgommers/anaconda3/envs/scipy-dev/lib/./libgfortran.so.5 (0x00007ff0b3a6b000)
        /usr/lib64/ld-linux-x86-64.so.2 (0x00007ff0b5fb1000)
        libquadmath.so.0 => /home/rgommers/anaconda3/envs/scipy-dev/lib/././libquadmath.so.0 (0x00007ff0b3a2f000)
        libgcc_s.so.1 => /home/rgommers/anaconda3/envs/scipy-dev/lib/././libgcc_s.so.1 (0x00007ff0b3a1a000)

What is going to be included here? Does the answer change if, for example, libgfortran or libopenblas come from /usr/lib/?

I did a bit of testing. First, a trivial fix for this PR:

Ah, yes! I forgot to add the file to meson.build. We should add a build action in the CI. The pep621 should not be specified there, as it is not always necessary, and we put it in get_requires_for_* when it is.

Unless there's a way to add the purelib/platlib info in the meson.build file (which I don't think there is?), those warnings should not be shown.

We can work around this by using the new Meson tags feature, but that would be a dirty hack. Other possibility would be to add a setting in project.toml for users to clarify where the files should be placed.

Side note: if heuristics are used, it seems odd that .py is mapped to platlib.

.py does not mean it should necessarily go to purelib, it might still be architecture dependent (eg. a generated Python source file that depends on the architecture). If you have Meson file to install the file to platlib, the heuristics should not be putting it in purelib just because it is a .py. Because we can't tell where it was supposed to be installed, we put it in the safest place, which is platlib.

For the error:
  File "/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/mesonpy/__init__.py", line 285, in _install_file
    assert elf.rpath in ('', '$ORIGIN/'), f'Unexpected RPATH in {origin}: {elf.rpath}'
AssertionError: Unexpected RPATH in /tmp/mesonpy-2qdx8tuh/build/mesonpy-wheel-libs/scipy/_lib/_ccallback_c.cpython-39-x86_64-linux-gnu.so: /home/rgommers/anaconda3/envs/scipy-meson/lib

The path /home/rgommers/anaconda3/envs/scipy-meson/lib is indeed where libopenblas.so et al. are located.

Right, I have to look into this. The assert is there because I'd rather to blow up on unexpected values than to possibly silently introduce a bug for letting it work in a situation we did not take into account.
In this case, I think we might be taking the file from the build directory, rather from the dummy installation, which would be a bug on our part, so this would be a good catch. I have to look into it.

Finally a question: how do I build in a directory that's not /tmp? In this case I wanted to debug, but the build output had already disappeared because it was living in a tmpdir. I don't see a flag in build that lets me do this.

This is not yet documented, as I wanted to resolve the auditwheel situation and then document all config options, but you can pass a builddir option (eg. python -m build -Cbuilddir=build) to build from a local Meson build directory. This is extremely useful in large projects like SciPy, to cache the build.

So far I haven't been able to produce a wheel yet because both auditwheel and mesonpy._elf gave exceptions. Rpath rewriting is not completely trivial, so more likely to break than just skipping that step.

You can remove that assert if you want, it is not very problematic.

What is going to be included here?

Everything that is not provided by the manylinux tag should be included.

Does the answer change if, for example, libgfortran or libopenblas come from /usr/lib/?

As opposed to where?

EDIT: another question: what about name mangling and problems with duplication? If multiple packages pick up libopenblas, now I have multiple copies (e.g., import numpy; import scipy.linalg). So now I've created a huge problem even if name mangling was done correctly and I can load both copies - because I now have two copies of openblas, which means two threadpools instead of one, etc.
It still feels like something that's impossible to get right automatically.

Yes, that is one of the drawbacks of this approach. How problematic would that be? One possible solution would be excluding those symbols from name mangling, but this requires making sure both libraries have compatible ABIs, as far as those symbols are concerned at least.
I don't think we have any way around these sort of problems, as PyPI is just not fit to deal with native dependencies. One option would be focusing on improving the Conda ecosystem as an alternative, and have upstreams like SciPy build Conda packages (to be distributed directly or via a well-known channel like conda-forge).

You can remove that assert if you want, it is not very problematic.

Okay, I did that and then it built me a wheel. Unpacking it I run into:

Looking back in the build log, I do see:
WARNING Duplicate name: '_zeros.cpython-39-x86_64-linux-gnu.so'

A much bigger problem is that the vendoring code seems to do completely the wrong thing: it vendors the SciPy extension modules in the top-level code, and they've disappeared from their regular in-tree place:

External dependencies are not vendored as far as I can tell.

It still feels like something that's impossible to get right automatically.

Yes, that is one of the drawbacks of this approach. How problematic would that be? One possible solution would be excluding those symbols from name mangling, but this requires making sure both libraries have compatible ABIs, as far as those symbols are concerned at least.

I think that mesonpy should do the basic approach here: simply zip up the files that are installable, add correct metadata, and call it a day. This is what wheel does now, and yes it has gaps but in practice it mostly works. As we can see here the deeper problems aren't solvable, and this vendoring introduces obvious new problems that are significantly worse than the existing ones.
The main thing you're trying to solve here is that a wheel depends on locally installed libraries from the system, and then a system upgrade may break the installed package. This isn't all that likely though for many packages (e.g. BLAS/LAPACK have a stable ABI/API and actual breakage is very unlikely), and people by now know this may happen. And if it does, typically what happens is that import mypackage breaks. Which is much better than things like deadlocks at runtime because I have duplicate libraries being loaded.
If you want to keep vendoring, I think it should be opt-in. The use case of building a pure C library within the project itself and that library then needing to be vendored is a real use case, but for now there are no users who actually need it.

I don't think we have any way around these sort of problems, as PyPI is just not fit to deal with native dependencies.

Yes, that is true. I think we have to live with that for the time being.

One option would be focusing on improving the Conda ecosystem as an alternative, and have upstreams like SciPy build Conda packages (to be distributed directly or via a well-known channel like conda-forge).

We already do that (conda-forge is a first class citizen by now), but I don't think it's realistic to drop PyPI support for libraries like SciPy - there's too much demand for it. It is possible I think for packages with much more complex dependencies, like the geospatial stack.
And separately, yes we should work on the "PyPI interop with other package managers" to get a better handle on this stuff. So we can at least express dependencies that are now missing.

Looking back in the build log, I do see:

Uh, we had issues with that in the past, but I thought I had fix it, perhaps this is a regression. We definitely need a test case for this.

A much bigger problem is that the vendoring code seems to do completely the wrong thing: it vendors the SciPy extension modules in the top-level code, and they've disappeared from their regular in-tree place:

That is not the vendoring code, we are very likely just not using the correct path when we adding the files to the wheel, which is resulting in the files being added to the root. It only seems to be affecting one code path, the one I touched in this PR, so I likely just messed things up somewhere.

External dependencies are not vendored as far as I can tell.

Yes, they are not, only internal libraries, but that is required.

I think that mesonpy should do the basic approach here: simply zip up the files that are installable, add correct metadata, and call it a day. This is what wheel does now, and yes it has gaps but in practice it mostly works.

Then I don't think we should have an option to run auditwheel, the user can do that afterwards if they want.

As we can see here the deeper problems aren't solvable, and this vendoring introduces obvious new problems that are significantly worse than the existing ones.

The problems here are not due to vendoring, they are a bug in the wheel generation.

The main thing you're trying to solve here is that a wheel depends on locally installed libraries from the system, and then a system upgrade may break the installed package. This isn't all that likely though for many packages (e.g. BLAS/LAPACK have a stable ABI/API and actual breakage is very unlikely), and people by now know this may happen. And if it does, typically what happens is that import mypackage breaks. Which is much better than things like deadlocks at runtime because I have duplicate libraries being loaded.

Other thing to consider is that you will have external dependencies, while having them you can never be truly manylinux compatible, and pip install ... can result in broken modules if the user does not have the necessary libraries, which will of course fail with obscure errors (for a normal user at least).

Yes, they are not, only internal libraries, but that is required.

Okay, if it's internal only, then this should be a do-nothing operation for SciPy. We don't have internal libraries that are not Python extensions.

Yep, this is what a wheel that vendors a library looks like ATM:
$ unzip -l /tmp/mesonpy-test-deg32gtn/link_against_local_lib-1.0.0-cp310-cp310-linux_x86_64.whl
Archive:  /tmp/mesonpy-test-deg32gtn/link_against_local_lib-1.0.0-cp310-cp310-linux_x86_64.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
       65  2022-01-10 21:38   link_against_local_lib-1.0.0.dist-info/METADATA
       88  2022-01-10 21:38   link_against_local_lib-1.0.0.dist-info/WHEEL
    25216  2022-01-10 21:38   example.cpython-310-x86_64-linux-gnu.so
    17128  2022-01-10 21:38   .mesonpy.libs/libexample.so
      431  2022-01-10 21:38   link_against_local_lib-1.0.0.dist-info/RECORD
---------                     -------
    42928                     5 files

Other thing to consider is that you will have external dependencies, while having them you can never be truly manylinux compatible

We are manylinux compatible, because the PyPI wheels are carefully vendoring one specific version of libopenblas - something that cannot be done with general builds. It is unnecessary to be manylinux compatible for building from source, because you will be compatible with the system you're on. Redistributing locally built wheels to machines running other Linux distros is not a valid use case imho - in that case you should be more careful and use/reproduce our manylinux wheels.

I understand, but for local builds you lose the benefit of the binary wheels where you carefully vendor libopenblas, as things will break as soon as the user updates the system and OpenBLAS is updated.
But like I said, there is no perfect solution :shrug:

Uh, we had issues with that in the past, but I thought I had fix it, perhaps this is a regression. We definitely need a test case for this.

I remember now! Sorry, with all the confusion of christmas, new years, and the Conda plugin system implementation, this feel out of my mind. This is the issue that I attempted to fix in https://github.com/mesonbuild/meson/pull/9474.
Meson is returning the path of some modules without the subdirs (eg. {moduledir_shared}/_zeros.cpython-310-x86_64-linux-gnu.so), causing the file to be installed to the root of platlib. We can switch to install location matching heuristics in the meanwhile.
Can you try again?

You can remove that assert if you want, it is not very problematic.

I am still trying to understand exactly the situation here, so you will need to keep commenting it out or running Python with -O.
It seems to me that Meson is setting the RPATH to libdir, but I don't really understand why.

Tested again, after commenting out the problematic assert for rpath and using:
$ python -m build --wheel --no-isolation --skip-dependency-check -Cbuilddir=buildwheel
$ pip install dist/scipy-1.9.0.dev0-cp39-cp39-linux_x86_64.whl

The wheel now gets generated and the contents look about right - files are all in the right place according to some manual spot checking.
The import fails however with:
    from ._qhull import *
ImportError: libopenblas.so.0: cannot open shared object file: No such file or directory

Checking the extension that is installed into site-packages (coming from the wheel):
$ ldd _qhull.cpython-39-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007ffee8db7000)
        libopenblas.so.0 => not found
        libc.so.6 => /usr/lib/libc.so.6 (0x00007ff9b7060000)
        /usr/lib64/ld-linux-x86-64.so.2 (0x00007ff9b735b000)

Compare to the same extension in the wheel build dir:
$ cd buildwheel/scipy/spatial/
$ ldd _qhull.cpython-39-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007ffc90728000)
        libopenblas.so.0 => /home/rgommers/anaconda3/envs/scipy-meson/lib/libopenblas.so.0 (0x00007f48118b7000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f48116c4000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007f4811580000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f481155f000)
        libgfortran.so.5 => /home/rgommers/anaconda3/envs/scipy-meson/lib/./libgfortran.so.5 (0x00007f48113b6000)
        /usr/lib64/ld-linux-x86-64.so.2 (0x00007f4813984000)
        libquadmath.so.0 => /home/rgommers/anaconda3/envs/scipy-meson/lib/././libquadmath.so.0 (0x00007f481137a000)
        libgcc_s.so.1 => /home/rgommers/anaconda3/envs/scipy-meson/lib/././libgcc_s.so.1 (0x00007f4811365000)

The unmangled dependencies are fine, the mangled have somehow messed with libopenblas.so.0 so it's no longer found. The other changes are transitive dependencies of libopenblas.

Ah it's because of the RPATH in the .so's:
Correct:
$ readelf -d ../../../buildwheel/scipy/spatial/_qhull.cpython-39-x86_64-linux-gnu.so  | grep ibrary
 0x0000000000000001 (NEEDED)             Shared library: [libopenblas.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000f (RPATH)              Library rpath: [/home/rgommers/anaconda3/envs/scipy-meson/lib]

After rewriting:
$ readelf -d _qhull.cpython-39-x86_64-linux-gnu.so  | grep ibrary
 0x0000000000000001 (NEEDED)             Shared library: [libopenblas.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN/../../.scipy.mesonpy.libs]

Do you have the build environment enabled when trying to use the wheel? It need libopenblas.so.0 to be present on the system, if you are using some sort of isolated environment, LD_LIBRARY_PATH should be set to include the libs folder.

How does setuptools behave in this case?

Do you have the build environment enabled when trying to use the wheel?

Yes of course. The build env == the runtime env. It's an activated conda environment.

It need libopenblas.so.0 to be present on the system, if you are using some sort of isolated environment, LD_LIBRARY_PATH should be set to include the libs folder.

We should never have a need for LD_LIBRARY_PATH.  libopenblas.so.0 is present right where it was found during the build. mesonpy is simply rewriting the rpath here when it shouldn't be touching it.

How does setuptools/distutils behave in this case?

Let me check. Probably the same as Meson.

This is where libopenblas.so lives. It's an external dependency, so that seems correct.

But the linker should already search libdir by default, Meson shouldn't have to set the RPATH for this.
ldconfig -p | grep  libopenblas should be showing the library already if it is in libdir. Can you post the output of that command?

Rebuilding SciPy with distutils takes a while (doing that now), but checked a local NumPy build:
 0x000000000000000f (RPATH)              Library rpath: [/home/rgommers/anaconda3/envs/numpy-dev/lib]

And for completeness, conda-build:
0x000000000000000f (RPATH)              Library rpath: [$ORIGIN/../../../..]

But the linker should already search libdir by default, Meson shouldn't have to set the RPATH for this.

No, because we're using the linker from the operating system (see /usr/lib64/ld-linux-x86-64.so.2 output from ldd), so there is no way it is going to know about a libdir in a conda environment.

ldconfig -p | grep libopenblas should be showing the library already if it is in libdir. Can you post the output of that command?

$ which ldconfig
/usr/bin/ldconfig
$ ldconfig -p | grep libopenblas
# empty output

I understand. Can you post the Meson discovery log then?

meson-logs/meson-log.txt:

Build started at 2022-01-21T23:39:57.426955
Main binary: /home/rgommers/anaconda3/envs/scipy-meson/bin/python3.9
Build Options: -Dprefix=/home/rgommers/anaconda3/envs/scipy-meson '--native-file /home/rgommers/code/bldscipy/.mesonpy-native-file.ini'
Python system: Linux
The Meson build system
Version: 0.61.0.rc1
Source dir: /home/rgommers/code/bldscipy
Build dir: /home/rgommers/code/bldscipy/buildwheel
Build type: native build
Project name: SciPy
Project version: 1.9.0.dev0
Sanity testing C compiler: /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-cc
Is cross compiler: False.
Sanity check compiler command line: /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-cc sanitycheckc.c -o sanitycheckc.exe -Wno-unused-function -Wno-conversion -Wno-misleading-indentation -Wno-incompatible-pointer-types -D_FILE_OFFSET_BITS=64 -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/rgommers/anaconda3/envs/scipy-meson/lib -Wl,-rpath-link,/home/rgommers/anaconda3/envs/scipy-meson/lib -L/home/rgommers/anaconda3/envs/scipy-meson/lib
Sanity check compile stdout:

-----
Sanity check compile stderr:

-----
Running test binary command: /home/rgommers/code/bldscipy/buildwheel/meson-private/sanitycheckc.exe
C compiler for the host machine: /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-cc (gcc 9.4.0 "x86_64-conda-linux-gnu-cc (GCC) 9.4.0")
C linker for the host machine: /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-cc ld.bfd 2.36.1
Sanity testing C++ compiler: /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-c++
Is cross compiler: False.
Sanity check compiler command line: /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-c++ sanitycheckcpp.cc -o sanitycheckcpp.exe -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/rgommers/anaconda3/envs/scipy-meson/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/rgommers/anaconda3/envs/scipy-meson/include -D_FILE_OFFSET_BITS=64 -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/rgommers/anaconda3/envs/scipy-meson/lib -Wl,-rpath-link,/home/rgommers/anaconda3/envs/scipy-meson/lib -L/home/rgommers/anaconda3/envs/scipy-meson/lib
Sanity check compile stdout:

-----
Sanity check compile stderr:

-----
Running test binary command: /home/rgommers/code/bldscipy/buildwheel/meson-private/sanitycheckcpp.exe
C++ compiler for the host machine: /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-c++ (gcc 9.4.0 "x86_64-conda-linux-gnu-c++ (GCC) 9.4.0")
C++ linker for the host machine: /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-c++ ld.bfd 2.36.1
Running compile:
Working directory:  /tmp/tmpzg1r2zdx
Command line:  cython /tmp/tmpzg1r2zdx/testfile.pyx -o /tmp/tmpzg1r2zdx/output.exe --fast-fail

Code:
 print("hello world")
Compiler stdout:

Compiler stderr:
 /home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/tmpzg1r2zdx/testfile.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)

Cython compiler for the host machine: cython (cython 0.29.26)
Sanity testing C compiler: /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-cc
Is cross compiler: False.
Sanity check compiler command line: /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-cc sanitycheckc.c -o sanitycheckc.exe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/rgommers/anaconda3/envs/scipy-meson/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/rgommers/anaconda3/envs/scipy-meson/include -D_FILE_OFFSET_BITS=64 -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/rgommers/anaconda3/envs/scipy-meson/lib -Wl,-rpath-link,/home/rgommers/anaconda3/envs/scipy-meson/lib -L/home/rgommers/anaconda3/envs/scipy-meson/lib
Sanity check compile stdout:

-----
Sanity check compile stderr:

-----
Running test binary command: /home/rgommers/code/bldscipy/buildwheel/meson-private/sanitycheckc.exe
C compiler for the build machine: /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-cc (gcc 9.4.0 "x86_64-conda-linux-gnu-cc (GCC) 9.4.0")
C linker for the build machine: /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-cc ld.bfd 2.36.1
Sanity testing C++ compiler: /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-c++
Is cross compiler: False.
Sanity check compiler command line: /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-c++ sanitycheckcpp.cc -o sanitycheckcpp.exe -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/rgommers/anaconda3/envs/scipy-meson/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/rgommers/anaconda3/envs/scipy-meson/include -D_FILE_OFFSET_BITS=64 -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/rgommers/anaconda3/envs/scipy-meson/lib -Wl,-rpath-link,/home/rgommers/anaconda3/envs/scipy-meson/lib -L/home/rgommers/anaconda3/envs/scipy-meson/lib
Sanity check compile stdout:

-----
Sanity check compile stderr:

-----
Running test binary command: /home/rgommers/code/bldscipy/buildwheel/meson-private/sanitycheckcpp.exe
C++ compiler for the build machine: /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-c++ (gcc 9.4.0 "x86_64-conda-linux-gnu-c++ (GCC) 9.4.0")
C++ linker for the build machine: /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-c++ ld.bfd 2.36.1
Using cached compile:
Cached command line:

Code:
 print("hello world")
Cached compiler stdout:

Cached compiler stderr:
 /home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/tmpzg1r2zdx/testfile.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)

Cython compiler for the build machine: cython (cython 0.29.26)
Build machine cpu family: x86_64
Build machine cpu: x86_64
Host machine cpu family: x86_64
Host machine cpu: x86_64
Target machine cpu family: x86_64
Target machine cpu: x86_64
Running compile:
Working directory:  /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpamas4px5
Command line:  /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-cc /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpamas4px5/testfile.c -o /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpamas4px5/output.obj -c -Wno-unused-function -Wno-conversion -Wno-misleading-indentation -Wno-incompatible-pointer-types -D_FILE_OFFSET_BITS=64 -O0 -Wunused-but-set-variable -Wno-unused-but-set-variable

Code:
 extern int i;
int i;

Compiler stdout:

Compiler stderr:

Compiler for C supports arguments -Wno-unused-but-set-variable: YES
Fortran compiler for the host machine: /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-gfortran (gcc 9.4.0 "GNU Fortran (GCC) 9.4.0")
Fortran linker for the host machine: /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-gfortran ld.bfd 2.36.1
Program cython found: YES (/home/rgommers/anaconda3/envs/scipy-meson/bin/cython)
Program pythran found: YES (/home/rgommers/anaconda3/envs/scipy-meson/bin/pythran)
Program cp found: YES (/usr/bin/cp)
Program python3 found: YES (/home/rgommers/anaconda3/envs/scipy-meson/bin/python3.9)
Pkg-config binary for 1 is not cached.
Pkg-config binary missing from cross or native file, or env var undefined.
Trying a default Pkg-config fallback at pkg-config
Found pkg-config: /home/rgommers/anaconda3/envs/scipy-meson/bin/pkg-config (0.29.2)
Determining dependency 'python-3.9' with pkg-config executable '/home/rgommers/anaconda3/envs/scipy-meson/bin/pkg-config'
env[PKG_CONFIG_LIBDIR]: /home/rgommers/anaconda3/envs/scipy-meson/lib/pkgconfig
env[PKG_CONFIG_PATH]:
Called `/home/rgommers/anaconda3/envs/scipy-meson/bin/pkg-config --modversion python-3.9` -> 0
3.9
env[PKG_CONFIG_LIBDIR]: /home/rgommers/anaconda3/envs/scipy-meson/lib/pkgconfig
env[PKG_CONFIG_PATH]:
Called `/home/rgommers/anaconda3/envs/scipy-meson/bin/pkg-config --cflags python-3.9` -> 0
-I/home/rgommers/anaconda3/envs/scipy-meson/include/python3.9
env[PKG_CONFIG_LIBDIR]: /home/rgommers/anaconda3/envs/scipy-meson/lib/pkgconfig
env[PKG_CONFIG_ALLOW_SYSTEM_LIBS]: 1
env[PKG_CONFIG_PATH]:
Called `/home/rgommers/anaconda3/envs/scipy-meson/bin/pkg-config --libs python-3.9` -> 0

env[PKG_CONFIG_LIBDIR]: /home/rgommers/anaconda3/envs/scipy-meson/lib/pkgconfig
env[PKG_CONFIG_PATH]:
Called `/home/rgommers/anaconda3/envs/scipy-meson/bin/pkg-config --libs python-3.9` -> 0

Running command: /home/rgommers/anaconda3/envs/scipy-meson/bin/python3.9 -c import os; os.chdir(".."); import numpy; print(numpy.get_include())
--- stdout ---
/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/numpy/core/include

--- stderr ---

Running compile:
Working directory:  /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpr08gmfds
Command line:  /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-cc -L/home/rgommers/anaconda3/envs/scipy-meson/lib /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpr08gmfds/testfile.c -o /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpr08gmfds/output.exe -Wno-unused-function -Wno-conversion -Wno-misleading-indentation -Wno-incompatible-pointer-types -D_FILE_OFFSET_BITS=64 -O0 -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/rgommers/anaconda3/envs/scipy-meson/lib -Wl,-rpath-link,/home/rgommers/anaconda3/envs/scipy-meson/lib

Code:
 #include<stdio.h>

        int main(void) {
            printf("%ld\n", (long)(sizeof(void *)));
            return 0;
        }
Compiler stdout:

Compiler stderr:

Program stdout:

8

Program stderr:

Running compile:
Working directory:  /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpivvcaske
Command line:  /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-cc /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpivvcaske/testfile.c -o /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpivvcaske/output.obj -c -Wno-unused-function -Wno-conversion -Wno-misleading-indentation -Wno-incompatible-pointer-types -D_FILE_OFFSET_BITS=64 -O0 --print-search-dirs

Code:

Compiler stdout:
 install: /home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/
programs: =/home/rgommers/anaconda3/envs/scipy-meson/bin/../libexec/gcc/x86_64-conda-linux-gnu/9.4.0/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../libexec/gcc/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../../x86_64-conda-linux-gnu/bin/x86_64-conda-linux-gnu/9.4.0/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../../x86_64-conda-linux-gnu/bin/
libraries: =/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../../x86_64-conda-linux-gnu/lib/x86_64-conda-linux-gnu/9.4.0/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../../x86_64-conda-linux-gnu/lib/../lib/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../x86_64-conda-linux-gnu/9.4.0/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../../lib/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../x86_64-conda-linux-gnu/sysroot/lib/x86_64-conda-linux-gnu/9.4.0/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../x86_64-conda-linux-gnu/sysroot/lib/../lib/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/x86_64-conda-linux-gnu/9.4.0/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/../lib/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../../x86_64-conda-linux-gnu/lib/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../:/home/rgommers/anaconda3/envs/scipy-meson/bin/../x86_64-conda-linux-gnu/sysroot/lib/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/

Compiler stderr:

Library npymath found: YES
Library npyrandom found: YES
Running command: /home/rgommers/anaconda3/envs/scipy-meson/bin/python3.9 -c import pybind11; print(pybind11.get_include())
--- stdout ---
/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/pybind11/include

--- stderr ---

Running command: /home/rgommers/anaconda3/envs/scipy-meson/bin/python3.9 -c import os; print(os.environ.get("SCIPY_USE_PYTHRAN", 1))
--- stdout ---
1

--- stderr ---

Running command: /home/rgommers/anaconda3/envs/scipy-meson/bin/python3.9 -c import os; os.chdir(".."); import pythran; print(os.path.dirname(pythran.__file__));
--- stdout ---
/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/pythran

--- stderr ---

Pkg-config binary for 1 is cached.
Determining dependency 'openblas' with pkg-config executable '/home/rgommers/anaconda3/envs/scipy-meson/bin/pkg-config'
env[PKG_CONFIG_PATH]:
Called `/home/rgommers/anaconda3/envs/scipy-meson/bin/pkg-config --modversion openblas` -> 0
0.3.12
env[PKG_CONFIG_PATH]:
Called `/home/rgommers/anaconda3/envs/scipy-meson/bin/pkg-config --cflags openblas` -> 0
-I/home/rgommers/anaconda3/envs/scipy-meson/include
env[PKG_CONFIG_ALLOW_SYSTEM_LIBS]: 1
env[PKG_CONFIG_PATH]:
Called `/home/rgommers/anaconda3/envs/scipy-meson/bin/pkg-config --libs openblas` -> 0
-L/home/rgommers/anaconda3/envs/scipy-meson/lib -lopenblas
env[PKG_CONFIG_PATH]:
Called `/home/rgommers/anaconda3/envs/scipy-meson/bin/pkg-config --libs openblas` -> 0
-L/home/rgommers/anaconda3/envs/scipy-meson/lib -lopenblas
Running compile:
Working directory:  /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpbdycvgq0
Command line:  /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-c++ -L/home/rgommers/anaconda3/envs/scipy-meson/lib /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpbdycvgq0/testfile.cpp -o /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpbdycvgq0/output.exe -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/rgommers/anaconda3/envs/scipy-meson/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/rgommers/anaconda3/envs/scipy-meson/include -D_FILE_OFFSET_BITS=64 -O0 -fpermissive -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/rgommers/anaconda3/envs/scipy-meson/lib -Wl,-rpath-link,/home/rgommers/anaconda3/envs/scipy-meson/lib

Code:
 #include<stdio.h>

        int main(void) {
            printf("%ld\n", (long)(sizeof(void *)));
            return 0;
        }
Compiler stdout:

Compiler stderr:

Program stdout:

8

Program stderr:

Running compile:
Working directory:  /home/rgommers/code/bldscipy/buildwheel/meson-private/tmp31bj7jvz
Command line:  /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-c++ /home/rgommers/code/bldscipy/buildwheel/meson-private/tmp31bj7jvz/testfile.cpp -o /home/rgommers/code/bldscipy/buildwheel/meson-private/tmp31bj7jvz/output.obj -c -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/rgommers/anaconda3/envs/scipy-meson/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/rgommers/anaconda3/envs/scipy-meson/include -D_FILE_OFFSET_BITS=64 -O0 -fpermissive --print-search-dirs

Code:

Compiler stdout:
 install: /home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/
programs: =/home/rgommers/anaconda3/envs/scipy-meson/bin/../libexec/gcc/x86_64-conda-linux-gnu/9.4.0/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../libexec/gcc/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../../x86_64-conda-linux-gnu/bin/x86_64-conda-linux-gnu/9.4.0/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../../x86_64-conda-linux-gnu/bin/
libraries: =/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../../x86_64-conda-linux-gnu/lib/x86_64-conda-linux-gnu/9.4.0/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../../x86_64-conda-linux-gnu/lib/../lib/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../x86_64-conda-linux-gnu/9.4.0/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../../lib/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../x86_64-conda-linux-gnu/sysroot/lib/x86_64-conda-linux-gnu/9.4.0/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../x86_64-conda-linux-gnu/sysroot/lib/../lib/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/x86_64-conda-linux-gnu/9.4.0/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/../lib/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../../x86_64-conda-linux-gnu/lib/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../:/home/rgommers/anaconda3/envs/scipy-meson/bin/../x86_64-conda-linux-gnu/sysroot/lib/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/

Compiler stderr:

Run-time dependency openblas found: YES 0.3.12
Dependency openblas found: YES 0.3.12 (cached)
Running compile:
Working directory:  /home/rgommers/code/bldscipy/buildwheel/meson-private/tmp1otdpurn
Command line:  /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-cc -L/home/rgommers/anaconda3/envs/scipy-meson/lib /home/rgommers/code/bldscipy/buildwheel/meson-private/tmp1otdpurn/testfile.c -o /home/rgommers/code/bldscipy/buildwheel/meson-private/tmp1otdpurn/output.exe -Wno-unused-function -Wno-conversion -Wno-misleading-indentation -Wno-incompatible-pointer-types -D_FILE_OFFSET_BITS=64 -O0 -std=c99 -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/rgommers/anaconda3/envs/scipy-meson/lib -Wl,-rpath-link,/home/rgommers/anaconda3/envs/scipy-meson/lib

Code:
 #include <stdio.h>
#include <limits.h>

        #if defined __stub_open_memstream || defined __stub___open_memstream
        fail fail fail this function is not going to work
        #endif

int main(void) {
            void *a = (void*) &open_memstream;
            long long b = (long long) a;
            return (int) b;
        }
Compiler stdout:

Compiler stderr:
 /home/rgommers/code/bldscipy/buildwheel/meson-private/tmp1otdpurn/testfile.c: In function 'main':
/home/rgommers/code/bldscipy/buildwheel/meson-private/tmp1otdpurn/testfile.c:9:32: error: 'open_memstream' undeclared (first use in this function)
    9 |             void *a = (void*) &open_memstream;
      |                                ^~~~~~~~~~~~~~
/home/rgommers/code/bldscipy/buildwheel/meson-private/tmp1otdpurn/testfile.c:9:32: note: each undeclared identifier is reported only once for each function it appears in

Running compile:
Working directory:  /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpqkgh33b_
Command line:  /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-cc -L/home/rgommers/anaconda3/envs/scipy-meson/lib /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpqkgh33b_/testfile.c -o /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpqkgh33b_/output.exe -Wno-unused-function -Wno-conversion -Wno-misleading-indentation -Wno-incompatible-pointer-types -D_FILE_OFFSET_BITS=64 -O0 -std=c99 -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/rgommers/anaconda3/envs/scipy-meson/lib -Wl,-rpath-link,/home/rgommers/anaconda3/envs/scipy-meson/lib

Code:
 #include <stdio.h>
        int main(void) {

        /* With some toolchains (MSYS2/mingw for example) the compiler
         * provides various builtins which are not really implemented and
         * fall back to the stdlib where they aren't provided and fail at
         * build/link time. In case the user provides a header, including
         * the header didn't lead to the function being defined, and the
         * function we are checking isn't a builtin itself we assume the
         * builtin is not functional and we just error out. */
        #if !0 && !defined(open_memstream) && !0
            #error "No definition for __builtin_open_memstream found in the prefix"
        #endif

        #ifdef __has_builtin
            #if !__has_builtin(__builtin_open_memstream)
                #error "__builtin_open_memstream not found"
            #endif
        #elif ! defined(open_memstream)
            __builtin_open_memstream;
        #endif
        return 0;
        }
Compiler stdout:

Compiler stderr:
 /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpqkgh33b_/testfile.c: In function 'main':
/home/rgommers/code/bldscipy/buildwheel/meson-private/tmpqkgh33b_/testfile.c:12:14: error: #error "No definition for __builtin_open_memstream found in the prefix"
   12 |             #error "No definition for __builtin_open_memstream found in the prefix"
      |              ^~~~~
/home/rgommers/code/bldscipy/buildwheel/meson-private/tmpqkgh33b_/testfile.c:20:13: error: '__builtin_open_memstream' undeclared (first use in this function)
   20 |             __builtin_open_memstream;
      |             ^~~~~~~~~~~~~~~~~~~~~~~~
/home/rgommers/code/bldscipy/buildwheel/meson-private/tmpqkgh33b_/testfile.c:20:13: note: each undeclared identifier is reported only once for each function it appears in

Checking for function "open_memstream" : NO
Configuring messagestream_config.h using configuration
Running compile:
Working directory:  /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpao65w5ub
Command line:  /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-c++ /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpao65w5ub/testfile.cpp -o /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpao65w5ub/output.obj -c -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/rgommers/anaconda3/envs/scipy-meson/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/rgommers/anaconda3/envs/scipy-meson/include -D_FILE_OFFSET_BITS=64 -O0 -fpermissive -fvisibility=hidden

Code:
 extern int i;
int i;

Compiler stdout:

Compiler stderr:

Compiler for C++ supports arguments -fvisibility=hidden: YES
Running compile:
Working directory:  /tmp/tmp3ekh9d9x
Command line:  /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-c++ /tmp/tmp3ekh9d9x/testfile.cpp -o /tmp/tmp3ekh9d9x/output.obj -D_FILE_OFFSET_BITS=64 -c -std=c++14

Code:
 int i = static_cast<int>(0);
Compiler stdout:

Compiler stderr:

Compiler accepts -std=c++14: YES
Running compile:
Working directory:  /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpxtjnop1j
Command line:  /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-c++ -L/home/rgommers/anaconda3/envs/scipy-meson/lib /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpxtjnop1j/testfile.cpp -o /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpxtjnop1j/output.exe -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/rgommers/anaconda3/envs/scipy-meson/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/rgommers/anaconda3/envs/scipy-meson/include -D_FILE_OFFSET_BITS=64 -O0 -fpermissive -std=c++14 -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/rgommers/anaconda3/envs/scipy-meson/lib -Wl,-rpath-link,/home/rgommers/anaconda3/envs/scipy-meson/lib

Code:
 #include<stdio.h>

        int main(void) {
            printf("%ld\n", (long)(sizeof(void*)));
            return 0;
        }
Compiler stdout:

Compiler stderr:

Program stdout:

8

Program stderr:

Checking for size of "void*" : 8
Run-time dependency threads found: YES
Running compile:
Working directory:  /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpbbfx2o5f
Command line:  /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-c++ -L/home/rgommers/anaconda3/envs/scipy-meson/lib /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpbbfx2o5f/testfile.cpp -o /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpbbfx2o5f/output.exe -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/rgommers/anaconda3/envs/scipy-meson/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/rgommers/anaconda3/envs/scipy-meson/include -D_FILE_OFFSET_BITS=64 -O0 -fpermissive -std=c++14 -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/rgommers/anaconda3/envs/scipy-meson/lib -Wl,-rpath-link,/home/rgommers/anaconda3/envs/scipy-meson/lib

Code:
 #include<stdio.h>

        int main(void) {
            printf("%ld\n", (long)(sizeof(void*)));
            return 0;
        }
Compiler stdout:

Compiler stderr:

Program stdout:

8

Program stderr:

Checking for size of "void*" : 8
Dependency threads found: YES unknown (cached)
Using cached compile:
Cached command line:

Code:
 extern int i;
int i;

Cached compiler stdout:

Cached compiler stderr:

Compiler for C++ supports arguments -fvisibility=hidden: YES (cached)
Using cached compile:
Cached command line:

Code:
 extern int i;
int i;

Cached compiler stdout:

Cached compiler stderr:

Compiler for C supports arguments -Wno-unused-but-set-variable: YES (cached)
Dependency openblas found: YES 0.3.12 (cached)
Using cached compile:
Cached command line:

Code:
 extern int i;
int i;

Cached compiler stdout:

Cached compiler stderr:

Compiler for C supports arguments -Wno-unused-but-set-variable: YES (cached)
Running compile:
Working directory:  /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpiy5b7yld
Command line:  /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-c++ /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpiy5b7yld/testfile.cpp -o /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpiy5b7yld/output.obj -c -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/rgommers/anaconda3/envs/scipy-meson/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/rgommers/anaconda3/envs/scipy-meson/include -D_FILE_OFFSET_BITS=64 -O0 -fpermissive -Wformat-truncation -Wno-format-truncation

Code:
 extern int i;
int i;

Compiler stdout:

Compiler stderr:

Compiler for C++ supports arguments -Wno-format-truncation: YES
Running compile:
Working directory:  /home/rgommers/code/bldscipy/buildwheel/meson-private/tmp38z0eyqh
Command line:  /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-c++ /home/rgommers/code/bldscipy/buildwheel/meson-private/tmp38z0eyqh/testfile.cpp -o /home/rgommers/code/bldscipy/buildwheel/meson-private/tmp38z0eyqh/output.obj -c -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/rgommers/anaconda3/envs/scipy-meson/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/rgommers/anaconda3/envs/scipy-meson/include -D_FILE_OFFSET_BITS=64 -O0 -fpermissive -Wclass-memaccess -Wno-class-memaccess

Code:
 extern int i;
int i;

Compiler stdout:

Compiler stderr:

Compiler for C++ supports arguments -Wno-class-memaccess: YES
Build targets in project: 204

SciPy 1.9.0.dev0

  User defined options
    Native files: /home/rgommers/code/bldscipy/.mesonpy-native-file.ini
    prefix      : /home/rgommers/anaconda3/envs/scipy-meson

Found ninja-1.10.2.git.kitware.jobserver-1 at /home/rgommers/anaconda3/envs/scipy-meson/bin/ninja
Running compile:
Working directory:  /tmp/tmpoynv8q1p
Command line:  /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-c++ /tmp/tmpoynv8q1p/testfile.cpp -o /tmp/tmpoynv8q1p/output.obj -D_FILE_OFFSET_BITS=64 -c -std=c++14

Code:
 int i = static_cast<int>(0);
Compiler stdout:

Compiler stderr:

Compiler accepts -std=c++14: YES
Running compile:
Working directory:  /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpjf17_g59
Command line:  /home/rgommers/anaconda3/envs/scipy-meson/bin/x86_64-conda-linux-gnu-gfortran /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpjf17_g59/testfile.f90 -o /home/rgommers/code/bldscipy/buildwheel/meson-private/tmpjf17_g59/output.obj -c -Wno-conversion -D_FILE_OFFSET_BITS=64 -O0 --print-search-dirs

Code:

Compiler stdout:
 install: /home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/
programs: =/home/rgommers/anaconda3/envs/scipy-meson/bin/../libexec/gcc/x86_64-conda-linux-gnu/9.4.0/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../libexec/gcc/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../../x86_64-conda-linux-gnu/bin/x86_64-conda-linux-gnu/9.4.0/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../../x86_64-conda-linux-gnu/bin/
libraries: =/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../../x86_64-conda-linux-gnu/lib/x86_64-conda-linux-gnu/9.4.0/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../../x86_64-conda-linux-gnu/lib/../lib/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../x86_64-conda-linux-gnu/9.4.0/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../../lib/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../x86_64-conda-linux-gnu/sysroot/lib/x86_64-conda-linux-gnu/9.4.0/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../x86_64-conda-linux-gnu/sysroot/lib/../lib/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/x86_64-conda-linux-gnu/9.4.0/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/../lib/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../../x86_64-conda-linux-gnu/lib/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../:/home/rgommers/anaconda3/envs/scipy-meson/bin/../x86_64-conda-linux-gnu/sysroot/lib/:/home/rgommers/anaconda3/envs/scipy-meson/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/

Compiler stderr:
 ]

I fixed up commits to handle this situation, can you reset to the remote and try again?

Wheels build this way will not be reproducible, or rather, will be less reproducible, so I'd recommend avoiding this approach when building wheels for distribution, but it should be suitable for development.

Sure, for distribution one should always take care to build manylinux compliant wheels. And same for other OSes, need to make things relocatable.

Argh, why is pip dumb, first it refuses and then it overwrites a conda-installed numpy:
$ pip install scipy-1.9.0.dev0-cp39-cp39-linux_x86_64.whl
Processing ./scipy-1.9.0.dev0-cp39-cp39-linux_x86_64.whl
Requirement already satisfied: numpy>=1.18.5 in /home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages (from scipy==1.9.0.dev0) (1.21.5)
scipy is already installed with the same version as the provided wheel. Use --force-reinstall to force an installation of the wheel.

(scipy-meson) rgommers@machine:~/code/bldscipy/dist (test-mesonpy)$ pip install scipy-1.9.0.dev0-cp39-cp39-linux_x86_64.whl --force-reinstall
Processing ./scipy-1.9.0.dev0-cp39-cp39-linux_x86_64.whl
Collecting numpy>=1.18.5
  Downloading numpy-1.22.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.8 MB)
     |████████████████████████████████| 16.8 MB 4.4 MB/s
Installing collected packages: numpy, scipy
  Attempting uninstall: numpy
    Found existing installation: numpy 1.21.5
    Uninstalling numpy-1.21.5:
      Successfully uninstalled numpy-1.21.5
  Attempting uninstall: scipy
    Found existing installation: scipy 1.9.0.dev0
    Uninstalling scipy-1.9.0.dev0:
      Successfully uninstalled scipy-1.9.0.dev0
Successfully installed numpy-1.22.1 scipy-1.9.0.dev0

Test collection starting without errors, so looks promising. Let's see if the test suite gets to the finish line.

All tests passed 🎉
That's enough for tonight:) Thanks Filipe!

Finally tracked down the origin of the CI fail on Python 3.7 :sweat_smile:

Have you thought about how to make this configurable? We established that there are valid reasons to do this bundling, and also valid reasons to skip it (xref #20 (comment)).

I think I read the whole thread, and please forgive me if I have missed something, but now that this is merged and released, how do I build a SciPy wheel without the vendoring? Vendoring libopenblas in the openSUSE RPM package is not an option. Yet I want to use meson-python so that I get the proper .dist-info metadata.
Right now, I get tons of rpmlint errors like this:
[ 1412s] python310-scipy.x86_64: E: binary-or-shlib-defines-rpath (Badness: 10000) /usr/lib64/python3.10/site-packages/scipy/_lib/_ccallback_c.cpython-310-x86_64-linux-gnu.so (RUNPATH: :$ORIGIN/../../.scipy.mesonpy.libs)

I think we can consider that a bug. Setting RPATH like that is needed only in case there's a project-internal shared library. In this case, RPATH is still set unconditionally on Linux.
I would say this is covered by gh-125. Except I didn't know it caused actual failures in Linux packaging. I'll comment there. Thanks @bnavigator

Vendoring libopenblas in the openSUSE RPM package is not an option.

Note that that is not what is actually happening right now. The RPATH entry just points at a non-existing directory.


The performance of the model

I followed the example code to train hypernerf scenes such as virg/broom, but the training results are not as expected. Why is that?

The model doesn't converge.

I have tried multiple environments, including py37+torch1.31+cu11.6 and py38+torch2.01+cu11.8. However, the final results are all not satisfactory.

I have tried multiple environments, including py37+torch1.31+cu11.6 and py38+torch2.01+cu11.8. However, the final results are all not satisfactory.

It looks very strange.
Can you show me the point clouds generated by colmap? It may look like:

I have tried multiple environments, including py37+torch1.31+cu11.6 and py38+torch2.01+cu11.8. However, the final results are all not satisfactory.

It looks very strange. Can you show me the point clouds generated by colmap? It may look like:
which is previewed by visual studio code.

This is the point clouds generated by colmap.

This is the point clouds generated by the model.

It seems that the original 3D Gaussians do not fit well.
In my test, after 3000 coarse iterations, the training PSNR will be up to 20. Have you tested in any other scenes?


Image Classification Python Task Implementation

Added the implementation for the image classifier and simple test to try out the API

I've added support to accept an roi (region of interest) parameter to the classify and classify_for_video APIs now.

Looped in Maxime to confirm the namespace, options, I/O containers, etc.


Fix safety issues with baseName and dirName

If you're dealing with types which are convertible to string, they must be converted at the call point, or if any portion of the string is returned by the function, you're returning a slice of a memory local to the function, which is obviously very bad. I don't know if -dip1000 can catch all such cases yet, but it at least did with baseName once I added some additional tests.
In any case, isConvertibleToString is inherently unsafe because of this problem, and we need to excise it from Phobos, but first we need to fix it so that nothing in Phobos uses it anymore. std.path also has BaseOf that it uses in conjunction with StringTypeOf to essentially do what isConvertibleToString does. So, BaseOf and all uses of it need to go too.
This PR just fixes baseName and dirName in order to keep the changes to review minimal, but it's a start. The other changes will still need to be done in other PRs.

Does this mean that baseName will now no longer accept a struct that converts to a string?

Does this mean that baseName will now no longer accept a struct that converts to a string?

No, it will. As far as I can determine, the only way to templatize a function and have it accept types that implicitly convert to string and have the conversion occur at the call site (which needs to happen for it to be safe for stuff like static arrays) is to have an overload which accepts strings and templatizes on the character type. So, for a function like baseName and dirName to accept both ranges of characters and types that implicitly convert to string, you end up if two overloads (one for strings and one for ranges of characters that aren't strings) which then call a helper function which then only operates on ranges. All of the implicit conversions are then safely done at the call site.

Ah, I think I understand your point now. Thanks!

Auto-merge toggled on


Allow automerge for bridge updates

Do we have a PoC trial workflow for this? If yes and behavior is expected, LGTM.


feat: delete Deployments tab, module tree uses old deployments components

Removes Deployments tab
Removes all the extraneous page headers (https://github.com/TBD54566975/ftl/issues/2490)
Fixes all the routing to point to /modules/{moduleName}/... instead of /deployments/{deploymentKey}/...
Minor cleanup of deployment components' internal logic after replacing deploymentKey with moduleName in the outermost interface

https://github.com/user-attachments/assets/ad785b7c-3e88-4d8d-8edb-b14bed441c98

So nice to be able to reuse all this for now! Looks good!!
This does break the scrolling in the treeview/page so we should fix that first then I'm good to merge 👍
[Screenshots redacted]

Rebasing since the scrollability PR merged seemed to fix it. Thanks for catching that!


Add bot_type field.

Automated message from Dropbox CLA bot
@TomaszKolek, it looks like you've already signed the Dropbox CLA. Thanks!

@timabbott
The hardest task I've made for zulip.
I'd done lot of things, broke something and so on...
So eventually I created new PR.
Previous one: #761

Why does the last commit delete various tests in test_hooks.py?

I merged the first commit with a bit of extra text in the commit message, since that one looks great!
  Also will post a few inline comments.

@timabbott It shoudn't happen probably something went wrong during resolving conflicts. I will fix it asap

@timabbott
I added notes you mentioned. Please take a look.

@TomaszKolek can you rebase this to resolve some merge conflicts?  I should have time to look at this again in detail tomorrow (and if not then, the next day)

@TomaszKolek I merged the first two commits (with a small tweak to remove an extra blank line that was deleted in one of them), since they looked good.
Let me know when you have a new version of the rest ready!

@timabbott  Whoa!

Posted a few small comments, otherwise looks good!

@timabbott Comments added

Sorry for the slow review; this looks great, merged!  Thanks @TomaszKolek for building this!
Next I guess we should design a UI for actually creating bots with this property.


Fix cmesh argument wrongly typed as Cint.

Closes https://github.com/DLR-AMR/T8code.jl/issues/47

Something went wrong here. Gonna redo.


Unable to push packages anonymously

I've created an empty ASP.NET application, added your NuGet package and deployed it to IIS.
The service works fine, available packages will be listed using nuget.exe list.
But when I'm trying to push a new package using NuGet.exe push "..\bin\Packages\CHG.iCLASSFW.Core.<IP_ADDRESS>.nupkg" -s http://bobthebuilder/packages2/ mykey
the service requests for a username and password.:
Please provide credentials for: http://bobthebuilder/packages2/api/odata/
I've already set the app setting <add key="NuGet.Lucene.Web:allowAnonymousPackageChanges" value="true"/>.
What I'm doing wrong?

Which version of NuGet.Lucene are you using?

The latest 2.9.4.

I've downloaded the source code. I guess you've rewritten some parts. Owin, Autofac etc.
Can you give me short introduction how to setup an IIS website with the new code?

Problem solved.
I've defined my global API key with "apiKey" app setting instead of "localAdministratorApiKey". So the service did not recognized my key and returned a HttpStatusCode.Unauthorized with the phrase "Invalid API-Key".
Unfortunately the nuget client won't show/handle this error correctly.

Glad you got it working. The intended design is that when allowAnonymousPackageChanges is true that no api key is needed at all. Did you get that to work correctly or did you work around it by setting localAdministratorApiKey?
To answer your other question, to use the NuGet.Lucene.Web 3.0.0-beta packages, you would create an empty asp.net application, and create a OWIN Startup class that subclasses the one provided in NuGet.Lucene.Web. There's a sample OWIN self-hosted project at https://github.com/themotleyfool/NuGet.Lucene/tree/master/source/NuGet.Lucene.Web.OwinHost.Sample, and Klondike is another example of how to embed this library in a web project.

I've worked around it by setting the localAdministratorApiKey.

Am 28.04.2015 um 16:27 schrieb Chris Eldredge<EMAIL_ADDRESS>Glad you got it working. The intended design is that when allowAnonymousPackageChanges is true that no api key is needed at all. Did you get that to work correctly or did you work around it by setting localAdministratorApiKey?
To answer your other question, to use the NuGet.Lucene.Web 3.0.0-beta packages, you would create an empty asp.net application, and create a OWIN Startup class that subclasses the one provided in NuGet.Lucene.Web. There's a sample OWIN self-hosted project at https://github.com/themotleyfool/NuGet.Lucene/tree/master/source/NuGet.Lucene.Web.OwinHost.Sample, and Klondike is another example of how to embed this library in a web project.
—
Reply to this email directly or view it on GitHub.


Given class A, Class<A> subtyping isn't analogous to A subtyping

This smells kinda like #814.
class A {}
class B extends A {}

var ap: A = new A();
var app: A = new B(); // Okay: instance of B <: instance of A
var Ap: Class<A> = A;
var App: Class<A> = B; // NG?: statics of B <: statics of A

/*
someFile.js:1:7,7: A
This type is incompatible with
someFile.js:7:21,21: B

Found 1 error
*/

With ES6 class statics chaining up to Function parallel to prototypes chaining up to Object, shouldn't statics of B <: statics of A?

This sharpens the distinction between inheritance under Node's utils.inherits and ES6's inheritance model (and every other random implementation out there):  utils.inherits uses a super_ attribute on the constructor function-object to expose the statics ancestry, while ES6 uses prototype chaining (ctor.__proto__ = theSuperClass emulates it under Node).
Constructor refinement would bring some complexity, e.g. new this(...) within a static method (but this currently doesn't work [#822]).

My actual use case is a function that returns a class extension, i.e.
export function list<B: BuilderList, R: ReaderList>(BList: Class<B>): Class<List<B, R>> {
  class ListList extends BuilderList<B, R> {
    _getUnsafe(index: uint29): B {
      var p = {
        segment: this._segment,
        position: this._begin + (index << 3)
      };
      return BList._deref(this._arena, p);
    }
    _setUnsafe(index: uint29, value: B | BList._Reader): void {
      var p = {
        segment: this._segment,
        position: this._begin + (index << 3)
      };
      setList(value._arena, value._layout(), this._arena, p);
    }
    static _hash() {
      return "L(" + BList._hash() + ")";
    }
    static _b64Null() {
      return "AQAAAAYAAAA=";
    }
    static _ct() {
      return new Meta(6, 0, 8);
    }
  }

  return ListList;
}

Since my extension class, ListList, is not a subtype of BuilderList, this function refuses to return.  I hope that #814 provides my workaround....


HDDS-7163. Ozone debug container export command support Container V3.

https://issues.apache.org/jira/browse/HDDS-7163

cc @aswinshakil

* I see that there is a new option `--count` added, and I'm not sure about it's behavior(e.g. when count > 1 and we have a failed `copyData()` call but exception != CONTAINER_NOT_FOUND, what is desired: bale out or continue).

If the exception is CONTAINER_NOT_FOUND, then it will continue. If there is other unexpected exception, then the command will exit with fail.

* And do we want some test for this command for compatibilty?

Yes.  the command is tested with V3 enabled, V3 disabled two scenarios.


fix(or-build-tools) fix openssl download 404

Quickfix for openssl moving releases around. The pattern substitution is not perfect, but works for now. A better regex could work, but would also fail in some cases, ie: 0.9.x.
The other option is to get the release directly from github: https://github.com/openssl/openssl/releases . We would need then to translate 1.1.1c -> OpenSSL_1_1_1c

:tada: This PR is included in version 4.2.2 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


After js production webpack,  code throws TypeError

I developed an website using this project. In development mode everything is fine. After compile to production, landing page shows
TypeError: Cannot read properties of undefined (reading '$metadata$')

After a long time of debugging, I figure out it is withService which throw this error.
My snippet of code:
log.info { "initing non auth" }
val y = getHost()
log.info { "got host" }
val rpc = client.rpc { url("ws://${y}/api/auth") }
log.info { "got rpc" }
authService = rpc.withService<AuthService>()
log.info { "got service" }

And log in browser is:

I initially thought it is websocket's connection problem, but it seems like everything is fine.

I have check all my dependencies, looked at all samples in this project, and still have no idea what's wrong.

Hi! Thank you for the report, that is indeed concerning, I'll take a look

I think it is this code which throw the error

Maybe after complied with production mode, js name is mangled so $metadata$ can not reference to what it should be?

Another guess is, in production mode, plugin-generated code is not send to bundle and being mangled together?

You'll find a minimal reproducer in nothing other than samples/ktor-web-app.
Although it no longer even builds for me, it can be pressed into service.
./gradlew frontend:jsRun, from the README, now yields:
> Task :frontend:jsBrowserDevelopmentRun FAILED
[webpack-cli] Invalid options object. Dev Server has been initialized using an options object that does not match the API schema.
 - options.proxy should be an array:
   [object { … } | function, ...]
   -> Allows to proxy requests, can be useful when you have a separate API backend development server and you want to send API requests on the same domain.
   -> Read more at https://webpack.js.org/configuration/dev-server/#devserverproxy

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':frontend:jsBrowserDevelopmentRun'.
> [webpack-cli] Invalid options object. Dev Server has been initialized using an options object that does not match the API schema.
   - options.proxy should be an array:
     [object { … } | function, ...]
     -> Allows to proxy requests, can be useful when you have a separate API backend development server and you want to send API requests on the same domain.
     -> Read more at https://webpack.js.org/configuration/dev-server/#devserverproxy

But if I hack the index.html into the resources/static folder of the server (which I think is the only file that the dev proxy is actually proxying at that point),
cp frontend/src/jsMain/resources/index.html server/src/main/resources/static/
and ./gradlew server:runApp (which does a production js build and copies over frontend.js).
Then, in the browser at http://<IP_ADDRESS>:8080/, this issues' exception:
react-dom.production.min.js:188 TypeError: Cannot read properties of undefined (reading '$metadata$')
    at dynamic.kt:48:310
    at WithRPCStubObject.js.kt:27:35
    at t.$_$.y1 (withService.kt:56:20)
    at l (withService.kt:24:12)
    at t.$_$.b (Standard.kt:83:5)
    at ft (ChildrenBuilder.kt:79:9)
    at FC.kt:16:31
    at Standard.kt:83:5
    at FC.kt:16:9
    at du (react-dom.production.min.js:160:137)
ho @ react-dom.production.min.js:188

(As you can tell from #147 I really want the sample code to build and run, mostly so I can use it as a nice minimal template to make you another reproducer!)


cmd/pprof: macOS 10.12 profile overcounts system calls

I've encountered a strong discrepancy between pprof and dtrace while profiling quic-go. I've managed to reduce it to a somewhat simple test case (link), where (in a loop) data is sent via UDP to localhost and some CPU intensive computation that takes a few ms runs.
Profiling this via dtrace gives the expected result, with the CPU intensive part taking the majority of the time. dtrace (imo correctly) shows doSomething taking almost all of the runtime, whereas pprof says 62% is spent in syscall.Syscall6, called by the netFD.Write method.
The dtrace profile is generated using flamegraph with
sudo dtrace -x ustackframes=1000 -n 'profile-99 /execname == "pprof-test" && arg1/ { @[ustack()] = count(); } tick-20s { exit(0); }' -o out.dtrace && ~/src/FlameGraph/stackcollapse.pl out.dtrace > folded.dtrace && ~/src/FlameGraph/flamegraph.pl folded.dtrace > dtrace.svg

The pprof profile is generated using
go build && ./pprof-test
# in another shell
go tool pprof pprof-test localhost:8080/debug/pprof/profile

### All files
- [The dtrace flamegraph SVG](https://dl.dropboxusercontent.com/u/1428776/pprof-report/dtrace.svg)
- [The dtrace output](https://dl.dropboxusercontent.com/u/1428776/pprof-report/out.dtrace)
- [The pprof SVG](https://dl.dropboxusercontent.com/u/1428776/pprof-report/pprof.svg)
- [The pprof profile](https://dl.dropboxusercontent.com/u/1428776/pprof-report/pprof.pprof-test.localhost%3A8080.samples.cpu.002.pb.gz)
- [The compiled binary from which the profile was generated](https://dl.dropboxusercontent.com/u/1428776/pprof-report/pprof-test)
### What version of Go are you using (`go version`)?
go version go1.7.1 darwin/amd64
What operating system and processor architecture are you using (go env)?
macOS 10.12 (16A323)
~/D/pprof-test → go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/lucas/src/go"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.7.1/libexec"
GOTOOLDIR="/usr/local/Cellar/go/1.7.1/libexec/pkg/tool/darwin_amd64"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/8c/y8j1444j1fbddn6bl788p1ym0000gn/T/go-build975132377=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"

Would it be possible for you to try a similar C program with the C compiler's -pg option?  The Go runtime is simply using setitimer with ITIMER_PROF.  While there could be a bug in how Go collects profiling data, it is also possible that this is simply due to differences between how Darwin handles setitimer/ITIMERPROF and dtrace.

I'm not sure whether -pg works properly on macOS, I couldn't get it to produce the profiler output. I did however try a similar thing with a C version (link, also needs this go server) and gperftools, and got the same result as with go pprof, with the syscall wrongly shown as taking the majority of the runtime. How much code do gperftools and go's pprof share? Is it possible there's a common bug?
I'm not sure how to put this, but as a pprof user I would be quite unhappy if the solution was "there's simply a difference in reporting". In that case, we should definitely find out in what cases exactly pprof is unreliable, and document this somewhere or/and alert users on Darwin about these issues. A profiler that is sometimes wrong can cause quite a lot of lost time ;)

gperftools and Go's pprof command are based on the same code, but I doubt that is in the issue.  If the same thing happens with C and Go, then the issue is almost certainly that the Darwin kernel setitimer call and dtrace measure different things.  The Darwin man page for setitimer says that ITIMER_PROF "decrements both in process virtual time and when the system is running on behalf of the process."  That is, it measures both user time and system time.  I don't really know what dtrace measures.
If there is a bug here, it's likely to be in the kernel implementation of setitimer, not in any part of the Go distribution.  But further investigation will require someone familiar with the Darwin kernel.

Thanks for the program. I ran it on my OS X 10.11.6 laptop and I get a profile that matches dtrace:
(pprof) top10
29.48s of 29.67s total (99.36%)
Dropped 14 nodes (cum <= 0.15s)
Showing top 10 nodes out of 14 (cum >= 0.34s)
      flat  flat%   sum%        cum   cum%
    24.60s 82.91% 82.91%     24.60s 82.91%  main.doSomething
     4.67s 15.74% 98.65%      4.67s 15.74%  syscall.Syscall
     0.20s  0.67% 99.33%      0.34s  1.15%  syscall.Recvfrom
     0.01s 0.034% 99.36%      0.35s  1.18%  net.(*UDPConn).readFrom
         0     0% 99.36%     29.28s 98.69%  main.main
         0     0% 99.36%      0.35s  1.18%  main.udpSink
         0     0% 99.36%      0.35s  1.18%  net.(*UDPConn).ReadFromUDP
         0     0% 99.36%      4.68s 15.77%  net.(*conn).Write
         0     0% 99.36%      4.68s 15.77%  net.(*netFD).Write
         0     0% 99.36%      0.34s  1.15%  net.(*netFD).readFrom

@quentinmit tried it on an OS X 10.12.1 beta laptop and got a similar profile (actually an even higher percentage of doSomething).
So neither of us can reproduce this.
Prior to OS X 10.11, there were problems with profiling signals being sent to the wrong thread, and the only workaround was to apply a binary patch to the kernel (see #6047). It is possible that the bug was reintroduced in OS X 10.12 and fixed in 10.12.1 beta, of course, but that seems unlikely.
@lucas-clemente, two more pieces of data that would be useful from your machine that exhibits the problem:

The full output of go test -v runtime/pprof. I'd like to see all the output but the most important part is whether there are any SKIP lines.

The output of running the C program in the 071d212 commit message.

Thanks.

I have since upgraded to 10.12.1 (16B2555), and can still reproduce the problem. I also got @marten-seemann to try it on his machine, and he's seeing the same issue, so we're 2:2 ;)
The output of go test -v runtime/pprof is here: https://gist.github.com/lucas-clemente/676c3ad5d5a92fd79c33f8315579b2ec. I can't see any SKIP lines.
The output of the C file from the commit message (compiled with clang test.c -o test is just 10 times signal on cpu-chewing looper thread, so I guess it's a new issue, not #6047 again.
I'd be really happy to help in any way, but I'm not sure what to do next :/

@lucas-clemente, is this a work machine that might have some kind of antivirus software on it watching the network I/O? Would the same be true of @marten-seemann's machine?
Failing that, I'm trying to figure out what's different between your machines and ours. The two I tried are laptops (MacBook Pros). What hardware are you using?
It's possible that dtrace is simply not reporting stacks at all for user code in a system call, so if the problem here is that the kernel is spending a lot more time in syscall.Syscall on your machine than it does on mine, that might not show up in dtrace. Even on our machines I see 5-15% of time in the system calls, and that's just not on your flame graph at all. Surely the program does spend some time in the writes, which makes me wonder if they are missing entirely from the dtrace.

Both private retina MacBooks Pros, mine is from late 2013 (2.3 GHz i7). I'm not aware of anything I have running that should interfere with the network.
I also checked that I'm not running any strange kexts, and only found VirtualBox (which I assume @marten-seemann has installed too). I tried uninstalling it and testing again, but that didn't change anything. I'm also using Docker's Mac app (which uses Hypervisor.framework), so I tried shutting that down too. Nothing changed.
~ → kextstat | grep -v com.apple
Index Refs Address            Size       Wired      Name (Version) UUID <Linked Against>
  142    3 0xffffff7f837bf000 0x61000    0x61000    org.virtualbox.kext.VBoxDrv (5.1.6) 2432AE53-EC50-36E5-B33B-799910A80FB2 <7 5 4 3 1>
  147    0 0xffffff7f83820000 0x8000     0x8000     org.virtualbox.kext.VBoxUSB (5.1.6) 4F861754-8C13-3623-BE58-893A2303341D <146 142 40 7 5 4 3 1>
  148    0 0xffffff7f83828000 0x5000     0x5000     org.pqrs.driver.VirtualHIDManager (1.1.0) 07DBA1E3-533B-3F80-B1F9-F2B0F417D96D <32 5 4 3 1>
  149    0 0xffffff7f8382d000 0x5000     0x5000     org.virtualbox.kext.VBoxNetFlt (5.1.6) 89A8D50A-DB9F-34B7-B1D0-20E305507610 <142 7 5 4 3 1>
  150    0 0xffffff7f83832000 0x6000     0x6000     org.virtualbox.kext.VBoxNetAdp (5.1.6) C1DDA422-665A-316D-AD52-FA64268ADA2C <142 5 4 1>
  162    0 0xffffff7f83838000 0x6000     0x6000     com.getdropbox.dropbox.kext (1.8.1) 81F3EECE-2B2A-35D5-A57A-62066A404BFC <7 5 4 2 1>

Today I am getting 30-50% in syscall, both from master and from Go 1.7, so I guess I've reproduced the problem? Not sure what has changed.

Even so I don't think we're going to get to this for Go 1.8.

I've also found that CPU profiling on macOS is pretty much useless lately.
@rsc, do you want to take a look for Go 1.9 or should we move to Go 1.10?

Not going to happen for Go 1.9, sorry.

I can also confirm profile is broken (again) on 10.12.6.
I am running High Sierra 10.13 Beta 2 on a second laptop (identical hardware) and everything works as expected.
dtrace is also broken on 10.12.6 per this stack-trace bug yet works fine on 10.13.

Happy to hear this is fixed in High Sierra. Closing.

I'm seemingly having this issue on http pprof (running a light benchmark with 10 simultaneous connections).
The trace clearly shows running the following path (overshadowing anything else, even tho some heavier tasks were performed):
1 net/http.(*conn).serve /usr/local/opt/go/libexec/src/net/http/server.go (42.58s)
2 net/http.(*response).finishRequest /usr/local/opt/go/libexec/src/net/http/server.go (11.77s)
3 bufio.(*Writer).Flush /usr/local/opt/go/libexec/src/bufio/bufio.go (9s)
4 time.Now /usr/local/opt/go/libexec/src/time/time.go (14.88s)
5 time.now /usr/local/opt/go/libexec/src/runtime/sys_darwin_amd64.s (14.88s)
and it branches from there and also gets an entry from my project.com/projectname/vendor/github.com/go-sql-driver/mysql.(*mysqlConn).writePacket /Users/erwin/Go/src/espaldd.com/espal/vendor/github.com/go-sql-driver/mysql/packets.go (8.49s):
5 net.(*conn).Write /usr/local/opt/go/libexec/src/net/net.go (10.14s)
6 net.(*netFD).Write /usr/local/opt/go/libexec/src/net/fd_unix.go (10.14s)
7 internal/poll.(*FD).Write /usr/local/opt/go/libexec/src/internal/poll/fd_unix.go (10.12s)
8 syscall.Syscall /usr/local/opt/go/libexec/src/syscall/asm_darwin_amd64.s (20.88s)
Running:
macOS High Sierra 10.13.2 (17C88)
go version go1.9.2 darwin/amd64

@Allendar This issue is closed because we think it is fixed.  Please open a new issue with full details.  Thanks.


PP-5937 Frontend blocks prepaid cards

add error code key for prepaid cards (en)
logic for blocking cards if the gateway account configuration is set
to block
validate behaviour with cypress

@kbottla https://github.com/alphagov/pay-frontend/pull/1256/commits/38d974c3f1d1e990796fdc0d1e65ba181f9a53ad


Add ability to specify ignore paths in the deps array

Something like
deps = [
  'server/**/*.js',
  '!server/ssr/*.js',
  'node.importmap'
]

That would ignore all js files in server/ssr, but include all js files in server/ssr/src

We use https://docs.rs/glob/0.3.0/glob/struct.Pattern.html which has the rules in that documentation page.
Specifically, there is some negation syntax:

[...] matches any character inside the brackets. Character sequences can also specify ranges of characters, as ordered by Unicode, so e.g. [0-9] specifies any character between 0 and 9 inclusive. An unclosed bracket is invalid.

[!...] is the negation of [...], i.e. it matches any characters not in the brackets.

The metacharacters ?, *, [, ] can be matched by using brackets (e.g. [?]). When a ] occurs immediately following [ or [! then it is interpreted as being part of, rather then ending, the character set, so ] and NOT ] can be matched by []] and [!]] respectively. The - character can be specified inside a character sequence pattern by placing it at the start or the end, e.g. [abc-].

Some combination of the above may be able to achieve your use case.

Ah, thank you.
Can that be used for hash-style deps = dep = 'src/##.ts'?
I can't easily see how I would structure a dep with an ignore. If you know this offhand, could be helpful in the docs. Because I didn't know how to do this I've restructured my files so that I don't have to ignore anything. But could be useful in the future.

No, interpolations do not permit additional glob filter rules due to the 1-1 requirement.


Fix bad commiter info from #491

@marcingrzejszczak I push the commits of issue #491 with no proper git config and as such it created bad commiter info (non existing user), this PR fixes that if you can please check it out

Sure, can you set this PR vs the 1.1.x branch?

This commit was ported only to 1.2.x, do you want me to create another PR to it?

Was it? Ok, so create a PR to 1.2.x

Sure
On Dec 19, 2017 10:49 AM, "Marcin Grzejszczak"<EMAIL_ADDRESS>wrote:

Was it? Ok, so create a PR to 1.2.x
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/spring-cloud/spring-cloud-contract/pull/497#issuecomment-352678060,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFggn3ffGut-ZtOkl2vF6qPYWWFRv3V2ks5tB3iygaJpZM4RGMVG
.

see #498

Done via https://github.com/spring-cloud/spring-cloud-contract/commit/f69a2f63d004eab950bb9fda737e6e89c3478ea6


Python 3.2 compatibility

Jprops breaks on python 3.2.
Python 3.2.3 (default, Feb 20 2013, 14:44:27)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import jprops
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.2/dist-packages/jprops.py", line 155
    c = u'\\\\'
              ^
SyntaxError: invalid syntax

Yeah, as mentioned in the README, jprops works with 2.6, 2.7, or 3.3 (or higher). Python 3.3 added back the unicode literal support in order to simplify porting code to Python 3, so this library currently expects that.
Can you upgrade your Python installation?

Yeah, I saw the readme, still I opened the issue.
If a script needs to be running on multiple servers with mixed environments, some of them with python 3.4, some with 3.2, this issue is quite a headache. Upgrade is not an option, I know there are other ways to get a python3.4, but that's not the question here. I am suggesting that jprops should avoid those unicode literals... ;)

Well, Python 3.3 added back support for the u prefix because it makes it much easier to maintain code that is compatible with both Python 2 and 3.
The unicode literals only appear a few times in jprops.py itself, so if you only need to run on Python 3 you could make a copy to include with your code and remove the u prefixes.
They appear more times throughout the tests, so changing the whole codebase to be 3.2 compatible would require more work. I'm not really using jprops much at this time, so given the age and adoption rate of Python 3.2 it's not something I'm likely to work on.


new(driver,userspace/libsinsp,test): properly manage fd in connect exit event too

What type of PR is this?
/kind feature
Any specific area of the project related to this PR?
/area driver-kmod
/area driver-bpf
/area driver-modern-bpf
/area libsinsp
/area tests
Does this PR require a change in the driver versions?
Yep but it was already bumped during this release cycle.
/version driver-SCHEMA-version-minor
What this PR does / why we need it:
This improves connect resiliency when connect enter event is dropped.
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?:
new(driver,userspace): send fd in connect exit event too to improve resiliency in case connect enter gets dropped.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: FedeDP
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [FedeDP]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/milestone next-driver

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: FedeDP, incertum
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [FedeDP,incertum]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/hold

@incertum @LucaGuerra  @leogr PTAL! Thanks!

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: FedeDP, incertum, LucaGuerra
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [FedeDP,LucaGuerra,incertum]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/unhold


Ignore remote certificate validation errors if accept any configured

Fixes / New Feature
We can not use WebSocket SignalR protocol in Ocelot with self-signed certificate on downstream service side because DownstreamRoute.DangerousAcceptAnyServerCertificateValidator not used to skip ClientWebSocket's remote certificate validation.
Proposed Changes

Use fake validator and assign it to the RemoteCertificateValidationCallback property of the client to have successful validation result always

Some updates about this?

Some updates about this?

we eventually moved to YARP..

@abelevtsov Does YARP has all the features of the Ocelot such as Authentication, Delegation Handler, Header Transform etc?

@PratikPatel-Mtech  yes it have and much more over.

Hey - is this to be fixed? We have the same problem: unable to use SignalR with self-signed dev certificate.

@abelevtsov  Hi Aleksandr!
What issue is this PR related to?
Please note, Ocelot does not support wss-protocol. So, you cannot use WebSocket Secure vs SignalR.
Moreover, SignalR is supported only for legacy HTTP1 pairing.
See more here: #1636

@abelevtsov commented on Sep 9, 2022

Sad to hear about this fact, man! 😢
What is the reason of moving to Yarp? Inability to use self-signed certificates for Web Socket client?!
Or the reason is  different...

Hey, @abelevtsov @zewa666 @thomaschampagne @PratikPatel-Mtech @alexandis
Please, watch for #1377 and enable notifications.
Going to close this PR...
Hope you will contribute to Ocelot project in coming future. 😉

@abelevtsov commented on Sep 9, 2022

Sad to hear about this fact, man! 😢
What is the reason of moving to Yarp? Inability to use self-signed certificates for Web Socket client?! Or is there another reason...

the reason is about that PR was opened in 2021 and response was received in 2023, when our the project was already in past...