Build Failed Command - "npm run build" exited with 1

Unable to deploy app on Vercal
[10:44:19.228] Cloning github.com/jschof1/my-daily-blog (Branch: main, Commit: 6de7271)
[10:44:20.230] Cloning completed: 1.002s
[10:44:20.608] No Build Cache available
[10:44:20.653] Running "vercel build"
[10:44:21.296] Vercel CLI 28.2.2
[10:44:21.686] Installing dependencies...
[10:44:23.181] npm WARN old lockfile
[10:44:23.181] npm WARN old lockfile The package-lock.json file was created with an old version of npm,
[10:44:23.181] npm WARN old lockfile so supplemental metadata must be fetched from the registry.
[10:44:23.181] npm WARN old lockfile
[10:44:23.181] npm WARN old lockfile This is a one-time fix-up, please be patient...
[10:44:23.182] npm WARN old lockfile
[10:44:45.718] npm WARN deprecated<EMAIL_ADDRESS>Modern JS already guarantees Array#sort() is a stable sort, so this library is deprecated. See the compatibility table on MDN: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/sort#browser_compatibility
[10:44:50.502] npm WARN deprecated<EMAIL_ADDRESS>Please upgrade to latest, formidable@v2 or formidable@v3! Check these notes: https://bit.ly/2ZEqIau
[10:44:52.328] npm WARN deprecated<EMAIL_ADDRESS>Please upgrade to v7.0.2+ of superagent.  We have fixed numerous issues with streams, form-data, attach(), filesystem errors not bubbling up (ENOENT on attach()), and all tests are now passing.  See the releases tab for more information at <https://github.com/visionmedia/superagent/releases>.
[10:45:04.682] npm WARN deprecated<EMAIL_ADDRESS><EMAIL_ADDRESS>is no longer maintained and not recommended for usage due to the number of issues. Because of the V8 engine whims, feature detection in old core-js versions could cause a slowdown up to 100x even if nothing is polyfilled. Some versions have web compatibility issues. Please, upgrade your dependencies to the actual version of core-js-pure.
[10:45:07.929]
[10:45:07.929] ><EMAIL_ADDRESS>prepare
[10:45:07.930] > husky install
[10:45:07.930]
[10:45:08.015] husky - Git hooks installed
[10:45:08.045]
[10:45:08.045] added 1324 packages in 46s
[10:45:08.045]
[10:45:08.045] 319 packages are looking for funding
[10:45:08.045]   run `npm fund` for details
[10:45:08.083] Detected Next.js version: 12.2.2
[10:45:08.100] Detected `package-lock.json` generated by npm 7+...
[10:45:08.100] Running "npm run build"
[10:45:08.483]
[10:45:08.483] ><EMAIL_ADDRESS>build
[10:45:08.484] > next build && node ./scripts/generate-sitemap
[10:45:08.484]
[10:45:09.013] Attention: Next.js now collects completely anonymous telemetry regarding usage.
[10:45:09.014] This information is used to shape Next.js' roadmap and prioritize features.
[10:45:09.014] You can learn more, including how to opt-out if you'd not like to participate in this anonymous program, by visiting the following URL:
[10:45:09.014] https://nextjs.org/telemetry
[10:45:09.014]
[10:45:09.142] info  - Linting and checking validity of types...
[10:45:13.666]
[10:45:13.666] ./pages/api/klaviyo.js
[10:45:13.667] 1:1  Warning: Assign arrow function to a variable before exporting as module default  import/no-anonymous-default-export
[10:45:13.667]
[10:45:13.667] info  - Need to disable some ESLint rules? Learn more here: https://nextjs.org/docs/basic-features/eslint#disabling-rules
[10:45:13.670] info  - Creating an optimized production build...
[10:45:17.062] Browserslist: caniuse-lite is outdated. Please run:
[10:45:17.062]   npx browserslist@latest --update-db
[10:45:17.063]   Why you should do it regularly: https://github.com/browserslist/browserslist#browsers-data-updating
[10:45:45.305] Failed to compile.
[10:45:45.306]
[10:45:45.306] ./node_modules/mdx-bundler/dist/client.js
[10:45:45.306] Module not found: Package path ./compat/jsx-runtime.js is not exported from package /vercel/path0/node_modules/preact (see exports field in /vercel/path0/node_modules/preact/package.json)
[10:45:45.306]
[10:45:45.306] Import trace for requested module:
[10:45:45.307] ./node_modules/mdx-bundler/client/index.js
[10:45:45.307] ./components/MDXComponents.js
[10:45:45.307] ./pages/about.js
[10:45:45.307]
[10:45:45.307]
[10:45:45.308] > Build failed because of webpack errors
[10:45:45.359] Error: Command "npm run build" exited with 1

Module not found: Package path ./compat/jsx-runtime.js is not exported from package /vercel/path0/node_modules/preact (see exports field in /vercel/path0/node_modules/preact/package.json)

It seems that for some reason Preact is not found. Could you try updating your preact version and next.config.js to match the current master? Otherwise, if you don't mind not using Preact, you can remove these lines and it should work.

Thank for your help, I finally managed to get it deployed!
Steps to fixing:

I updated package.json by replacing the one I was using with the one in the latest version of this repo.
replaced the next.config.js with the latest repo.
I was still having the same problem so I changed this

 webpack: (config, { dev, isServer }) => {
    config.module.rules.push({
      test: /\.svg$/,
      use: ['@svgr/webpack'],
    })

To this
webpack: (config, { dev, isServer }) => {
    (config.resolve.alias = {
      ...config.resolve.alias,
      // fixes next-mdx-remote: Package path ./jsx-runtime.js is not exported from package react
      // https://github.com/hashicorp/next-mdx-remote/issues/237
      'react/jsx-runtime.js': require.resolve('react/jsx-runtime'),
    }),
      config.module.rules.push({
        test: /\.svg$/,
        use: ['@svgr/webpack'],
      })

Then but then it was still failing to build (!) because of the prettier extension so I ran this in my root directory

npm run lint --fix
Ta da!


Error while activating Python

After starting VS code i get this error message from intellicode.
Acquiring model 'intellisense-members-lstm' for python
Querying IntelliCode service for available models.
Cached model is up to date.
vs-intellicode-python was passed a model: {"modelId":"15708AE89896CA5DF1690433B7A8D93D28B7","modelPath":"c:\Users\Pedro\.vscode\extensions\visualstudioexptteam.vscodeintellicode-1.2.7\cache\15708AE89896CA5DF1690433B7A8D93D28B7_215F8B4FAB1F4663ABD624559FD205B9","outputId":"215F8B4FAB1F4663ABD624559FD205B9"}.
Error while activating Python: {}

Hey @nbkgit!
Thanks for reaching out. It sounds like the Python extension may not have started.
Can you confirm if the Python extension successfully activated?
Also, can you provide me with your Python settings:

Interpreter version
Extension version
python.languageServer setting value (and version)

Extensuion version : visualstudioexptteam.vscodeintellicode-1.2.7
python.languageServer Says default microsoft without version
the python extension is version 2020.5.80290
I can't tell you if it is now installed correctly.
It was some 1-2 updates ago, when is´ still could debug my code
but know i have also another python related problem, which i have also made an issue out. see https://github.com/tht13/Python-vscode/issues/51

I posted the log in my other maybe this helps here too https://github.com/tht13/Python-vscode/issues/51#issuecomment-635000588

Thanks for the info!
I think that this is likely a Python extension-related issue so to simplify tracking this issue, I'd suggest filing an issue to the official python-vscode repo with the activation issues you are describing, as it looks like you filed that issue to a fork of the extension.

Thx to savannahostrowski
After some intensive research and experiment i found the solution
https://github.com/tht13/Python-vscode/issues/51#issuecomment-639092724


[Enhancement] [CameraView] Remove unused code

Summary
I would like to remove unused code to make it easier to maintain, starting from FormsCameraView.

API Changes
Removed:

UIImage RotateImage(UIImage image)

Intended Use Case

NA
Who Will Do The Work?

[x] I am willing to take this on myself
[ ] Just putting this out there for someone to pick up

@jfversluis what do you think about it?

@pictos I will leave the PR as a draft for now 👍

Don't spend too much time on this. I appreciate the effort, but I think there might be big changes when we transition this to the .NET MAUI Community Toolkit. If you want to do some clean up and get some contributions in, I don't have any problem with that :)

Hey @pictos could you please close this issue? Thanks 🙏


coreir executable cannot find liftclockports or wireclocks?

git info: I am working on a fork of "dev" as usual.
I am trying to run some of the CGRAMapper examples through the simulator and before I can do that I need to add a clock to the inputs and wire up all sequential elements to the clock.
It looks like the passes liftclockports and wireclocks should do these two tasks for me, but when I try to run them as passes I get an error. For example when I am in /Users/dillon/CppWorkspace/CGRAMapper and I run:
bash-3.2$ ~/CppWorkspace/coreir/bin/coreir -i ./examples/conv_3_1.json -o ./examples/sim_ready_conv_3_1.json -p rungenerators,flattentypes,flatten,liftclockports,wireclocks --load_libs ~/CppWorkspace/coreir/lib/libcoreir-commonlib.dylib
Inline Pass: bitconst has no definition, skipping...
Inline Pass: bitterm has no definition, skipping...
Inline Pass: bitterm has no definition, skipping...
ERROR: Can not run pass "liftclockports" because it was never loaded!

Assertion failed: (passMap.count(oname)), function pushAllDependencies, file passmanager.cpp, line 145.
Abort trap: 6

rungenerators, flattentypes, and flatten all work just fine, but coreir does not seem to be able to find liftclockports and wireclocks.
What am I doing wrong here?

Wrong name.


fix(tests): Properly set default route

This PR relates to #121 and fixes some underlying test issues uncovered when attempting to fix the failing tests in that PR.

By default happy-dom will have the URL about:blank. In the tests it seems we were assuming we would immediately end up on an index page when in fact we were not due to this behavior of happy-dom.
This change properly sets the default URL to http://localhost:3000 which is the same as Vitest sets for JSDOM.
There were two tests that started failing for this change:

"defaults to index page"
This test previously was ending up on the "slug" page, since the URL was matching "about:blank" to this. This snapshot was updated since it now correctly goes to the index page.

"allows pushing to other pages"
The useRoute() in this test can no longer be called sync. We need to wait for the reactive route variable to update. Assuming this is due to some previous quirk when vue-router encountered "about:blank".

@danielroe Seems the route test is failing in CI 🤔 I cannot reproduce this locally. Do you think there's some other approach we could take here to await for the reactive route to update?

@danielroe I find it quite hard to debug this unfortunately 🤔
Could you help explain what the difference is between running test:unit and test:dev?
Is the later of these the de-facto way of running nuxt-vitest?
Debugging the failing test, it seems there a significantly different context between them: in test:unit we actually have something rendered to the DOM. In test:dev the DOM is mostly empty.

Found the root cause. Fix is in the second commit. 👍

Oh, very nice find.


How to remove the seperator or change its color?

              <QuillToolbar
                    editor={quillEditorRef}
                    theme={{
                      background: '#000',
                      color: '#000',
                      overlay: 'rgba(55,99,115, .1)',
                      size: 30,
                    }}
                    options={['emoji', 'mention', 'giphy']}
                    styles={{
                      toolbar: {
                        provider: () => ({}),
                        root: () => ({
                          backgroundColor: 'white',
                        }),
                      },
                      separator: (separator) => ({
                        ...separator,
                        color: '#fff',
                      }),
                    }}
                    custom={{
                      handler: (name) => {
                        handleToolbarActions(name);
                      },
                      actions: ['emoji', 'mention', 'giphy'],
                      icons: {
                        emoji: require('../assets/img/emoji.png'),
                        mention: require('../assets/img/mention.png'),
                        giphy: require('../assets/img/gif.png'),
                      },
                    }}
                  />

@imnapo

you can't cause separator uses the theme.color
unless you fork the repo and change it to your own requirements

@abdulwasey Can you share what your "handleToolbarActions" do? I am trying to implement "mention" feature will help me with that


Add additional DATE_PART units

Which issue does this PR close?
Closes https://github.com/apache/arrow-datafusion/issues/3502.
Rationale for this change
The quarter, doy and dow DATE_PART units are already supported by arrow-rs kernels, so lets add them to DataFusion!
The names and semantics match those in PostgreSQL: https://www.postgresql.org/docs/8.1/functions-datetime.html
Are there any user-facing changes?
Adds support for quarter, doy and dow DATE_PART units.
I'm not sure if there is any documentation that should be updated as well.

Thanks @Dandandan @andygrove !


An annotation to either prevent or force inlining of a function

We've been using GCC in the simple mode for compiling React for a few months, and are really happy with its heuristics. However, recently I've been noticing a minor pain point.
Problem
In some cases, as library authors we want to have more control over whether a function should be inlined or not. In particular, there are two scenarios:

Some functions are in a very hot path, and we have determined that not inlining them would hurt the runtime performance. Usually at this point we just inline them manually in the code. However, that often produces duplicate code that is hard to work with. Ideally, we'd love to have a way to tell GCC that it has to either inline a function or fail the build (if it’s impossible). This way we know that we won’t regress on this particular function.

There is an opposite problem as well. Sometimes we split a function into several to have tighter control over the argument types. For example, if some path is very hot, and we need to ensure a function is called with monomorphic arguments. However, GCC can break these optimizations by inlining the separated function (and thus preventing engines from optimizing that code due to deopts). For this use case, we'd like to have a way to tell GCC to not inline a function, even if the resulting code size will be larger.

Prior Art
The other GCC 🙂

inline and always_inline
noinline

Proposal
New @inline annotation forces inlining
/** @inline */
function someHelper() {
  // ...
}

// We want to fail the build if these couldn't be inlined.
someHelper();
someHelper();

This forces the function to be inlined even if the code size becomes bigger.
If it can’t be inlined, the build fails.
Alternative possible naming: @always-inline, @force-inline.
New @noinline annotation prevents inlining
/** @noinline */
function doSomethingWithString(stringValue) {
  // ...
}

/** @noinline */
function doSomethingWithNumber(numberValue) {
  // ...
}

function doSomething(mixedValue) {
  // It is important that these calls never get inlined
  if (typeof mixedValue === 'string') {
    doSomethingWithString(mixedValue);
  } else if (typeof mixedValue === 'number') {
    doSomethingWithNumber(mixedValue);
  }
}

This forces a function to never be considered for inlining, even if there is a size win.
Alternative possible naming: @never-inline, @prevent-inline.
Other hint formats?
I don't have a strong opinion about how to supply these hints. A comment, a JSDoc annotation, special function name, etc, would all work for me. Please let me know if this is something you would consider!
Current Workarounds
In case anyone’s curious, I’ve thought a bit about how to achieve this today.
Forcing inlining
There's no way to force inlining, but we could plausibly adopt a special naming convention (e.g. *Inline). We already have a special build where we disable the renaming pass (#2707) so we could analyze the bundle, and fail the build if any *Inline function still exists. This wouldn’t force GCC to inline it, but at least we’d know if it was inlined but then started bailing.
Preventing inlining
This file might contain some hints. For example attaching function to an object would prevent inlining that function.
Of course it would be nice to have an official solution for this, as both options are hacky and incomplete.

We very much shy away from annotations which direct optimizations. They tend to be major footguns. In general we'd like to improve the compiler heuristics around inlining.
You can easily block inlining of a function with a sink value. Closure-Library has an example.
Forcing inlining is a bit trickier. I don't have a good answer here. Perhaps @concavelenz has thoughts?

Totally understand if that’s an intentional design decision 👍
Is the sink method you’re linking to all that different though? Perhaps it lifts the responsibility from GCC itself so it doesn’t feel like something you explicitly recommend. That makes sense to me. Is the exact approach it’s using more or less guaranteed to work in the future?

Is the sink method you’re linking to all that different though?

It behaves exactly the same no matter which compiler passes are enabled. That's one of the major issues.

The "noinline" annotation might be reasonable but I would like to understand a bit more how we break the heuristic by inlining.  Is it not sufficient that the inlined code be inlined into monomorphic branch? Or do we end up de-duplicating the code branches and end up creating megamorphic code?  We do have a knob to control how large a function inlining is allowed to create if it is overall function size that triggers the issue.
I'm curious about the "inline" heuristic.  Where does this actually help?  It has been my assumption that with the modern VMs that they would do a better job of inlining for performance, and that we should only be inlining for code simplification.   Or is this a case of code running without a JIT (react native?)

From conversations among the team last week, it sounds like we're leaning toward implementing "noinline" but not implementing "force inline"

It looks like we want to use @noinline ourselves in order to prevent $jscomp.polyfill from getting inlined, which makes it a lot harder to find when trying to remove unused polyfill code.
I recently added @nocollapse to the definition of $jscomp.polyfill as a temporary work around when
a performance improvement change caused the inlined version to become even harder to identify by removing its unused parameters early in the compilation.

@inline would be an extremely useful addition for the complex algorithms.

Just FYI, I've started the work to add @noinline. We don't currently have plans to do the opposite yet.

I've now done enough work for @noinline to work for $jscomp.polyfill, and that was my primary motivation to work on it.
There are a few more *Inline*.java classes I haven't touched, so there are probably cases where it
still gets ignored.
I volunteer to be the primary person to handle issues related to it being ignored and PRs to fix them,
but I don't plan to work on it further for now other than that.

hey guys, came here to open the same issue.
my long-time work-around to prevent Closure's aggressive inlining has been to expose these functions in the window namespace and strip them out of the bundle via regex after compilation [1].
is there any information as to why the JIT is negatively affected? i always assumed it's because the function hits a certain AST size threshold and the JIT simply bails on more advanced/expensive tracing. i remember back in the day when actual function character count was supposedly used as a de-opt heuristic (including whitespace and comments!) [2], [3].
[1] https://github.com/leeoniya/domvm/issues/144
[2] https://www.nearform.com/blog/node-js-is-getting-a-new-v8-with-turbofan/
[3] https://github.com/davidmarkclements/v8-perf/blob/master/bench/function-size.js

@leeoniya
Our team doesn't have any info about Chrome's JIT behavior.
However, if your goal is just to prevent inlining in your code, I suggest you try using the @noinline annotation. If it works for you, great. If not, let us know.
I know that I didn't update every possible code path that could lead to inlining, but I believe I got most of them, so @noinline should work for you.
Also, if you're trying to prevent inlining because you expect your compiled code to be used from other JavaScript code, you should add @export to the visible classes and functions. Then you won't even need the @noinline, since the form will be preserved to make sure your exported definitions are available to external code.

thanks @brad4d,
@noinline appears to have worked with closure-compiler-js, too. for some reason i figured annotations only worked in ADVANCED mode.
also, not everything's exposed in the js version. ahem trustedStrings 😉 [1], [2].
cheers!
[1] https://github.com/google/closure-compiler-js/issues/79
[2] https://github.com/leeoniya/domvm/blob/1da486030be281d1f7599b55ef5aa16ebc89e68e/build.js#L152-L160


CI uses actions/checkout@v3, which uses a deprecated version of Node.js

https://github.com/tembo-io/pgmq/actions/runs/9290147430


enh: Pass DB credentials and URL separately instead of fully-formed DB_URL

Is your feature request related to a problem? Please describe.
In case of deployment where the database credentials are rotated but the URL remains the same (for ex. AWS RDS with rotated DB credentials in Secretsmanager), it would be helpful in CICD and IaC to have password and URL separated.
So, URL can go as a environment variable but the credentials can be passed on separately as a secret.
Describe the solution you'd like
DB_PASS and DB_URL are passed as separate environment variables in the docker container
Describe alternatives you've considered
Manually constructing the url via custom CFN resource in AWS. But, that seems very hacky.
Additional context
The OpenWebUI is deployed in Amazon ECS via Cloudformation hooked to AWS RDS with rotating credentials.
LiteLLM sidecar container acting as a gateway to Amazon Bedrock.

PR welcome!


Support country search

Following https://github.com/EnsembleUI/ensemble/issues/867, please add the ability to search for a country by name, alpha-2, and alpha-3.
ensemble.utils.findCountry(searchText);

Please come up with a proposal before implementation on the Search mechanism.

how is it going @fullstackctp ?

how is it going @fullstackctp ?
It is going ohk. I had a few doubts regarding the functionality that needs to be implemented. I asked sharjeel and vinoth to help me out.

https://github.com/EnsembleUI/ensemble/assets/75938927/4db3784d-a2e7-42b0-968b-7c90dcbaec46

fixed in #880


XRI 2.3 Upgrade

Original issue opened by:
 Zee2

This ticket tracks upgrade to XRI 2.3, along with the XR coreutils upgrade

[ ] Evaluate new input simulator, how much of our own input simulator we can remove

[ ] Already works shockingly well, main problem is that they don't have a concept of the "anchor point" so the hand ray/grab flails around wildly (they just anchor on the device position, obviously)

[ ] Evaluate Transformers and how they will influence our constraints/solvers story
[ ] Evaluate the new datum feature in XR Coreutils, possibly use it for config on BC, UX?
[x] Tests pass
[ ] Check in with internal partners before bumping XRI minver
[ ] Possibly refactor our Selectedness to point to their interactionStrength

[ ] This would require us to implement our poke logic as an MRTKPokeFilter
[ ] This would also require us to deprecate our IVariableSelectInteractor and migrate to IXRInteractionStrengthInteractor (phew, what a mouthful)

[ ] Evaluate replacing our GazeInteractor with their XRGazeInteractor
[ ] Evaluate replacing InteractionModeManager with XRI interaction groups

Groups may not support enabling/disabling based on external rules/detectors :(

Notes

Something seems very wrong with our ray interactor/controller execution order, at least in-editor. The default XRI XRGrabInteractable goes crazy when grabbing/rotating. Definitely an execution order issue
They have a hardcoded "Enable gaze interaction" checkbox on their interactable settings, which compares incoming interactors against their concrete impl of XRGazeInteractor, instead of any kind of generic interface like IGazeInteractor

This can/will confuse our users when our gaze interactor does not respond to the XRI options for gaze.

Our line visuals do not respond to XRInteractableSnapVolume
Most things seem to work just fine so far, even with some of the conflicts with gaze, etc
Their new XRPokeInteractor does not do inter-frame collision checks so it may be susceptible to "phasing through" interactables

Confirmed, their poke interactor will not process pokes when the interactor is moving quickly

Their "poking depth" is a property of the poke interactor itself, not the interactable, which is a really strange decision because buttons can be different sizes and depths 🤔🤔🤔
Their new XRPokeInteractor implements IUIInteractor, which means we can do the exact same thing and get touch interactors on vanilla canvas!

Unfortunately, the API for adding this to our own interactors is internal, so currently only official XRI XRPokeInteractors can do UGUI interactions. Huge bummer :(

They now have their own "Selectedness", which is "interaction strength". We can pretty easily adapt to this by implementing our own PokeFilter and just routing the Selectedness API down to their interaction strength value
Their PokeFilter is hardcoded to only work with XRPokeInteractors, not any generic interface like an IXRPokeInteractor :(
Our PokeInteractorVisuals script assumes a PokeInteractor which is a little silly. We should change it to a generic interactor.
New input simulator already works shockingly well, main problem is that they don't have a concept of the "anchor point" so the hand ray/grab flails around wildly (they just anchor on the device position, obviously)

Not sure how feasible it is to alter their simulator to do this
Possibly we can change the joint synthesis subsystem to be smarter about this (i.e., anchor the simulated joints, not necessarily the simulated device pose)

The new XRPokeInteractor and XRPokeFilter API is incredibly strange and I literally cannot make heads nor tails of it

I've tried for like 2 hours at this point to make a simple button that presses at a given distance and I cannot do it
This will require a lot more work to understand and possibly work with the XRI folks to improve things

Looks like the XRInteractionGroups have a lot of limitations and won't support enabling/disabling based on external rules like we do with our modes

ISSUE MIGRATION
Issue migrated from: https://github.com/microsoft/MixedRealityToolkit-Unity/issues/11282

I am having an issue where I upgraded my project from pre release to general release, unity 2022.2.12 and XR Interaction Toolkit 2.3 and when using my input simulator I will get my hand ray when I press shift on PC but when I mouse over a button the pokeinteractor does not rollover the button even though I can click the button still. Any ideas?


IPsec phase 2 unreliable if IPsec NAT is enabled

If you configure IPsec NAT, the IPsec phase 2 tunnels may become unstable. In my case all phase 2 tunnels remain disconnected:
Oct 19 12:09:28 opnsense charon: 15[IKE] <con1-000|1> IKE_SA con1-000[1] established between X[X]...Y[Y]
Oct 19 12:09:28 opnsense charon: 15[IKE] IKE_SA con1-000[1] established between X[X]...Y[Y]
Oct 19 12:09:28 opnsense charon: 15[IKE] <con1-000|1> scheduling reauthentication in 28101s
Oct 19 12:09:28 opnsense charon: 15[IKE] <con1-000|1> maximum IKE_SA lifetime 28641s
Oct 19 12:09:32 opnsense charon: 13[IKE] <con1-000|1> sending retransmit 1 of request message ID 951205933, seq 4
Oct 19 12:09:39 opnsense charon: 12[IKE] <con1-000|1> sending retransmit 2 of request message ID 951205933, seq 4
Oct 19 12:09:52 opnsense charon: 06[IKE] <con1-000|1> sending retransmit 3 of request message ID 951205933, seq 4
Oct 19 12:10:16 opnsense charon: 08[IKE] <con1-000|1> sending retransmit 4 of request message ID 951205933, seq 4

The only solution in this case is to disable all IPsec NAT entries, stop ipsec and restart it. Afterwards all phase 2 tunnels will come up immediately. Once all phase 2 tunnels are established, it is possible to enable the IPsec NAT entries again (but this is dangerous because a reconnect of the tunnel is very unlikely to succeed).
We do not have this issue with IPsec NAT disabled.
While debugging #369 with @AdSchellevis we've been wondering why entries to ipsec.conf are added for IPsec NAT. Is this actually required for IPsec NAT to work? It seems that this confuses the ipsec daemon.

@fraenki just checking, but if I'm not mistaken it functions normal if the nat entry is an actual tunnel, but if you want to nat some parts of your network, you can create entries over here which will never be active and may confuse strongswan sometimes, but will do the nat part, because it has nothing to do with ipsec.
The problem over here seems to be, that we can't configure these "do not NAT all my traffic tunnels" in the nat section of the firewall.
Am I right?

@AdSchellevis I'm not sure what you mean with "...if the nat entry is an actual tunnel...".
On the other hand, yes, the problem seems to be that we're not able to create similar NAT rules elsewhere. Currently it's only possible on the IPsec phase 2 configuration page.
I've already tried to replace this "IPsec phase 2 NAT" entry with a regular "outbound NAT rule", but this didn't work. I'm not sure what magic was missing to make it work. Maybe it's related to #438?

Just "confirmed" this issue with a second tunnel, FWIW. Disabling the IPsec phase 2 NAT entries immediately solved the connection issue.

That's good, and yes it's related to the other issue. You can only create NAT rules for tunnels you actually have at the moment, and there doesn't seem to be a logic reason for it.
When the rules are disabled, your situation should work (because it seems to keep the rules, https://github.com/opnsense/core/issues/439), but eventually you need to be able to create outbound nat rules for this.

OK, let me summarize what I've tried so far.
I have the following IPsec phase 2 tunnel configured:

Local network:  <IP_ADDRESS>/24
Remote network: <IP_ADDRESS>/24

But I want to be able to reach the remote network from my second local network, <IP_ADDRESS>/24, too. That's why I've added the following IPsec phase 2 NAT configuration:

Local Network: <IP_ADDRESS>/24
NAT/BINAT Network: <IP_ADDRESS>/24
Remote Network: <IP_ADDRESS>/24

This should allow us to access the remote network <IP_ADDRESS>/24 from our second local network <IP_ADDRESS>/24, because it gets NAT to <IP_ADDRESS>/24. You see that a pf NAT rule is added:
binat on enc0 inet from <IP_ADDRESS>/24 to <IP_ADDRESS>/24 -> <IP_ADDRESS>/24

And it works. But the ipsec daemons gets confused, as described earlier.
That's why I tried the following:

disabled the IPsec phase 2 NAT configuration
the pf BINAT rule was still there
added a reglar outbound NAT rule

Now the pf NAT table looks like this:
nat on enc0 inet from <IP_ADDRESS>/24 to <IP_ADDRESS>/24 -> <IP_ADDRESS>/24 port 1024:65535
binat on enc0 inet from <IP_ADDRESS>/24 to <IP_ADDRESS>/24 -> <IP_ADDRESS>/24

Unfortunately, the connection from <IP_ADDRESS>/24 to <IP_ADDRESS>/24 stopped working immediately after disabling the IPsec phase 2 NAT configuration.
Is there any way to configure this IPsec NAT without interferring with the ipsec daemon?

@AdSchellevis I've seen some (rather old) posts stating that PF can't do NAT for IPsec (at least not on FreeBSD). Is this statement still true? On the other hand some people advise to set net.inet.ipsec.filtertunnel=1 in this regard... do you have any comments on this to share? :)

Setting the sysctl net.inet.ipsec.filtertunnel to 1 did not help to solve the NAT issue, but it would allow us to filter the traffic inside the tunnel (as the name suggests).

@fraenki what's the status here?

@fichtner Currently it cannot be solved so easily. I think I should open a new RFE issue with only details on the technical limitations/issues.

We can update the first post if needed, just let me know how to transform the ticket (title+description). would be sad to lose the extra info in the comments.

@fichtner I've updated my original submission with the description of the underlying problem.

@fraenki thank you :)

FWIW, the quick fix ("patch strongSwan, revive some old code") is still working on 16.1.3.

more discussion on this topic: https://forum.opnsense.org/index.php?topic=3696.0

So it seems that the described fix no longer works on 17.1.x :(
Any idea which change might have caused this breakage?

enc(4) was reworked for 11.0, which is probably why this happens. pfSense has patches that may be related, but I don't want to review things that are not in FreeBSD. IPsec is complicated as is :(

Guys not sure why i get notifications from here but can i be removed?

@ermal  there's an unsubscribe button at the top right next to the OP. github mentions are working more sturdily now than they used to ;)

@fraenki I tried forcing NATon IPsec device with local address via traffic from a different network, then added a gateway (code edits) to force the packet down the tunnel device, which seems to go through and the other end answers but the return path "hangs" when handing the payload back to the original network. this may be workable still. need to look into it more :)

@fichtner Thanks for taking the time to search for a solution. It's very unfortunate that this old issue now turns into a show stopper for us. :( I was OK maintaining our custom package, but non-working "NAT before IPsec" would kill many services on our side.

@fichtner thanks for the tip i always miss that button.
While here i might give you insight into it.
With FreeBSD 10 the best way is to use the patches i made since the other way of using nat+rdr is very tricky.
With FreeBSD 11-stable you have to be smart and use if_ipsec with routing and send needed traffic to that interface, nat there as usual with any other interface.
The only complication is that you have to manager routes with it.
Have fun and hope it helps.

@fraenki This really means OPN doesn't support scenarios with overlapping networks? :(

@AdSchellevis I digged into this and perhaps near to a solution. I did not enable any binat or something like that. My test setup is
LAN <IP_ADDRESS> - ROUTER - LAN <IP_ADDRESS> - OPNSENSE --- VPN --- SOPHOS - LAN <IP_ADDRESS>
So my VPN is <IP_ADDRESS> to <IP_ADDRESS> and I want to allow <IP_ADDRESS> without touching VPN.
What I did was:
kldload ipfw_nat.ko
ipfw nat 1 config ip <IP_ADDRESS> log reverse
ipfw add 180 nat 1 all from <IP_ADDRESS>/24 to <IP_ADDRESS>/24
Now I'm seeing the packets arriving at vtnet1 (LAN):
root@PB-FW1-FRA:~ # tcpdump -n -i vtnet1
22:22:02.368542 IP <IP_ADDRESS> > <IP_ADDRESS>: ICMP echo request, id 38914, seq 1792, length 8
And they go through enc0 to the tunnel and the reply is there:
root@PB-FW1-FRA:~ # tcpdump -n -i enc0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enc0, link-type ENC (OpenBSD encapsulated IP), capture size 262144 bytes
22:22:52.372686 (authentic,confidential): SPI 0x55c35db5: IP <IP_ADDRESS> > <IP_ADDRESS>: ICMP echo request, id 64031, seq 14592, length 8
22:22:52.381177 (authentic,confidential): SPI 0xca29fc32: IP <IP_ADDRESS> > <IP_ADDRESS>: ICMP echo reply, id 64031, seq 14592, length 8
The problem is, that the reply doesn't leave the interface. Isn't there any state tracking within ipfw? Or is there a second nat needed like <IP_ADDRESS> -> <IP_ADDRESS> -> <IP_ADDRESS> (so we would need 1:1 nat pool).
I think this is a good step forward without touching the code.

@mimugmail it has been quite some time ago that I did testing on this, but if I'm not mistaken the issue with pf looked similar, the kernel drops the return package.
If you can get it to work with ipfw, we should be able to integrate it into our current template, but so far I haven't seen a working configuration.

@AdSchellevis Thanks for confirming, I did some further debugging and playing around with sysctl, no luck. I'll try my luck at the mailing lists.

@fichtner You are playing around with a custom kernel right? Can you check if this topic is perhaps also related to: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=220217
I was asked at the freebsd-net list

it looks as bit like https://forum.opnsense.org/index.php?topic=5416.0 but maybe not relevant here. I will build a test kernel anyway. Thanks for bringing this up.

@AdSchellevis @fraenki
I have a solution now.
kldload ipfw_nat
ipfw nat 1 config ip
ipfw add 179 nat 1 log all from  to  out xmit enc0
ipfw add 179 nat 1 log all from  to  in recv enc0
setkey -PD | grep unique
Get the last number
setkey -v -c
spdadd -4   any -P out ipsec esp/tunnel/-/unique: ;
^D
setkey -PD
The only restriction is that you need to enable some kind of shaping in order to have basic ipfw ruleset enabled. @AdSchellevis any chance to get this templated? Perhaps we can also test with pf, but I'm quite unfamiliar with it.

@mimugmail I'm not sure it will work with pf, but it's certainly worth a try. as soon as I can find some time I will look into this and find a way to implement it. nice work!

I really like this doesn't affect pf-nat and strongswan... It could be a simple policy edit IPsec: Outbound NAT page, it just needs some merging with ipfw ruleset for shaper / cp.

it looks promising indeed

For reference NAT before ipsec using pf:
register our SA (in setkey -v -c):
spdadd -4 <my_local_network> <remote_network> any -P out ipsec esp/tunnel/<my identifier>-<remote identifier>/unique:<key from setkey -PD> ;
Next apply nat using pf (in /tmp/rules.debug):
nat on enc0 inet from <my_local_network> to <remote_network> -> <my_assigned_network> port 1024:65535
binat on enc0 inet from <my_local_network> to <remote_network> -><my_assigned_network>

Now we need to make sure we can add the nat setup using our normal nat rules, so we don't have to clutter our pf code any further (there's still a rewrite of the nat part on my list) and make sure we can register our SAD entry.

one addition, if I'm interpreting the documentation correctly (https://www.freebsd.org/cgi/man.cgi?query=setkey&sektion=8), we should be able to register all extra spd entries using:
spdadd -4 <my_local_network> <remote_network> any -P out ipsec esp/tunnel/<my identifier>-<remote identifier>/require;
Which skips the need of finding the correct id, so we should be able to generate a config file directly from the configuration we already know.
This seems to be working on my end.

For reference, setup the binat mapping (we should use either nat or binat, this only works on an equally sized network)

So we only need to bootstrap the additional spd entries and we're good to go.

What a beauty! Works perfectly! Thanks @AdSchellevis

Can also confirm require without ID also works! 👍

@mimugmail thanks! next steps are adding an option to the phase2 entries for additional spd entries and probably extend the one-to-one to support regular nat too (so you can also use different sized networks).

@mimugmail https://github.com/opnsense/core/commit/9351e45d59bd7cfd9e925f6bede4c2bf8156bc48 and https://github.com/opnsense/core/commit/814d18ac37ac477213fcacb43bb5e9920b92f2ff add the first version to add spd entries manually. it still needs a cleanup to record changes, but the rough idea is there.

@AdSchellevis Works great, thanks! 👍

@mimugmail nice team effort this is!
So, cleanup is in too https://github.com/opnsense/core/commit/1fe8341a19ccc4765373696d509f509df9889fcb
In case @fraenki wants to test too, you need to pull in the following changes (in this order):
https://github.com/opnsense/core/commit/9351e45d59bd7cfd9e925f6bede4c2bf8156bc48
https://github.com/opnsense/core/commit/814d18ac37ac477213fcacb43bb5e9920b92f2ff
https://github.com/opnsense/core/commit/1fe8341a19ccc4765373696d509f509df9889fcb
https://github.com/opnsense/core/commit/76839db73ad41b05f49144d0f963b0a150fab880
Last item on my list is to extend the one-on-one nat and then we should close this issue in my opinion.

I applied all patches and will test when I'm in the office.
I think @fraenki first has to update from 16.1 to 17.1 (or .7) for testing it :)

I'll add this to 17.7.1 if you all agree, as it doesn't seem to interfere with previous settings.

One-to-one NAT works also with different sized networks. Will there be further changes to the GUI? Then I'll document the steps and @jschellevis can import it to the docs.
As far as I can see this should also allow IPSEC with overlapping networks on both sides, will test this with a second lab ...

@mimugmail I think we're done, I don't have anything else planned there.

@fichtner 17.7.1 sound good to me, thanks!

@fichtner Would you mind to name @bu7cher (Andrey) in the Changelog as he did most of the research (it's ok for him)?

I'll mention in the backport, for the changelog the credits only name code contributions itself.

I can confirm that this NAT-before-IPsec implementation is indeed working for me. Great job, @AdSchellevis and @mimugmail!
There's one limitation that should be mentioned in the help text (and documentation):
root@opnsense:~ # /sbin/setkey -f /tmp/setkeyAsL4tW
libipsec: invalid IP address while parsing "host.example.com"
line 1: hostname nor servname provided, or not known at [ out ipsec esp/tunnel/host.example.com-<IP_ADDRESS>/require]
parse failed, line 1.

This means that it's required to use an IP address for "My identifier". Anything else, for example "Distinguished name", will not work. (Well, maybe it might work to get the IP address from the selected interface for those cases... but I'm not sure.)

@fraenki we can hide the option when not valid (! my ip or ip) or add a validation, what do you think?

@AdSchellevis I've opened issue #1773, because I've found another limitation and a possible fix.

Forum thread https://forum.opnsense.org/index.php?topic=7282.0 asks for official docs

I'll take this one ... long overdue :)


image.lut was moved to changed to image.cahedLut

It was changed to avoid deop when the lut typedarray was used to carry other values.
Looks like it is used again here :
https://github.com/chafey/cornerstone/blob/master/src/falseColorMapping.js#L56

Looks like this is fixed now. Thanks!


[FEATURE]: transform/alias raw row properties

Describe what you want
Hello, i'm currently using drizzle, but must rely on native postgres-js for features not currently supported by drizzle (e.g. triggers/function, publications, subscriptions).
While this works kinda fine, an issue i frequently face is the lack of transformation with this raw postgres usage.
Let assume i have a users table with: lastName: text(last_name)
When fetching data via drizzle last_name would be aliased to lastName and i can access the property via value.lastName on the result.
When i do e.g. a subscription via raw postgres-js
await client.subscribe("insert:users", (row) => {
    console.log(row.last_name); // <-- unaliased
});

the row properties will be unaliased and all the helper functions i built around the drizzle types will break.
Does drizzle somehow expose this transform / alias? I couldn't find anything on the docs.
I imagine a schema.transformToAliased(rawValue) to map from unaliased to aliased object properties would be quite helpful to workaround shortcomings on more complex db functionality not currently supported by drizzle.

This happens using supabase frequently.
I want to use supabase realtime but the events from it send back the raw rows. It would be great if I could run them through some helper function from drizzle so the typings match.

import { type InferSelectModel, type Table, getTableColumns } from 'drizzle-orm'
function toAliasedRow<T extends Table>(table: T, row: Record<string, unknown>) {
  const columns = getTableColumns(table)
  const newRow: Record<string, unknown> = {}
  for (const col in columns) {
    const _col = columns[col]
    newRow[col] = _col.mapFromDriverValue(row[_col.name])
  }
  return newRow as InferSelectModel<T>
}

I am using this snippet in my codebase, not sure if I implemented everything correctly.
should I open a pr?


Book music room

Current situation
Old C# project (working): https://github.com/AsWeSee/InnoMusicRoomBot.
Abandoned attempt to rewrite it in Python: https://github.com/Leon-Parepko/Inno-Music-Club-Bot.
DB - Postgres
Server - a computer with linux.
Registration:

Quick - InnoID or SSO
Standard - email + phone + surname + first name
Confirmation by mail

Functionality:

A person has an available number of hours per day and per week.
Person's status can be changed by admin
Generation of picture with schedule
Booking

Selection of booking day

For the next 7 days

Select booking time

The person writes by text

The number of hours is written off

Functionality in admin:

Ban
Deleting armour
Changing user status, adding/removing available hours

Additional features (optional):

Status of a person depending on donation in busti and patreon
Additional info in profile: e.g. hobbies; roles: bassist, keyboardist, etc.
Integration with discord server? (give out necessary roles)

Moved to https://github.com/one-zero-eight/InNoHassle-MusicRoom/issues/1


iOS: Documentation label is not read easily when there is a shape behind it

A picture is worth a thousand words:

This page is obsolete.


updates to meg validator

Please find some updates to the MEG schema, and also a few small ones to the BOLD

Thanks @robertoostenveld ! I cherry-picked your commits in my PR


Can't enable gpu on latest edge

Enabling NVIDIA GPU
Enabling DNS
Applying manifest
serviceaccount/coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created
clusterrole.rbac.authorization.k8s.io/coredns created
clusterrolebinding.rbac.authorization.k8s.io/coredns created
Restarting kubelet
DNS is enabled
Enabling Helm 3
Fetching helm version v3.5.0.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11.7M  100 11.7M    0     0   100M      0 --:--:-- --:--:-- --:--:--  100M
Helm 3 is enabled
Installing NVIDIA Operator
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /var/snap/microk8s/2093/credentials/client.config
"nvidia" has been added to your repositories
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /var/snap/microk8s/2093/credentials/client.config
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "nvidia" chart repository
Update Complete. ⎈Happy Helming!⎈
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /var/snap/microk8s/2093/credentials/client.config
Error: ClusterPolicy.nvidia.com "cluster-policy" is invalid: [spec.driver.repoConfig.destinationDir: Invalid value: "null": spec.driver.repoConfig.destinationDir in body must be of type string: "null", spec.driver.repoConfig.configMapName: Invalid value: "null": spec.driver.repoConfig.configMapName in body must be of type string: "null", spec.driver.licensingConfig.configMapName: Invalid value: "null": spec.driver.licensingConfig.configMapName in body must be of type string: "null"]

Not sure if this has been broken on our end or the Nvidia side.

https://github.com/NVIDIA/gpu-operator/issues/174

Workaround, for now, is to install 1.21/beta.

It does look like 1.21/beta is better, but I end up with a few Evicted pods, is that expected?  I wouldn't think so since my nodes don't seem to have a usable gpu when that happens.
I launched a g3.4xlarge instance on AWS with Ubuntu 20.04 and ran the following:
sudo snap install microk8s --channel=1.21/beta --classic
microk8s enable dns
microk8s enable gpu

This is what I had right after I did that:
gpu-operator-resources   pod/nvidia-driver-daemonset-hlw5v                                1/1     Running     0          3m26s
gpu-operator-resources   pod/nvidia-container-toolkit-daemonset-jsszp                     1/1     Running     0          3m
gpu-operator-resources   pod/nvidia-device-plugin-daemonset-hhc94                         1/1     Running     0          74s
gpu-operator-resources   pod/nvidia-device-plugin-validation                              0/1     Completed   0          50s
gpu-operator-resources   pod/nvidia-dcgm-exporter-r4jnk                                   0/1     Evicted     0          8s

ubuntu@ip-172-31-4-4:~$ microk8s kubectl get nodes -o yaml | grep -i gpu
      nvidia.com/gpu.present: "true"
      nvidia.com/gpu: "1"
      nvidia.com/gpu: "1"
      - nvcr.io/nvidia/gpu-operator@sha256:98878888117a2e3d53924c39f300e4263ce3a25ed7fb12125ed0eaf5d16ec03a
      - nvcr.io/nvidia/gpu-operator:1.6.2

But a few minutes later, I have this:
ubuntu@ip-172-31-4-4:~$ microk8s kubectl get pods -A
NAMESPACE                NAME                                                         READY   STATUS      RESTARTS   AGE
kube-system              calico-node-xpjf2                                            1/1     Running     0          16m
kube-system              calico-kube-controllers-847c8c99d-w72qc                      1/1     Running     0          16m
kube-system              coredns-86f78bb79c-n7n74                                     1/1     Running     0          15m
default                  gpu-operator-node-feature-discovery-master-dcf999dc8-dv9sv   1/1     Running     0          15m
gpu-operator-resources   nvidia-device-plugin-validation                              0/1     Completed   0          12m
gpu-operator-resources   nvidia-driver-daemonset-bb872                                0/1     Evicted     0          2m26s
gpu-operator-resources   nvidia-dcgm-exporter-pcftl                                   0/1     Evicted     0          2m56s
default                  gpu-operator-65d474cc8-rk5zq                                 0/1     Evicted     0          10m
default                  gpu-operator-65d474cc8-nlfwf                                 0/1     Pending     0          2m1s
gpu-operator-resources   nvidia-device-plugin-daemonset-5k6hb                         0/1     Evicted     0          111s
default                  gpu-operator-node-feature-discovery-worker-w6wqn             0/1     Evicted     0          80s
gpu-operator-resources   nvidia-container-toolkit-daemonset-l66bg                     0/1     Evicted     0          54s

ubuntu@ip-172-31-4-4:~$ microk8s kubectl get nodes -o yaml | grep -i gpu
      nvidia.com/gpu.present: "true"
      nvidia.com/gpu: "0"
      nvidia.com/gpu: "0"
      - nvcr.io/nvidia/gpu-operator@sha256:98878888117a2e3d53924c39f300e4263ce3a25ed7fb12125ed0eaf5d16ec03a
      - nvcr.io/nvidia/gpu-operator:1.6.2

It seems to be caused by disk pressure:
ubuntu@ip-172-31-4-4:~$ microk8s kubectl describe pod gpu-operator-node-feature-discovery-worker-w6wqn
Name:           gpu-operator-node-feature-discovery-worker-w6wqn
Namespace:      default
Priority:       0
Node:           ip-172-31-4-4/
Start Time:     Tue, 23 Mar 2021 12:18:53 +0000
Labels:         app.kubernetes.io/component=worker
                app.kubernetes.io/instance=gpu-operator
                app.kubernetes.io/name=node-feature-discovery
                controller-revision-hash=c4748898b
                pod-template-generation=1
Annotations:    <none>
Status:         Failed
Reason:         Evicted
Message:        Pod The node had condition: [DiskPressure].
IP:
IPs:            <none>
Controlled By:  DaemonSet/gpu-operator-node-feature-discovery-worker
Containers:
  node-feature-discovery-master:
    Image:      quay.io/kubernetes_incubator/node-feature-discovery:v0.6.0
    Port:       <none>
    Host Port:  <none>
    Command:
      nfd-worker
    Args:
      --sleep-interval=60s
      --server=gpu-operator-node-feature-discovery:8080
    Environment:
      NODE_NAME:   (v1:spec.nodeName)
    Mounts:
      /etc/kubernetes/node-feature-discovery/ from nfd-worker-config (rw)
      /etc/kubernetes/node-feature-discovery/features.d/ from features-d (rw)
      /etc/kubernetes/node-feature-discovery/source.d/ from source-d (rw)
      /host-boot from host-boot (ro)
      /host-etc/os-release from host-os-release (ro)
      /host-sys from host-sys (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5lshg (ro)
Volumes:
  host-boot:
    Type:          HostPath (bare host directory volume)
    Path:          /boot
    HostPathType:
  host-os-release:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/os-release
    HostPathType:
  host-sys:
    Type:          HostPath (bare host directory volume)
    Path:          /sys
    HostPathType:
  source-d:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/node-feature-discovery/source.d/
    HostPathType:
  features-d:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/node-feature-discovery/features.d/
    HostPathType:
  nfd-worker-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      gpu-operator-node-feature-discovery
    Optional:  false
  kube-api-access-5lshg:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
                             nvidia.com/gpu=present:NoSchedule
Events:
  Type     Reason     Age    From               Message
  ----     ------     ----   ----               -------
  Normal   Scheduled  7m29s  default-scheduler  Successfully assigned default/gpu-operator-node-feature-discovery-worker-w6wqn to ip-172-31-4-4
  Warning  Evicted    7m30s  kubelet            The node had condition: [DiskPressure].

FYI, I found the recipe. I'm using microk8s on Ubuntu 20.04, here's what I had to do to make this work on both a Nvidia DGX A100 as well as a ProLiant DL380 Gen10 machine with T4 GPU.

Make sure to start on a fresh Ubuntu 20.04 with no nvidia drivers (or apt purge the nvidia packages if you can't)
blacklist the nouveau driver (add modprobe.blacklist=nouveau nouveau.modeset=0 to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub and run sudo update-grub and reboot)
[for the A100 only] Install the package nvidia-fabricmanager-460 from the cuda repos (you won't be able to enable the systemd service until the K8s GPU operator has loaded the drivers)
Install microk8s with the 1.21/beta channel (I have version v1.21.0-beta.1)
microk8s enable dns (tip: make sure your DNS is working by launching a test pod and resolving internal and external hostnames)
microk8s enable gpu
[for the A100 only] enable nvidia-fabricmanager (sudo systemctl --now enable nvidia-fabricmanager)

Thank you for digging into this @davecore82 .
Would you be interested in putting the steps you followed and your findings in a topic over at https://discuss.kubernetes.io/ so we can have it as part of our docs at microk8s.io?
@evilnick wdyt?

@ktsakalozos  I did a write up on my adventures with microk8s and MIG on the A100   https://discuss.kubernetes.io/t/my-adventures-with-microk8s-to-enable-gpu-and-use-mig-on-a-dgx-a100/15366


Split almighty-core master job into 2; test/integration and code coverage

We need a code coverage run on the master branch on merge to see comparison values on PR, but there is no need for code coverage to delay the rollout of a deployment.
No need to run code coverage as part of deployment job and no need to build deploy image in code coverage job.

Duplicated by https://github.com/almighty/almighty-core/issues/759


eto: use overrides instead of eventhandlers

I think this is preferable (?)

I too think this is the way to go.

Yes, that's better, and could be changed in many places.
@inforithmics Thanks for taking care of the eto reviews.


Audit.Ended value is not set for events coming through RequestPipelineDiagnosticObserver

NEST/Elasticsearch.Net version: 7.11.1
Elasticsearch version: 7.11.1
Description of the problem including expected versus actual behavior:
Diagnostic events that are coming through RequestPipelineDiagnosticObserver have their Audit.Ended values not set.
Steps to reproduce:

Provide diagnostic listener and subscribe to DiagnosticSources.RequestPipeline.SourceName event using RequestPipelineDiagnosticObserver with an Action for onNextEnd.
Create a new instance of ElasticLowLevelClient and make a request to Elastic, i.e. IndexAsync.
Check incoming IApiCallDetails payloads and their AuditTrail objects.

Expected behavior
Audit.Ended value should be set correctly.

I'm the maintainer of https://github.com/romansp/MiniProfiler.Elasticsearch. My little plugin pushes audit events from ElasticClient or (its low-level version) into MiniProfiler as custom timings.
I would like to provide one more way of integrating via diagnostic listeners. It works fine but incoming AuditTrail events don't have a correct Ended value so I can't compute how much time did request take.
I did some digging into transport implementation here: https://github.com/elastic/elastic-transport-net/blob/85c235850a126a161560d4754941db377ca1c7e9/src/Elastic.Transport/Components/Pipeline/RequestPipeline.cs#L173. Seems that IApiCallDetails is pushed into diagnostics observables before Auditable gets disposed thus its Ended isn't set yet.
Though events that are coming through ConnectionSettings.OnRequestCompleted are correct.

Silly repro
internal class TestDiagnosticListener : IObserver<DiagnosticListener>, IDisposable {
  private ConcurrentBag<IDisposable> Disposables { get; } = new ConcurrentBag<IDisposable>();

  public Action<IApiCallDetails> OnEnded { get; }

  public TestDiagnosticListener(Action<IApiCallDetails> onEnded) {
      OnEnded = onEnded;
  }

  public void OnError(Exception error) { }
  public void OnCompleted() { }

  public void OnNext(DiagnosticListener value) {
      TrySubscribe(DiagnosticSources.RequestPipeline.SourceName,
          () => new RequestPipelineDiagnosticObserver(null, v => OnEnded(v.Value)), value);
  }

  private void TrySubscribe(string sourceName, Func<IObserver<KeyValuePair<string, object>>> listener, DiagnosticListener value) {
      if (value.Name != sourceName) return;
      var d = value.Subscribe(listener());

      Disposables.Add(d);
  }

  public void Dispose() {
      foreach (var d in Disposables) {
          d.Dispose();
      }
  }
}

[Fact]
public async Task DiagnosticListener_AuditTrailIsValid() {
  // Arrange
  using var listener = new TestDiagnosticListener((data) => {
      var auditTrailEvent = data.AuditTrail[0];
      Assert.True((auditTrailEvent.Ended - auditTrailEvent.Started) >= TimeSpan.Zero);
  });
  using var foo = DiagnosticListener.AllListeners.Subscribe(listener);
  var connectionPool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
  var settings = new ConnectionConfiguration(connectionPool, new InMemoryConnection());

  var client = new ElasticLowLevelClient(settings);
  var person = new { Id = "1" };

  await client.IndexAsync<BytesResponse>("test-index", PostData.Serializable(person));
}

Apologies for the delay @romansp. I'll try to carve some time to look into this during the week. Thanks for reporting it.

@romansp I have a PR in which will resolve this. It makes sense that our Auditable wrapper expose a way to stop the Audit and set the end time more explicitly, rather than just on disposal.

PR is merged for 7.12.0 release.


Redundant white spaces in class name

In case of multiple class names sometimes there are unnecessary white spaces which increases the size of the output:
class="l y  wb xb"

Fixed by #9


Fixes from m_reza_rahman

I just submitted the OCA. It should be simple enough to do and I think minimizes hassle going forward.

@edburns, this is now ready to move forward. I have done at least one pass over everything.

@galiacheng is going to push some more commits to fix the problems you identified, then we will advocate for merging this.

@edburns @rjeberhard The PR is ready for review. Please note that this PR introduces changes to the sample script. Could you please refresh the sample-scripts.zip in v4.2.8?


Add new step :put_aws_sigv4

Closes #62

Given aws_sigv4 is not as widely used as I hoped (e.g. by other cloud services) I'm going to close this for now but will definitely keep it in mind going forward especially as we add other AWS integrations. I'm very keen on revisiting it. Thank you @aleDsz!


Add possibility to represent modules as C4 Components

archifacts models modules of a monolith as containers. Due to this converation with @simonbrowndotje they should be modelled as components.
Unfortunately, the Structurizr Java API is limited and does not provide all the features we need to model modules as components appropriately.
As an alternative we could think about generating PlantUML directly using the C4-PlantUML-Stdlib or generating Structurizr DSL Code.
At first glance the DSL approach seems to be the better choice as it abstracts from the concrete technology (PlantUML).
It seems to be possible to parse the generated DSL files using StructurizrDslParser.
As part of this issue it should be possible to decide how modules should be represented in the C4 model.
archifacts should support representing them as containers (distributed system) or components (modular monolith).

Unfortunately, the Structurizr Java API is limited and does not provide all the features we need to model modules as components appropriately.

At first glance the DSL approach seems to be the better choice as it abstracts from the concrete technology (PlantUML).

The DSL parser uses the Java API, so anything you can do with the DSL is possible with the Java API. As an example, the parser for the component DSL keyword just calls container.addComponent() -> https://github.com/structurizr/dsl/blob/master/src/main/java/com/structurizr/dsl/ComponentParser.java

Thanks for clarifying, @simonbrowndotje.
Well, after investigating the PlantUML-Stdlib I realized that Codeelements are not part of the lib, too. So it feels like both of my ideas don't work.
So there is specific visual C4 representation for code elements and the relationships between them, right? A classic UML class diagram for the rescue?

So there is no specific visual C4 representation for code elements and the relationships between them, right? A classic UML class diagram for the rescue?

There is no "specific visual C4 representation" for any of the C4 model diagrams (the C4 model is notation independent), so yes, a UML class diagram is a good approach for level 4 -> https://c4model.com/#CodeDiagram


[BUG] 61.0.0 breaks "setup.py install" by missing dependencies in install_requires

setuptools version
61.0.0
Python version
3.10
OS
ubuntu-latest in GitHub Actions
Additional environment information
No response
Description
In our pywbem package, we test the different documented methods to install the package. One of them is still "setup.py install". I do understand that it is deprecated since setuptools v58.3.0, but that still means it should be expected to work.
The issue is that the "setup.py install" command successfully installs the pywbem package, but upon import it turns out that dependent packages were not installed, e.g. "six".
The GitHub Actions test run showing that is: https://github.com/pywbem/pywbem/runs/5687824533?check_suite_focus=true
"pip install" on the package works with setuptools 61.0.0 and does install the dependent packages.
The requirements.txt file of the pywbem project does specify the dependent packages and the setup.py script loads the dependencies from requirements.txt and specifies them in the "install_requires" parameter of setup().
The "setup.py install" approach worked before setuptools 61.0.0, e.g. with 60.10.0, see this test run.
Expected behavior
61.0.0 should still support "setup.py install" as before.
How to Reproduce

Create a new virtual env on Python 3.10
pip install setuptools==61.0.0; pip uninstall six
git clone https://github.com/pywbem/pywbem.git; cd pywbem
./setup.py install  # should succeed
pip list  # should display:Package    Version
---------- ----------
pip        22.0.4
pywbem     1.5.0.dev1
setuptools 61.0.0
wheel      0.37.1

and the issue is that the dependent packages are not installed.
pip install .  # should succeed
pip list  # should display the correctly installed dependencies:Package            Version
------------------ ----------
certifi            2021.10.8
charset-normalizer 2.0.12
idna               3.3
nocasedict         1.0.2
nocaselist         1.0.4
pip                22.0.4
ply                3.11
pywbem             1.5.0.dev1
PyYAML             6.0
requests           2.27.1
setuptools         61.0.0
six                1.16.0
urllib3            1.26.9
wheel              0.37.1
yamlloader         1.1.0

Output
See above

Probably related to #3196

I had the same problem this morning since I upgraded to setuptools 61.0.0 (no dependencies installed).
The problem is clearly coming from the last release because if we do :
pip install --upgrade setuptools==60.10.0
everything works fine!

Hi @andy-maier, thank you very much for reporting that.
I checked pywbem against the fix proposed for 3196, and it seems that the issue might be different. I am still investigating the problem.

Similar problem -- 61.0.0 broke a test in the pipenv CI that checks that when installing this setup.py it gets the sibling package which it no longer does.
https://github.com/techalchemy/test-project/blob/master/parent_folder/pep508-package/setup.py#L15

@abravalheri The pipenv CI just pulled in 61.1.1 but it still seems to be failing on that setupy.py line from my prior comment, ref:  https://github.com/pypa/pipenv/runs/5703684724?check_suite_focus=true
Note: the python 3.6 build is passing because it cannot install such a new setuptools, and despite my best efforts to pin the setup.py, CI, and run-tests.sh it was still pulling in the latest setuptools, perhaps pip was doing this.
Please let me know if I should open a new ticket for this issue or if you have any other suggestions.

Thanks @matteius, I will have a look.

Hi @matteius, please note that the problem seems to have been happening since before 61.0.0:
% git clone https://github.com/techalchemy/test-project
% cd test-project
% virtualenv .venv
% .venv/bin/python -m pip install -U pip setuptools==60.10.0 wheel
% .venv/bin/python -m pip list
Package    Version
---------- -------
pip        22.0.4
setuptools 60.10.0
wheel      0.37.1
% .venv/bin/python setup.py install
...
Searching for sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package
...
Couldn't find index page for 'sibling_package' (maybe misspelled?)
...
No local packages or working download links found for sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package
error: Could not find suitable distribution for Requirement.parse('sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package')

Do you known when was the last time this exact test worked before? Which was the version of setuptools were you using?
BTW, I think the issue you are facing is not the one reported originally here... I am even under the impression that setuptools never did manage requirements URL in the form of req @ git+https://... (but I might be wrong here).

@abravalheri  the test was passing three days ago.   I think the reason pinning setup tools doesn't fix it is because internal pip is installing the latest setup tools anyway I think.  If you look at the Python 3.6 build it is still passing because it will never install the new version of setup tools since it doesn't support py3.6.   I am skiing today but can look more into it tomorrow.  I do believe it is caused by setup tools upgrade at this point.

Hi @matteius please do enjoy your leisure time, we can have a look at this later :)

I will just add some notes here so when you have the time you can have a look.

I do believe it is caused by setup tools upgrade at this point.

That is completely possible. It might very well be that v61.0.0...v61.1.1 introduced something that is causing the problem. However this seems to be a second issue, and not the one originally reported here.

I think the reason pinning setup tools doesn't fix it is because internal pip is installing the latest setup tools anyway.

I don't understand what you mean here. The test I did in my machine does not use pip at all. The original problem mentioned in this issue is related to invoking python setup.py install, so I tried to replicate it locally with the repository you linked. python setup.py install does not use pip (it uses easy_install internally).
If I instead run:
% git clone https://github.com/techalchemy/test-project
% cd test-project/parent_folder/pep508-package
% virtualenv .venv
% .venv/bin/python -m pip install -U pip setuptools==61.1.1 wheel
% .venv/bin/python -m pip list
Package    Version
---------- -------
pip        22.0.4
setuptools 61.1.1
wheel      0.37.1
% .venv/bin/python -m pip install -e .
...
  Running command git clone --filter=blob:none --quiet https://github.com/techalchemy/test-project.git /tmp/pip-install-fapb6ksm/sibling-package_2bc442202d024507bd538a882a35d28a
...
Successfully installed pep508-package-1.0.0 sibling_package-1.0.0 six-1.16.0 toml-0.10.2 urllib3-1.26.9
# I also tried `pip install .` and the outcome is the same

Then both packages seem to install fine.
If instead I run (replacing install with develop:
% rm -rf .venv
% .venv/bin/python -m pip install -U pip setuptools==60.10.0 wheel
% .venv/bin/python -m pip list
Package    Version
---------- -------
pip        22.0.4
setuptools 60.10.0
wheel      0.37.1
% .venv/bin/python setup.py develop
...
Couldn't find index page for 'sibling_package' (maybe misspelled?)
...
No local packages or working download links found for sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package
error: Could not find suitable distribution for Requirement.parse('sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package')

Then I see the same problem with v60.10.0
If you manage to find a minimal reproducer along the lines of the examples before that works for setuptools==60.10.0 but does not work for setuptools==61.1.1 that would help a lot to understand this problem.

BTW, I am testing on:
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.4 LTS
Release:        20.04
Codename:       focal

Python 3.8.10 (default, Nov 26 2021, 20:14:08)
[GCC 9.3.0]

@abravalheri Thanks for the helpful feedback.  I am back now and trying to figure this out again.  So pip uses setuptools, and following this pip documentation:
lead me to a working pip install command:
pip install -e "git+https://github.com/techalchemy/test-project.git@master#egg=pep508_package&subdirectory=parent_folder/pep508-package"
matteius@matteius-VirtualBox:~/pipenv-triage/setuptools-3198/test-project/parent_folder/pep508-package$ pip install -e "git+https://github.com/techalchemy/test-project.git@master#egg=pep508_package&subdirectory=parent_folder/pep508-package"
Defaulting to user installation because normal site-packages is not writeable
Obtaining pep508_package from git+https://github.com/techalchemy/test-project.git@master#egg=pep508_package&subdirectory=parent_folder/pep508-package
  Updating ./src/pep508-package clone (to revision master)
  Running command git fetch -q --tags
  Running command git reset --hard -q 4f7b2b05e0c99d79ece1e8cfe5411a0e5a478648
  Preparing metadata (setup.py) ... done
Collecting sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package
  Cloning https://github.com/techalchemy/test-project.git (to revision master) to /tmp/pip-install-abye4d5v/sibling-package_0bce244a0b6649faa4d3ff6d1f7e36d0
  Running command git clone --filter=blob:none --quiet https://github.com/techalchemy/test-project.git /tmp/pip-install-abye4d5v/sibling-package_0bce244a0b6649faa4d3ff6d1f7e36d0
  Resolved https://github.com/techalchemy/test-project.git to commit 4f7b2b05e0c99d79ece1e8cfe5411a0e5a478648
  Preparing metadata (setup.py) ... done
Requirement already satisfied: six in /home/matteius/.local/lib/python3.9/site-packages (from pep508_package) (1.16.0)
Requirement already satisfied: toml in /home/matteius/.local/lib/python3.9/site-packages (from sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package->pep508_package) (0.10.2)
Requirement already satisfied: urllib3 in /home/matteius/.local/lib/python3.9/site-packages (from sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package->pep508_package) (1.26.8)
Building wheels for collected packages: sibling_package
  Building wheel for sibling_package (setup.py) ... done
  Created wheel for sibling_package: filename=sibling_package-1.0.0-py3-none-any.whl size=1518 sha256=73436892f4004a44600263189c2da681355968693c5417dd29f7bcd84b1abeb6
  Stored in directory: /tmp/pip-ephem-wheel-cache-3_di4jil/wheels/de/21/b8/e6c31ccdbf6afe5ab53d4ae0a0ae80a4f93c29e14828123eb6
Successfully built sibling_package
Installing collected packages: sibling_package, pep508_package
  Attempting uninstall: pep508_package
    Found existing installation: pep508-package 1.0.0
    Uninstalling pep508-package-1.0.0:
      Successfully uninstalled pep508-package-1.0.0
  Running setup.py develop for pep508_package
Successfully installed pep508_package-1.0.0 sibling_package-1.0.0

Now I try again with setuptools-61.0.0 ... it also seems to work.
matteius@matteius-VirtualBox:~/pipenv-triage/setuptools-3198/test-project/parent_folder/pep508-package$ pip install setuptools==61.0.0
Defaulting to user installation because normal site-packages is not writeable
Collecting setuptools==61.0.0
  Downloading setuptools-61.0.0-py3-none-any.whl (1.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 5.6 MB/s eta 0:00:00
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 59.8.0
    Uninstalling setuptools-59.8.0:
      Successfully uninstalled setuptools-59.8.0
Successfully installed setuptools-61.0.0

:atest setuptools does seem to work with pip in isolation, and here is an example with no setuptools first that show it is in fact setuptools it is using to do the install.   So now I am very perplexed how our test started failing with no code changes.
matteius@matteius-VirtualBox:~/pipenv-triage/setuptools-3198/test-project/parent_folder/pep508-package$ pip install -e "git+https://github.com/techalchemy/test-project.git@master#egg=pep508_package&subdirectory=parent_folder/pep508-package"
Defaulting to user installation because normal site-packages is not writeable
Obtaining pep508_package from git+https://github.com/techalchemy/test-project.git@master#egg=pep508_package&subdirectory=parent_folder/pep508-package
  Updating ./src/pep508-package clone (to revision master)
  Running command git fetch -q --tags
  Running command git reset --hard -q 4f7b2b05e0c99d79ece1e8cfe5411a0e5a478648
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
matteius@matteius-VirtualBox:~/pipenv-triage/setuptools-3198/test-project/parent_folder/pep508-package$ pip install setuptools --user
Collecting setuptools
  Using cached setuptools-61.1.1-py3-none-any.whl (1.1 MB)
Installing collected packages: setuptools
Successfully installed setuptools-61.1.1
matteius@matteius-VirtualBox:~/pipenv-triage/setuptools-3198/test-project/parent_folder/pep508-package$ pip install -e "git+https://github.com/techalchemy/test-project.git@master#egg=pep508_package&subdirectory=parent_folder/pep508-package"
Defaulting to user installation because normal site-packages is not writeable
Obtaining pep508_package from git+https://github.com/techalchemy/test-project.git@master#egg=pep508_package&subdirectory=parent_folder/pep508-package
  Updating ./src/pep508-package clone (to revision master)
  Running command git fetch -q --tags
  Running command git reset --hard -q 4f7b2b05e0c99d79ece1e8cfe5411a0e5a478648
  Preparing metadata (setup.py) ... done
Collecting sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package
  Cloning https://github.com/techalchemy/test-project.git (to revision master) to /tmp/pip-install-gkrre310/sibling-package_3cbedeaea7da4caabd0f8df1b4384bbf
  Running command git clone --filter=blob:none --quiet https://github.com/techalchemy/test-project.git /tmp/pip-install-gkrre310/sibling-package_3cbedeaea7da4caabd0f8df1b4384bbf
  Resolved https://github.com/techalchemy/test-project.git to commit 4f7b2b05e0c99d79ece1e8cfe5411a0e5a478648
  Preparing metadata (setup.py) ... done
Requirement already satisfied: six in /home/matteius/.local/lib/python3.9/site-packages (from pep508_package) (1.16.0)
Requirement already satisfied: toml in /home/matteius/.local/lib/python3.9/site-packages (from sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package->pep508_package) (0.10.2)
Requirement already satisfied: urllib3 in /home/matteius/.local/lib/python3.9/site-packages (from sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package->pep508_package) (1.26.8)
Installing collected packages: pep508_package
  Attempting uninstall: pep508_package
    Found existing installation: pep508-package 1.0.0
    Uninstalling pep508-package-1.0.0:
      Successfully uninstalled pep508-package-1.0.0
  Running setup.py develop for pep508_package
Successfully installed pep508_package-1.0.0

Thanks for your help, I am going to research if any other related packages would have bumped around the same time.

Hi @matteius

here is an example with no setuptools first that show it is in fact setuptools it is using to do the install

Please note that when you run pip install -e ., a few separated steps will be executed (it is not like pip delegates everything to setuptools):

Pip uses setuptools to list the packages dependencies (I might be wrong, but I think it does this via setuptools.build_meta.get_requires_for_build_wheel)
Pip installs the dependencies
Pip runs python setup.py develop

Setuptools itself is not able to handle VCS URLs as dependencies.
If python setup.py develop or python setup.py install is used directly, without pip adding the dependencies first, then the editable installation is expected to fail, as shown in my previous comments.
Any chance pip install -e . got replaced somewhere to python setup.py develop/install?

Any chance pip install -e . got replaced somewhere to python setup.py develop/install?

Doesn't seem to be -- I put a bunch of time into this last night and the perplexing thing is the breakage happened without any changes being pushed to pipenv repository.   I have summarized some additional notes here, as there seems to be a similar issue encountered in pip-tools (which we do not rely upon, but whatever upstream thing it is seemed to come about at the same time for both projects):  https://github.com/jazzband/pip-tools/issues/1603#issuecomment-1079831628
The telling thing to me is that our python 3.6 continues to pass for this test case, and I can see in the CI that is using an older version of setuptools because the newer versions aren't python 3.6 compatible.   Despite my best efforts to pin setuptools for the other python versions, it seems to all be for not -- I think pip is bringing in the new version anyway when it goes to resolve.    https://github.com/pypa/pipenv/runs/5708004672?check_suite_focus=true#step:7:38

Pip uses setuptools to list the packages dependencies (I might be wrong, but I think it does this via setuptools.build_meta.get_requires_for_build_wheel)

Yes well at least pipenv does this, it is my latest understanding too it uses that method in some places as well as prepare_metadata_for_build_wheel.
I am continuing to investigate.

@abravalheri I have been comparing internals of the last pipenv version that still works for this case on newer python versions 2021.5.29 with the newer versions.   There is a code path that gets setup info, and what I see is in the working copy, we are getting in the metadata a valid version and extras attribute, but in the newer versions they are both empty in the metadata:
SETUP_INFO NOW: SetupInfo(name='pep508_package', base_dir='/tmp/pipenv-fq_39y0k-src/pep508_package', _version='0.0.0',
_requirements=frozenset(), build_requires=(), build_backend='setuptools.build_meta:__legacy__', setup_requires=(),
python_requires=None, _extras_requirements=(), setup_cfg=PosixPath('/tmp/pipenv-fq_39y0k-src/pep508_package/parent_folder/pep508-package/setup.cfg'),
setup_py=PosixPath('/tmp/pipenv-fq_39y0k-src/pep508_package/parent_folder/pep508-package/setup.py'),
pyproject=PosixPath('/tmp/pipenv-fq_39y0k-src/pep508_package/pyproject.toml'),
ireq=<InstallRequirement object: pep508_package from git+https://github.com/techalchemy/test-project.git@master#egg=pep508_package&subdirectory=parent_folder/pep508-package editable=False>,
extra_kwargs={'build_dir': '/tmp/reqlib-buildlhyan8qz', 'src_dir': '/tmp/pipenv-fq_39y0k-src/pep508_package',
'download_dir': '/home/matteius/.cache/pipenv/pkgs', 'wheel_download_dir': '/home/matteius/.cache/pipenv/wheels'},
metadata=('name', 'parent-folder.sibling-package.src.sibling-package', 'version', '0.0.0', 'requires', ()),
stack=<contextlib.ExitStack object at 0x7f2e0a77c580>, _finalizer=<finalize object at 0x7f2e0a7b2a00; for 'SetupInfo' at 0x7f2e0a703510>)

SETUP INFO BEFORE: SetupInfo(name='pep508_package', base_dir='/tmp/pipenv-3c06bgxi-src/pep508_package', _version='1.0.0',
_requirements=frozenset({BaseRequirement(name='sibling-package',
requirement=Requirement.parse('sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package')),
BaseRequirement(name='six', requirement=Requirement.parse('six'))}), build_requires=('setuptools', 'wheel'),
 build_backend='setuptools.build_meta:__legacy__', setup_requires=(), python_requires=None, _extras_requirements=(),
 setup_cfg=PosixPath('/tmp/pipenv-3c06bgxi-src/pep508_package/parent_folder/pep508-package/setup.cfg'),
 setup_py=PosixPath('/tmp/pipenv-3c06bgxi-src/pep508_package/parent_folder/pep508-package/setup.py'),
 pyproject=PosixPath('/tmp/pipenv-3c06bgxi-src/pep508_package/pyproject.toml'),
 ireq=<InstallRequirement object: pep508_package from git+https://github.com/techalchemy/test-project.git@master#egg=pep508_package&subdirectory=parent_folder/pep508-package editable=False>,
 extra_kwargs={'build_dir': '/tmp/reqlib-buildfog79g70', 'src_dir': '/tmp/pipenv-3c06bgxi-src/pep508_package', 'download_dir': '/home/matteius/.cache/pipenv/pkgs',
 'wheel_download_dir': '/home/matteius/.cache/pipenv/wheels'},
 metadata=('name', 'pep508-package', 'version', '1.0.0',
 'requires', (Requirement.parse('six'), Requirement.parse('sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package'), Requirement.parse('six'), Requirement.parse('sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package')), ('dev', (Requirement.parse('parver'), Requirement.parse('invoke'), Requirement.parse('wheel'))), ('testing', (Requirement.parse('coverage'), Requirement.parse('flaky')))),
 stack=<contextlib.ExitStack object at 0x7fb97e62b730>, _finalizer=<finalize object at 0x7fb97e6af200; for 'SetupInfo' at 0x7fb97e717f60>)

This is unfortunately taking me hours to debug, and will take hours more I am sure to try and unravel what is going on.

Hi @matteius, let me see if I can help you somehow.
You mentioned:

There is a code path that gets setup info, and what I see is in the working copy, we are getting in the metadata a valid version and requires attribute, but in the newer versions they are both empty in the metadata

I am assuming you are getting this info either from the (a) a generated wheel, (b) a generated sdist or (c) you run egg_info and inspect the generated metadata. Anyhow, is there a snippet of code that I could try to replicate this without using the internals of pipenv?

I wonder if it the metadata has empty version and requires because the name is all weird in the new version 'parent-folder.sibling-package.src.sibling-package'

This is comment intrigued me a little, because it might be related to another change in v61.*.
If you are trying to install parent-folder as if it was a package (note that parent-folder does not contain setup.py or pyproject.toml), previously setuptools would generate a UNKNOWN package, but now with the new name discovery it will indeed create parent-folder.sibling-package.src.sibling-package (the details of this mechanic were first proposed in https://github.com/pypa/setuptools/issues/2887#issuecomment-970708430)
This happens because parent_folder/sibling_package/src/sibling_package is accidentally a valid package under the eyes of PEP 420. For example, if you are inside the test-folder, you can fire up the REPL and successfully run:
>>> import parent_folder.sibling_package.src.sibling_package

I have the impression that the name parent-folder.sibling-package.src.sibling-package is a valid under PEP 518.

Anyhow, is there a snippet of code that I could try to replicate this without using the internals of pipenv?

Unfortunately not yet and I may need to take a break from this for a while due to other obligations, I know the pip-tools project has a similar issue reported but no reproduction steps there either.   Pipenv and pip pull in the latest setuptools like its candy, all over the place, so its really hard to restrict it back but I have done some of that and see some behavioral changes, but not enough to get the end result.

If you are trying to install parent-folder as if it was a package (note that parent-folder does not contain setup.py or pyproject.toml), previously setuptools would generate a UNKNOWN package, but now with the new name discovery it will indeed create parent-folder.sibling-package.src.sibling-package

The thing is I am using this url git+https://github.com/techalchemy/test-project.git@master#egg=pep508_package&subdirectory=parent_folder/pep508-package which calls out the subdirectory of the actual package name to use -- it should therefore not be trying to use parent_folder.  This is valid under the URL Fragments section of the pip docs here:  https://pip.pypa.io/en/stable/topics/vcs-support/#url-fragments
Pip is using setuptools under the hood for this, so I think that is where the problem enters, because pipenv vendors in pip, so I am deep in the internals of pip code and the resolvelib code trying to figure out how this changed without us changing anything.

On a side note: if pipenv relies on the fact that projects may have name="UNKNOWN", that might no longer be true in v61.*, the new name discovery algorithm will try to always quick in.

UNKNOWN

We aren't directly relying on a name of UNKNOWN, I think when this actually works, it uses the right name from the #egg and the right subdirectory as well, yielding a package like pep508_package-1.0.0-py3-none-any.whl which matches the name from the Pipfile.

@abravalheri I have narrowed my test example down to a single factor, In requirementslib/models/setup_info.py if I modify the method build_pep517 to do this extra hack at the beginning (the hack has knowledge of the package sub-directories`:
def build_pep517(source_dir, build_dir, config_settings=None, dist_type="wheel"):
    if 'pep508_package' in source_dir or 'pep508-package' in source_dir:
        source_dir = f"{source_dir}/parent_folder/pep508-package"

When I have that extra logic enabled, and some extra print statements, I can see that this gets built: parent_folder.sibling_package.src.sibling_package-0.0.0-py3-none-any.whl

build pep517 base_dir /tmp/pipenv-ln7c16i9-src/pep508_package
pep517_config {'--global-option': []}
result of build_pep517: parent_folder.sibling_package.src.sibling_package-0.0.0-py3-none-any.whl

However, with my hack in place, the output instead is:

build pep517 base_dir /tmp/pipenv-2xqe9gf8-src/pep508_package
pep517_config {'--global-option': []}
result of build_pep517: pep508_package-1.0.0-py3-none-any.whl

This makes all the difference in the world to get the expected result of:

[{"version": "1.16.0", "hashes": ["sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926", "sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254"], "name": "six", "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2'"}, {"subdirectory": "parent_folder/pep508-package", "ref": "4f7b2b05e0c99d79ece1e8cfe5411a0e5a478648", "git": "https://github.com/techalchemy/test-project.git", "name": "pep508-package"}, {"subdirectory": "parent_folder/sibling_package", "ref": "4f7b2b05e0c99d79ece1e8cfe5411a0e5a478648", "git": "https://github.com/techalchemy/test-project.git", "name": "sibling-package"}, {"version": "1.16.0", "hashes": ["sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926", "sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254"], "name": "six", "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'"}, {"subdirectory": "parent_folder/pep508-package", "ref": "4f7b2b05e0c99d79ece1e8cfe5411a0e5a478648", "git": "https://github.com/techalchemy/test-project.git", "version": "", "name": "pep508-package"}, {"subdirectory": "parent_folder/sibling_package", "ref": "4f7b2b05e0c99d79ece1e8cfe5411a0e5a478648", "git": "https://github.com/techalchemy/test-project.git", "version": "", "name": "sibling-package"}]

However, I am still head scratching about what could have changed to cause it to no longer automatically get the subdirectory on the path in this step, or why it would have changed behaviors ~4 days ago.

Hi @matteius this might be caused by the change we have been discussing previously...
Am I correct to assume that this functions is being called with source_dir set as the parent directory instead of the directory that actually contains the package being build? If that is the case, why is it being called to build a directory that does not contain either setup.py or pyproject.toml?
Previous versions of setuptools would create an UNKOWN package. But now the name will be inferred to something different...

@abravalheri Thanks for being supportive and helping me asses what is going on.   Quick question -- Is it possible that somehow now in the Metadata is now lacking the subdirectory where before it might have included something about it?   We have this logic also in requirementslib that tries to pull the subdirectory from the requirement, but its always None ... It is hard to tell where setuptools begins and ends in some of this code, so I apologize for referencing things you may not be familiar with
    @classmethod
    def from_requirement(cls, requirement, finder=None):
        # type: (TRequirement, Optional[PackageFinder]) -> Optional[SetupInfo]
        ireq = requirement.as_ireq()
        subdir = getattr(requirement.req, "subdirectory", None)
        return cls.from_ireq(ireq, subdir=subdir, finder=finder)

Let me build the package normally and see what the generated metadata looks like.

These are all different ways of "building" the package and how the metadata looks like:
# Assuming the project is cloned somewhere
% cd test-project/parent_folder/pep508-package
% rm -rf src/pep508_package.egg-info build dist && pipx run build
% unzip -c dist/pep508_package-1.0.0-py3-none-any.whl pep508_package-1.0.0.dist-info/METADATA
...
Requires-Dist: sibling-package @ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package
...
% tar xOf dist/pep508_package-1.0.0.tar.gz pep508_package-1.0.0/src/pep508_package.egg-info/requires.txt
six
sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package
...

% .venv/bin/python -m pip install -U setuptools
% rm -rf src/pep508_package.egg-info build dist && .venv/bin/python setup.py bdist_wheel
% unzip -c dist/pep508_package-1.0.0-py3-none-any.whl pep508_package-1.0.0.dist-info/METADATA
...
Requires-Dist: sibling-package @ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package
...

% .venv/bin/python -m pip uninstall pep508-package
% cat .venv/lib/python3.8/site-packages/pep508_package-1.0.0.dist-info/METADATA
...
Requires-Dist: sibling-package @ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package
...

It seems that Requires-Dist in the metadata is correct.

Sorry, I forgot one line in the previous log:
.venv/bin/python -m pip install 'pep508-package @ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/pep508-package'

I just updated the previous comment to include it.

@abravalheri Well thanks for your help, I now have a patch now for requirementslib that is generic enough for any subdirectory fragment, but I still don't understand how this just broke.   Maybe before it wasn't passing the subdirectory properly either and setuptools was searching and finding the first setup.py in a subdirectory?
Check it out, this test passed: https://github.com/pypa/pipenv/runs/5712591302?check_suite_focus=true
with this change:  https://github.com/pypa/pipenv/pull/5017

Uummm... It is a possibility but I don't think setuptools is that smart 😅

The pip-tools daily scheduled CI began failing four tests 5 days ago, the day of setuptools v61.0.0 release.
These are:

test_cli_compile.py::test_input_formats[setup.py]
test_cli_compile.py::test_multiple_extras[setup.py-singular]
test_cli_compile.py::test_one_extra[setup.py]
test_cli_compile.py::test_multiple_extras[setup.py-comma-separated]

The pip-tools daily scheduled CI began failing four tests 5 days ago, the day of setuptools v61.0.0 release.

Hi @AndydeCleyre, I noticed that the fixtures seem to be creating distributions with no python modules.
When I compare METADATA_TEST_CASES for setup.cfg and setup.py I can see that setup.cfg has packages = find:, but setup.py lacks the equivalent packages=find(). Is that on purpose?

Note that, prior to setuptools v61, these different configurations are not equivalent:
a) The example with setup.cfg would include sample_lib/__init__.py in the distribution
b) The example with setup.py would be empty.
For the majority of use cases, the situation in (b) is accidental and unintended (maybe the same accident happened in your test case?).
With the improvements in v61, setuptools will try to auto-detect which files to include, but it will fail if the root directory has a confusing/unequivocal layout.
The example generated by the fixtures seem to have extra folders dists and packages.
setuptools>=61 does not know exactly how to deal with these folders and therefore, halts the build process asking for the user to improve the configuration or use a src-layout.
This is the breaking change motivating the major bump from v60 to v61 and is described in the CHANGELOG.
The layouts recognized by setuptools for auto-discovery are described in https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#automatic-discovery.
Layouts that differ from the ones documented are asked to explicitly list packages and/or py_modules in their configuration.

A late but big "Thank You!!" for fixing this :-)
We had already removed installation via setup.py from the documentation of our project some time ago, and I'll also remove it now from our install test.