[jdbc] made scan operation unimplemented

It was determined that since SQL does not have relevant scan
functionality, that it should be made unimplemented in the jdbc-binding.
It is possible to simulate scan results in SQL but the performance
metrics would not be comparable to a database system with true scan
functionality. The results also could not gaurantee consistency across
certain YCSB property configurations.
If users truly want to do the scan operation, the old methods will always be in
commit history and they can implement it themselves, but as is, it is broken.
See #171.

See my latest in #171.
I also think that since people are/were using this, it might be a little harsh to pull the rug out from underneath them and make it unimplemented. Let the users decide about contextual relevancy of the scan results as long as YCSB can provide performance metrics around performing a scan.


label_key not found during metrics steps for scanorama_full

During the metrics step in scanorama the label_ley (patient) might be missing in the integrated object. In this case, I think it causes an error that stops the metrics calculation.
Is this expected behavior? I suspect that for some reason the label was not propagated to the integrated object. The column found in the input ifle integration_benchmark_input_CP_log1p.h5ad.
Using shell: /usr/bin/bash
Provided cores: 4
Rules claiming more threads will be scaled down.
Job counts:
	count	jobs
	1	metrics_single
	1

[Mon Feb  8 08:54:54 2021]
Job 0:
        Metrics hvg=hvg,method=scanorama,o_type=full,scaling=unscaled,scenario=CP
        output: CP/metrics/unscaled/hvg/scanorama_full.csv

ERROR conda.cli.main_run:execute(32): Subprocess for 'conda run ['python', 'scripts/metrics.py', '-u', 'pancreatic_cancer/data/integration_benchmark_input_CP_log1p.h5ad', '-i', 'CP/integration/unscaled/hvg/scanorama.h5ad', '-o', 'CP/metrics/unscaled/hvg/scanorama_full.csv', '-m', 'scanorama', '-b', 'patient', '-l', 'label', '--type', 'full', '--hvgs', '2000', '--organism', 'human', '--assay', 'expression', '-v']' command failed.  (See above for error)
During startup - Warning messages:
1: package ‘methods’ was built under R version 3.6.1
2: package ‘datasets’ was built under R version 3.6.1
3: package ‘utils’ was built under R version 3.6.1
4: package ‘grDevices’ was built under R version 3.6.1
5: package ‘graphics’ was built under R version 3.6.1
6: package ‘stats’ was built under R version 3.6.1
/home/icb/ignacio.ibarra/miniconda3/envs/scIB-python/lib/python3.7/site-packages/anndata/_core/anndata.py:21: FutureWarning: pandas.core.index is deprecated and will be removed in a future version.  The public classes are available in the top-level namespace.
  from pandas.core.index import RangeIndex
Traceback (most recent call last):
  File "/home/icb/ignacio.ibarra/miniconda3/envs/scIB-python/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 2889, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 70, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 97, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1675, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1683, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'patient'

Thanks,

This looks like your config is incorrectly set up to work with the integration output you have for scanorama. I'm not sure what you have stored in your integrated object. Can you show adata.obs output?
Also, label_key would typically be cell type for us... and batch_key would normally be batch/patient.

adata.obs for input and scanorama.
scib_input = sc.read_h5ad('data/integration_benchmark_input_CP_log1p.h5ad')
scib_input.obs[['patient', 'underlying.cause', 'pathology.report', 'batch', 'label']]

scanorama_full = sc.read_h5ad('scIB_run/CP/integration/unscaled/hvg/scanorama.h5ad')
scanorama_full.obs

The DATA_SCENARIO for this case in config.yaml is
DATA_SCENARIOS:
  CP:
    batch_key: batch
    label_key: label
    assay: expression
    organism: human
    file: data/integration_benchmark_input_CP_log1p.h5ad

I'll keep debugging I suspect now it is a scanorama version issues.

In case it helps scanoramas version in scIB-python is 1.6
In [1]: import scanorama
In [2]: scanorama.__version__
Out[2]: '1.6'

Ah, yes... we have a fix for this and some other issues in scanorama in our metrics.py script. I think this has now been fixed in newer versions of scanorama as well though. We were initially using a patched version of 1.4... but now we have 1.7 as recommended install (our patch was included from 1.7) in our scIB-python environment. There's a small change to the Scanorama function though to make this work.

The scIB-python PR is currently being worked on by @mumichae and should be merged very soon.

@ilibarra the scIB-python environment has been updated  in #147 . Could you update your scIB-python environment, preferably by removing it completely and reinstalling it?
conda deactivate
conda env remove -n scIB-python
conda env create -f <path-to-scib-repo>/envs/scIB-python.yaml

Double-check that the scanorama version is 1.7
Also, you'll need to install kBET manually in R of scIB-python
devtools::install_github('theislab/kBET', quiet=T)

Hope this resolves the issue!

Hey @mumichae
Thanks for checking on this. So far I have been able to repeat the running of scanorama but I get stuck during the save_embedding.py step
[Tue Mar  2 14:09:30 2021]
Job 56:
        SAVE EMBEDDING
        Scenario: all unscaled full_feature
        Method: scanorama embed
        Input: /storage/groups/ml01/workspace/ignacio.ibarra/theislab/pancreatic_cancer/data/scIB_run/all/integration/unscaled/full_feature/scanorama.h5ad
        Output: /storage/groups/ml01/workspace/ignacio.ibarra/theislab/pancreatic_cancer/data/scIB_run/all/embeddings/unscaled/full_feature/scanorama_embed.csv /storage/groups/ml01/workspace/ignacio.ibarra/theislab/pancreatic_cancer/data/scIB_run/all/embeddings/unscaled/full_feature/scanorama_embed_batch.png /storage/groups/ml01/workspace/ignacio.ibarra/theislab/pancreatic_cancer/data/scIB_run/all/embeddings/unscaled/full_feature/scanorama_embed_labels.png

connect localhost port 6000: Connection refused
connect localhost port 6000: Connection refused
ERROR conda.cli.main_run:execute(32): Subprocess for 'conda run ['python', 'scripts/save_embeddings.py', '--input', '/storage/groups/ml01/workspace/ignacio.ibarra/theislab/pancreatic_cancer/data/scIB_run/all/integration/unscaled/full_feature/scanorama.h5ad', '--outfile', '/storage/groups/ml01/workspace/ignacio.ibarra/theislab/pancreatic_cancer/data/scIB_run/all/embeddings/unscaled/full_feature/scanorama_embed.csv', '--method', 'scanorama', '--batch_key', 'batch', '--label_key', 'label', '--result', 'embed']' command failed.  (See above for error)
qt.qpa.xcb: could not connect to display localhost:12.0
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: eglfs, minimal, minimalegl, offscreen, vnc, webgl, xcb.

/home/icb/ignacio.ibarra/miniconda3/envs/scIB-python/.tmph1easou4: line 3: 13848 Aborted                 (core dumped) python scripts/save_embeddings.py --input /storage/groups/ml01/workspace/ignacio.ibarra/theislab/pancreatic_cancer/data/scIB_run/all/integration/unscaled/full_feature/scanorama.h5ad --outfile /storage/groups/ml01/workspace/ignacio.ibarra/theislab/pancreatic_cancer/data/scIB_run/all/embeddings/unscaled/full_feature/scanorama_embed.csv --method scanorama --batch_key batch --label_key label --result embed

I think this is related to the QT_DEBUG_PLUGINS variable but trying to wonder how to modify it. I will try this in my interactive version first. If you think this has to be done in the pipelin scripts please let me know.
Thanks a lot!

Thanks for this comment @ilibarra! @danielStrobl or @mumichae could you add this to the env setup description? I guess it now needs to go into scib and scib-pipeline.

@ilibarra Are you perhaps working on a cluster?

@LuckyMD I think that should only go into scib-pipeline, because we transferred all the environments there

@LuckyMD I guess this issue can be closed?


Multiple repeated records when using Include and ThenInclude

This was suppose to be fixed by 2.0.0 but I still get this error, if I have an entity with Include and then include, I end up with multiple records for the same Entity Y
var q = dbTenantContext.Yes
                                        .Include(e => e.Y)
                                            .ThenInclude(e => e.YY)
                                                .ThenInclude(e => e.YYY)
                                                    .ThenInclude(e => e.YYYY)
                                        .Include(e => e.Y)
                                            .ThenInclude(e => e.YY)
                                                .ThenInclude(e => e.YYY)
                                                    .ThenInclude(e => e.YYYX)
                                                        .ThenInclude(e => e.YYYXX)
                                         .Include(e => e.Z)
                                            .ThenInclude(e => e.ZZ)
                                                .ThenInclude(e => e.ZZZ)
                                                    .ThenInclude(e => e.ZZZZ)
                                        .Include(e => e.Z)
                                            .ThenInclude(e => e.ZZ)
                                                .ThenInclude(e => e.ZZX)
                                                   .ThenInclude(e => e.ZZXX)
                                        .Where(e => e.IsDeleted != true)

I used to fix this in 1.1 by adding at the end of the query the following
.GroupBy(e => e.Id)
 .Select(e => e.First())

but now with 2.0.0 I am getting the following error if I add that group and select
Expression of type 'System.Object' cannot be used for parameter of type 'Microsoft.EntityFrameworkCore.Metadata.IEntityType'

Further technical details
EF Core version: 2.0
Database Provider: Microsoft.EntityFrameworkCore.SqlServer
Operating system: MS Windows 7
IDE: Visual Studio 2017

duplicate of #9551


How to use utf8 resource file path ???

It seems like that mediapipe only support ANSI file paths, and in some non-English operating systems, the folder name may not be ansi, how can I let it work?
JoinPath (file_path.cc) need support utf-8 file path.

Hi @endink,
Could you please elaborate your query with complete details to understand better on your query. Thank you!

Alright, Okay, let me be more specific:
As I know, we can use this code to change the folder that used for mediapipe search graph files:

ABSL_DECLARE_FLAG(std::string, resource_root_dir);
absl::SetFlag(&FLAGS_resource_root_dir, std::string(folder, charLen));

Even "folder" and "charLen" in the above code is UTF8 encoded char* and the correct utf8 length character length, mediapipe will get a file not found error because the folder contains Chinese characters.

Google doesn't seem to bother to revert to this “small“ issue, so I'd better close it.


Update from 2.13.2 to 2.13.3 fails on macOS

My Setup

Operating System: macOS High Sierra (10.13.6)
App Version: 2.13.2

[ ] I have tested with the latest version
[X] I can simulate the issue easily

Description
The Rocket.Chat electron app doesn't seem to successfully update from 2.13.2 to 2.13.3 using the built in updater. I initially installed this app through a Homebrew Cask.
Current Behavior
On opening Rocket Chat, the app prompts me that "A new version of the Rocket.Chat Desktop App is available!", which is 2.13.3. Click "Install update", then prompts that the "Update has been downloaded". I hit install now and Rocket.Chat exits out. On reopening the app, the same update notification appears, asking me to update to 2.13.3.
Expected Behavior
The app, upon hitting "Install Now" should install the update to 2.13.3. No update dialog box appears on reopening the app.

I'm getting the same issue updating from 2.13.4 to 2.14.0

@dontait @media32 Was the original setup via Homebrew Cask too?

In our office, we all installed from the Rocket Chat installer found on the website.

Same issue here. App was installed from the website.

Operating System: macOS High Sierra (10.13.6)
App Version: 2.13.0

installed from the website as well

@dontait @media32 Was the original setup via Homebrew Cask too?

no it was a download from the website

I'm debugging it; some odd behavior in autoUpdater.quitAndInstall() method happening only in MacOS.

Same issue attempting to update from 2.14.0 to 2.14.1 on Mojave 10.14

@sysfu

Unfortunately, autoupdate fixes can't be applied backwardly. Therefore, versions listed in #894 will fail on update regardlessly; we advise users to click the "Install later" button to update.


Feature request: support metrics with unit "None"

Use case
Original issue: https://github.com/aws-powertools/powertools-lambda-python/issues/4834
Consider changes to the Dotnet naming pattern in the text below.

docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html reports that "None" is a valid unit for CloudWatch metrics, but the MetricUnit enum does not have a member for this unit.

The lack of such a member means that there is no way to create metrics with unit "None" via Powertools. One cannot pass the string "None" as the unit parameter of Metric.add_metric, due to this code.

Solution/User Experience
Original issue: https://github.com/aws-powertools/powertools-lambda-python/issues/4834

Of course, I realize that a MetricUnit member for "None" cannot be named None, but can a member with a different name (perhaps named None_ or NoUnit) please be added to the enum?

Alternative solutions
No response
Acknowledgment

[X] This feature request meets Powertools for AWS Lambda (.NET) Tenets
[X] Should this be considered in other Powertools for AWS Lambda languages? i.e. Python, Java, and TypeScript

@leandrodamascena thanks for opening the issue.
None MetricUnit is already supported in Powertools .NET.
https://github.com/aws-powertools/powertools-lambda-dotnet/blob/04cbc46a91731ebc160bb4ff5b99014d46ca5daf/libraries/src/AWS.Lambda.Powertools.Metrics/Model/MetricUnit.cs#L34
Closing the issue


GraphQLServletListener's onSuccess is not in request scope after upgrading to 11.1.0

Describe the bug
I've encountered a strange bug when upgrading to graphql-spring-boot-starter:11.1.0 from 6.0.
I have a GraphQLServletListener that is used to trigger an applicationEvent using Spring's ApplicationEventPublisher to handle transaction status monitoring. The endpoint for listening to the event stream is scoped per request (WebApplicationContext.SCOPE_REQUEST), and that was working perfectly previously. The GraphQLListener's onRequest set the event to requestStart, the onSuccess of the listener set it to requestEnd. Now after upgrading our dependencies (kotlin 1.4.32, SpringBoot 2.4.5, GQL starter 11.1.0) the bean creation fails for the listener by telling me that there is no request scope (while obviously there should be as I'm currently handling the onSuccess part of the request before the onFinally)
Caused by: java.lang.IllegalStateException: No thread-bound request found: Are you referring to request attributes outside of an actual web request, or processing a request outside of the originally receiving thread? If you are actually operating within a web request and still receive this message, your code is probably running outside of DispatcherServlet: In this case, use RequestContextListener or RequestContextFilter to expose the current request.
Any ideas what's happening, what's changed or what am I doing wrong (and why did it work previously)?
To Reproduce
Project is a closed-source, haven't yet created an example project for the bug.
Expected behavior
The listener's onSuccess should be in request scope
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: Ubuntu 20.04

The default setting has been reverted to false in release 13.0.0 which has just been released.


Add weight column

Implementing #24

In general you should avoid this pattern because it often leads to memory leaks.

This is awful. Whoever taught you this should be fired.


error: failed to run custom build command (v0.1.0) (cli)

Cargo installation command run:
cargo install --git=https://github.com/antonWetzel/typst-languagetool cli --features=bundle-jar
Error output:
error: failed to run custom build command for `typst-languagetool v0.1.0 (/Users/dc/.cargo/git/checkouts/typst-languagetool-680d74d5831615e8/517bc76)`

Caused by:
  process didn't exit successfully: `/var/folders/x3/cd0296z56pj1pgyw3hkthb8h0000gn/T/cargo-installOGZC8E/release/build/typst-languagetool-55d1ca1ce96a5c2c/build-script-build` (exit status: 101)
  --- stdout
  cargo::rerun-if-changed=build.rs
  cargo::rerun-if-changed=maven/pom.xml
  cargo::rerun-if-changed=maven/src/assembly/dep.xml

  --- stderr
  thread 'main' panicked at build.rs:23:14:
  called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: failed to compile `cli v0.1.0 (https://github.com/antonWetzel/typst-languagetool#517bc769)`, intermediate artifacts can be found at `/var/folders/x3/cd0296z56pj1pgyw3hkthb8h0000gn/T/cargo-installOGZC8E`.
To reuse those artifacts with a future compilation, set the environment variable `CARGO_TARGET_DIR` to that path.
make: *** [install-languagetool-cli] Error 101

Looks like the maven command failed because mvn is not installed/available on path.
With f1b28501cf519240af14a308a4032ae50aa6904b I added better error messages.
If you want to use a bundled jar maven and Java must be installed.

I'm afraid we're still not quite there - although this seems like a different error.
I have maven and java in my path:
~ mvn --version
Apache Maven 3.9.8 (36645f6c9b5079805ea5009217e36f2cffd34256)
Maven home: /opt/homebrew/Cellar/maven/3.9.8/libexec
Java version: 22.0.2, vendor: Homebrew, runtime: /opt/homebrew/Cellar/openjdk/22.0.2/libexec/openjdk.jdk/Contents/Home
Default locale: en_JP, platform encoding: UTF-8
OS name: "mac os x", version: "13.1", arch: "aarch64", family: "mac"

~ java --version
openjdk 22.0.2 2024-07-16
OpenJDK Runtime Environment Homebrew (build 22.0.2)
OpenJDK 64-Bit Server VM Homebrew (build 22.0.2, mixed mode, sharing)

Current error message:
warning<EMAIL_ADDRESS>JAR at "/Users/dc/.m2/repository/invalid/typst-languagetool/1/typst-languagetool-1-jar-with-dependencies.jar".
error: couldn't read /var/folders/x3/cd0296z56pj1pgyw3hkthb8h0000gn/T/cargo-installPcOCDy/release/build/typst-languagetool-ee3bfa6fc348f81f/out./jar_path.rs: No such file or directory (os error 2)
  --> src/backends/jni.rs:36:14
   |
36 |         let path = include!(concat!(env!("OUT_DIR"), "./jar_path.rs"));
   |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: this error originates in the macro `include` (in Nightly builds, run with -Z macro-backtrace for more info)

The following warnings were emitted during compilation:

warning<EMAIL_ADDRESS>JAR at "/Users/dc/.m2/repository/invalid/typst-languagetool/1/typst-languagetool-1-jar-with-dependencies.jar".

error: could not compile `typst-languagetool` (lib) due to 1 previous error

Caused by:
  process didn't exit successfully: `rustc --crate-name typst_languagetool --edition=2021 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=120 --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg 'feature="bundle-jar"' --cfg 'feature="default"' -C metadata=11385b0e7a24b77b -C extra-filename=-11385b0e7a24b77b --out-dir /var/folders/x3/cd0296z56pj1pgyw3hkthb8h0000gn/T/cargo-installPcOCDy/release/deps -C strip=debuginfo -L dependency=/var/folders/x3/cd0296z56pj1pgyw3hkthb8h0000gn/T/cargo-installPcOCDy/release/deps --extern anyhow=/var/folders/x3/cd0296z56pj1pgyw3hkthb8h0000gn/T/cargo-installPcOCDy/release/deps/libanyhow-9a6a97412b156342.rmeta --extern jni=/var/folders/x3/cd0296z56pj1pgyw3hkthb8h0000gn/T/cargo-installPcOCDy/release/deps/libjni-0d65691fd31143f5.rmeta --extern serde=/var/folders/x3/cd0296z56pj1pgyw3hkthb8h0000gn/T/cargo-installPcOCDy/release/deps/libserde-6f1a64f9f80ca0e7.rmeta --extern typst=/var/folders/x3/cd0296z56pj1pgyw3hkthb8h0000gn/T/cargo-installPcOCDy/release/deps/libtypst-d9f595586c947bf5.rmeta -L native=/var/folders/x3/cd0296z56pj1pgyw3hkthb8h0000gn/T/cargo-installPcOCDy/release/build/psm-c4b891f690894094/out` (signal: 5, SIGTRAP: trace/breakpoint trap)
warning: build failed, waiting for other jobs to finish...
error: failed to compile `cli v0.1.0 (https://github.com/antonWetzel/typst-languagetool#f1b28501)`, intermediate artifacts can be found at `/var/folders/x3/cd0296z56pj1pgyw3hkthb8h0000gn/T/cargo-installPcOCDy`.
To reuse those artifacts with a future compilation, set the environment variable `CARGO_TARGET_DIR` to that path.
make: *** [install-languagetool-cli] Error 101

I only tested on windows, looks like the windows OUT_DIR contains a trailing / and some unix platforms not.
With 0db2e1c4a1bb80194ae32908e0ec1b2d192d1993 this should work for all platforms (not testetd).

Worked!


Would node cron jobs start automatically after a server reboots?

Does it natively use crontab?
Do I need to use some kind of daemon or scheduler to restart the node cron jobs after system reboot, or the jobs crashes?
Or does it restart itself?

Hi.
It seems like it doesn't, you can check by setting crontasks using node-cron > using a terminal > crontab -e.
You can see that nothing is there.
For some people that might be a problem, but I solved that by calling crontask setting each time the application is restarted.


TCP socket options from upstream cluster config are incorrectly sent down to QUIC

I ran into this issue when I tried speaking QUIC in Envoy-mobile but I think the root cause is from Envoy.
When the ClusterInfoImpl for HTTP/3 is created, it would fetch the cluster config and potentially parse some TCP socket options.
Later when we try to create a QUIC client connection, the HTTP/3 connection pool would get socket options from the cluster and apply them to the underlying UDP socket.
Apparently, socket options like TCPKEEPALIVE are specific to TCP and will fail in QUIC.
I wonder how we should solve this. Should we move socket option configs out of the cluster config? Or maybe we can add a parameter to specify whether an option can only be applied to a specific socket type?

cc @danzh2010

This needs a non-stale label.

@alyssawilk any thoughts on this?

I'd be inclined to allow tagging for TCP-only or UDP-only but I defer to @envoyproxy/api-shepherds

Yeah, I don't think we need any API semantic change here. Socket options at the POSIX-level have this weak typing and need to be applied to the correct socket type. So, just proper documentation and validation would make sense IMHO.

sorry @htuch are you suggesting that Envoy automatically check which socket option is being set and only apply it to the relevant socket types?  I agree that'd be ideal but it does mean that we either have to exhaustively add a check function for all known socket options and validate them and keep them up to date for all supported platforms, or we have to accept folks will write config for a not-yet-validated one, find it doesn't work, and require a binary update to set their socket option.
maybe there's something in between, like a runtime option folks could set to bridge that gap?

I guess I'm not just convinced that there is an API change needed, but happy to see a proposal for one that would make sense.

@ggreenway thoughts here? Looking just now at your comment https://github.com/envoyproxy/envoy/issues/22634#issuecomment-1247349596 seems to indicate that it might be hard to really use config guarding as the appropriate mechanism for socket option safety across platforms.


[Feature]: Freshness executions

Overview
I would like to have a SOURCE_EXECUTIONS table showing dbt source freshness results.
What problem would this solve?
The freshness "tests" are using to detect stale data in the sources.
It will be helpful to have it so we can monitor stale errors.
Would you be willing to contribute?

[ ] Yes - I'm on it!
[x] Yes, but I'd need help getting started.
[ ] No

Hi @dennysrega1ado .
Thanks for raising this. This is something we'd love to support, but are waiting for an upstream change in dbt core that would allow us to do it currently. Once https://github.com/dbt-labs/dbt-core/issues/5609 has been resolved, we'll definitely look to include it.
Related PR: https://github.com/brooklyn-data/dbt_artifacts/issues/166

I'm closing this issue so that this can be tracked in a single place - #166


Mentioning a user that has a space or other form of line break in their Discord tag doesn't work properly.

Tested and reproduced on Quarrel Insider 20.7.5.

This is still the case in 20.7.5? I thought I had fixed it.

It is still broken, unfortunately. :c


x/crypto/ssh: support RSA SHA-2 host key signatures

What version of Go are you using (go version)?

$ go version
1.13.8

Version of x/crypto: 1d94cc7ab1c630336ab82ccb9c9cda72a875c382
Does this issue reproduce with the latest release?
Yes.
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
n/a

What did you do?
I tried to connect to an OpenSSH 8.2 server with the ssh-rsa host key algorithm disabled on the server. I also tried to run an x/crypto/ssh server and connect from an OpenSSH client with ssh-rsa disabled. Lastly, I tried to sign a host certificate with ssh.Certificate.SignCert with a SHA-2 based signature.
What did you expect to see?
I expected the RSA host key and certificate to validate successfully with the new SHA-2 based signatures introduced in RFC 8332. I also expected to be able to sign host certificates and have them automatically received a SHA-2 based signature.
OpenSSH has already deprecated ssh-rsa (i.e. SHA-1 based) signatures in host certificates in version 8.2 because of safety reasons. They can still be used by the host key algorithm must be manually specified.
What did you see instead?
I was unable to connect either as a server or a client if ssh-rsa wasn't enabled while using RSA host keys or host certificates. I was able to sign a certificate with the AlgorithmSigner wrapper approach (i.e. by forcefully overriding Sign) proposed by @stoggi in #36261, but it's not a great experience for users.

I spent some time hacking away at a solution for this and believe I have something largely ready to make a CL from: https://github.com/hansnielsen/golang-x-crypto/compare/master...hans-rsa-sha2-support
The approach OpenSSH chose for these new signature types is somewhat interesting: they have the same key type (ssh-rsa) but different signature algorithms. This makes it slightly tricky to integrate while keeping the existing tests working. There's a little more explicit special-casing of the RSA SHA-2 signature family than I'd like.
The one main choice I made in here (beyond just adding support for the new signature types) is that certificates now default to rsa-sha2-512 instead of ssh-rsa. I think given the already-deprecated nature of ssh-rsa plus the reasonable threat model for host certificates, this is the correct choice. The "wrapped AlgorithmSigner" approach can still be used to force an RSA SHA-1 signature and there's no need to encourage this.
Feedback welcome! If the approach in the code above looks good, I'll work on submitting a CL.

cc @hanwen @FiloSottile

Sounds good on both adding the algorithms and changing the certificate default.
OpenSSH 8.2 already stopped supporting ssh-rsa signed certificates, right?

Only for host certificates I think, client certs still can use it.

I considered removing the SHA-1 RSA signature for host certs but wanted to keep this issue to algorithm addition. There are some choices in the client host key algorithm list that are probably worth revisiting as well and it’d probably be best to handle that as one issue.

How long has OpenSSH supported rsa-sha2-512 certificates?

OpenSSH 7.3 was the first version with rsa-sha2-512 certificate support, released on 2016-08-01. Ubuntu xenial stands out as being the largest distro that's still on 7.2 with backported security fixes.

I always regret asking things like this, but surely support non-SHA1 signature algorithms is as deserving of a security backport as an actual CVE?

Based on the discussion above, this seems like a likely accept.

No change in consensus, so accepted.

Looks like https://go-review.googlesource.com/c/crypto/+/220037 has had no progress since Feb 20th.
Since everyone seems to agree on the change, @hansnielsen @FiloSottile do you have the time to finish that CL review?
This is causing github.com/gravitational/teleport/ some interop problems with newer openssh versions.

This is causing some trouble for us over in https://github.com/hashicorp/packer/issues/8609. I can confirm that https://go-review.googlesource.com/c/crypto/+/220037 fixes the connection issues we are seeing.

This issue is hitting us over in Packer at https://github.com/hashicorp/packer/issues/8609. We use the ssh client, generating the Auth method first by calling ParsePrivateKey and then feeding that signer into the PublicKeys method to retrieve the publickeycallback. code: https://github.com/hashicorp/packer/blob/master/helper/communicator/config.go#L320-L324
I can create my own algorithm signer to set the algorithm and call the wrappedSigner's SignWithAlgorithm, but it doesn't work with the proposed change at https://go-review.googlesource.com/c/crypto/+/220037 unless the publickeycallback is also modified, because it's still sending the key type as the algorithm, in this case "ssh-rsa".
By hardcoding lines https://github.com/golang/crypto/blob/75b288015ac94e66e3d6715fb68a9b41bf046ec2/ssh/client_auth.go#L218 and https://github.com/golang/crypto/blob/75b288015ac94e66e3d6715fb68a9b41bf046ec2/ssh/client_auth.go#L232 to SigAlgoRSASHA2256, I can force this to work to connect to an ssh server that doesn't allow the "ssh-rsa" algorithm, but obviously this isn't a viable patch for the module.
This is my first time doing a real dive into this library so I'm wondering if anyone else on this thread has an instinct about how to extend the ssh.PublicKeys() method to make it work for AlgorithmSigners that are actually using custom algorithms.

I tried working around this by implementing a custom ssh.AlgorithmSigner around an ssh.Signer (which is derived from an SSH certificate).
You can sign new certificates using a CA this way and it seems to work fine with OpenSSH.
However, you can't use such signers for actual SSH handshakes against a Go SSH server, because of https://github.com/golang/crypto/blob/75b288015ac94e66e3d6715fb68a9b41bf046ec2/ssh/server.go#L556-L564. That's a hardcoded check for algorithms supported in handshake signatures.
So, without https://go-review.googlesource.com/c/crypto/+/220037, you can only get cert signing to work.
For reference, here's our workaround for certs: https://github.com/gravitational/teleport/pull/3777/files#diff-193b31b40b2b216b00568f93daa82603

Turns out my particular client key handling needs aren't actually directly relevant to this patch, though the underlying "this module is having some problems with handling rsa sha-2 algorithms" is the same -- I think it needs its own GH issue. I'll work on setting up my own case and my own patch suggestion 😬

In order to workaround this issue I tried to use an ecdsa key instead, which worked fine until we hit a brick wall because AWS only supports RSA keys.
Any chance we can increase the priority of the code review in https://go-review.googlesource.com/c/crypto/+/220037/ ?

Yes, I plan to get to it this week.
On Tue, Oct 6, 2020 at 3:59 PM Dusty Mabe<EMAIL_ADDRESS>wrote:

In order to workaround this issue I tried to use an ecdsa key instead
https://github.com/coreos/coreos-assembler/pull/1749, which worked fine
until we hit a brick wall because AWS only supports RSA keys
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#how-to-generate-your-own-key-and-import-it-to-aws
.
Any chance we can increase the priority of the code review in
https://go-review.googlesource.com/c/crypto/+/220037/ ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/37278#issuecomment-704287991, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJLETV5T6CYFTTKYBT6XZ3SJMPCJANCNFSM4KW5GR3A
.

When you look at this one, if you have time to take a look at a related patch over here: https://github.com/golang/go/issues/39885 that'd be super rad

I ran into this issue on a new install of Fedora 33. ssh-rsa is disabled by default now, so I was not able to connect to our golang ssh server.
The ssh error message for googleability:
debug1: send_pubkey_test: no mutual signature algorithm
Workarounds at https://bugzilla.redhat.com/show_bug.cgi?id=1884920 , but it would be nice if fixed in golang.

Hey @FiloSottile - friendly ping. Are there still plans to push https://go-review.googlesource.com/c/crypto/+/220037/ forward?

This issue makes Go SSH servers with RSA hostkey unusable with SSH clients having hardened configuration w/o insecure algorithms enabled. E.g. with following .ssh/config:
Host *
    Ciphers aes256-gcm@openssh.com,aes128-gcm<EMAIL_ADDRESS>    KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
    HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-256,rsa-sha2-512
    MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512

Are there any further updates on this?

Is there currently no reasonable way to do rsa-sha2-256 host key signing in golang? This was signed off on and then kind of died

I pinged @FiloSottile the other day on Slack. He mentioned it is on his radar to look at soon. He said he needs get back into the crypto/ssh pkg and hopefully take this to the finish line. So hopefully we'll see progress soon! 🙏

It's not clear to me that @hansnielsen is still actively working on this issue as @rolandshoemaker made some requests on August 23rd.
If @hansnielsen is not actively working on this, could someone from the golang security team take this over and make the last few changes needed to integrate it?
Certificate signing for RSA keys has been incompatible with the default settings in OpenSSH since 8.2 (released in February 2020!), and OpenSSH claims they are disabling SHA1 for RSA keys in the next release which will break all non-cert uses of RSA keys as well. This kind of breakage seems fairly unacceptable for a core package.

I'm still here (and active on the Gerrit changelist)! Hopefully we'll figure out the last few tweaks to get this merged soon.

The new version of Git for Windows uses OpenSSH version 8.8, which uses THE RSA-SHA2 algorithm by default. However, the Golang SSH library used by Gitee uses the RSA-SHA1 algorithm, so the public key authentication fails.
Add the following information to the config file under ~/.ssh/ :
Host gitee.com
HostkeyAlgorithms +ssh-rsa
PubkeyAcceptedAlgorithms +ssh-rsa

GitHub drops RSA with SHA-1 signatures as well.
https://github.blog/2021-09-01-improving-git-protocol-security-github/

Are there any workarounds for this on the client side? Here is a minimal example code to reproduce the issue. The linked code errors with:
$ go run main.go
2021/11/10 20:41:23 unable to connect: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
exit status 1

Using the same key with ssh -i /path/to/key localhost works perfectly

The only workaround right now is to use a forked crypto pkg and apply the changeset in the linked PR. It's how we are using it at the moment.

changeset in the linked PR

Sorry to ask, but which PR do you mean?
Sidebar shows:
Linked pull requests
None yet

@pinpox https://golang.org/cl/220037

In fact, this Issue should be reopened. Since https://golang.org/cl/220037 does not implement server-sig-algs, OpenSSH will not choose the local RSA Key because it does not know whether the server supports rsa-sha2- 256/rsa-sha2-512

In fact, this Issue should be reopened. Since https://golang.org/cl/220037 does not implement server-sig-algs, OpenSSH will not choose the local RSA Key because it does not know whether the server supports rsa-sha2- 256/rsa-sha2-512

I can confirm the error can still be reproduced with the code linked above.

In fact, this Issue should be reopened. Since https://golang.org/cl/220037 does not implement server-sig-algs, OpenSSH will not choose the local RSA Key because it does not know whether the server supports rsa-sha2- 256/rsa-sha2-512

I can confirm the error can still be reproduced with the code linked above.

PR: https://github.com/golang/crypto/pull/197 fixed.

I have the latest x/crypto.  I am using a user cert that is based on rsa generated using ssh-keygen with no options.  My ssh client is set to use the agent auth.
If I supply the id_ecdsa public key to generate the user cert everything works.  Kex match etc.

If this fix is in (merged..), I expected it to work but no.
(The cert works fine if I use Openssh via command line)
Is there any configuration on the Client side that will help.  ?


Changing how null is shown in dataviewjs

Hi,
In standard dataview you can specify in the settings how null is shown in a table.  Is it possible to do this with dataviewjs to replace the default bullet.
Thx.

Fixed for next release 0.3.8.


Loading pymzn throws syntax error in init.py

Hi,
I am using Anaconda Python 2.7.12 in Windows 8.1. I installed pymzn with pip as instructed. When I tried to import the library, it throws me this error:
Traceback (most recent call last):
  File "<pyshell#3>", line 1, in <module>
    import pymzn
  File "C:\Users\Indrajit\Anaconda2\lib\site-packages\pymzn\__init__.py", line 114
    args.func(**{**vars(args), **args.solver_args})
                  ^
SyntaxError: invalid syntax

Please look into it.
Regards,
Indrajit

Installing via pip gets the Python 3 version of pymzn, which is unfortunately not working with python 2.
You can install the python 2-compatible branch by: pip install git+https://github.com/paolodragone/pymzn.git@python2
However, it is not up-to-date with the main development, e.g. the example from the readme is not working.

I tried. This is the error I am getting:
Collecting git+https://github.com/paolodragone/pymzn.git@python2
  Cloning https://github.com/paolodragone/pymzn.git (to python2) to c:\users\indraj~2.sen\appdata\local\temp\pip-4gayvz-build
  Error [Error 2] The system cannot find the file specified while executing command git clone -q https://github.com/paolodragone/pymzn.git c:\users\indraj~2.sen\appdata\local\temp\pip-4gayvz-build
Cannot find command 'git'

I do have Git Bash installed.
Regards,
Indrajit

You need to have the git command on your %PATH% variable, so it can be found by pip (see here).
I also found another alternative: pip install https://github.com/paolodragone/pymzn/archive/python2.zip, this installs the same version as the command above, but does not require git to be available.

Hi, sorry about the lack of support for both Windows and Python 2. I personally use Python 3 on Linux so I often overlook other platforms. I'll try to put some effort on those now that there are more people using pymzn on them.
@HelgeS As said, I mainly develop in python 3.5, so I often make use of syntactic constructs of the newer versions of python, that is why I keep python 2 and 3 versions on separate branches.