Adds support for PostJSON, PUT and DELETE methods

Adds support for PostJSON, PUT and DELETE methods

hey @spuentesp
thanks for the awesome work
I just tested this but it failed at compile time
here's the log output I got:

/home/ibby/work/myproject/platforms/android/src/com/synconset/CordovaHttpPlugin.java:80: error: cannot find symbol
            CordovaHttpDelete delete = new CordovaHttpDelete(urlString, paramsMap, headersMap, callbackContext);
            ^
  symbol:   class CordovaHttpDelete
  location: class CordovaHttpPlugin
/home/ibby/work/myproject/platforms/android/src/com/synconset/CordovaHttpPlugin.java:80: error: cannot find symbol
            CordovaHttpDelete delete = new CordovaHttpDelete(urlString, paramsMap, headersMap, callbackContext);
                                           ^
  symbol:   class CordovaHttpDelete
  location: class CordovaHttpPlugin
/home/ibby/work/myproject/platforms/android/src/com/synconset/CordovaHttpPlugin.java:96: error: cannot find symbol
            CordovaHttpPut put = new CordovaHttpPut(urlString, jsonObj, headersMap, callbackContext);
            ^
  symbol:   class CordovaHttpPut
  location: class CordovaHttpPlugin
/home/ibby/work/myproject/platforms/android/src/com/synconset/CordovaHttpPlugin.java:96: error: cannot find symbol
            CordovaHttpPut put = new CordovaHttpPut(urlString, jsonObj, headersMap, callbackContext);
                                     ^
  symbol:   class CordovaHttpPut
  location: class CordovaHttpPlugin

/home/ibby/work/myproject/platforms/android/src/com/synconset/CordovaHttpPlugin.java:103: error: cannot find symbol
            CordovaHttpPostJson postJson = new CordovaHttpPostJson(urlString, jsonObj, headersMap, callbackContext);

            ^

  symbol:   class CordovaHttpPostJson

  location: class CordovaHttpPlugin

/home/ibby/work/myproject/platforms/android/src/com/synconset/CordovaHttpPlugin.java:103: error: cannot find symbol

            CordovaHttpPostJson postJson = new CordovaHttpPostJson(urlString, jsonObj, headersMap, callbackContext);

                                               ^

  symbol:   class CordovaHttpPostJson

  location: class CordovaHttpPlugin

Note: Some input files use or override a deprecated API.

Note: Recompile with -Xlint:deprecation for details.

Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

6 errors

Hi,
I've tried replicating the error, with no success.
Delete: Entry, ":NSAppTransportSecurity", Does Not Exist
File Doesn't Exist, Will Create: platforms/ios//-Info.plist
Delete: Entry, ":NSAppTransportSecurity:NSAllowsArbitraryLoads", Does Not Exist
File Doesn't Exist, Will Create: platforms/ios//-Info.plist
File Doesn't Exist, Will Create: platforms/ios//-Info.plist
File Doesn't Exist, Will Create: platforms/ios//-Info.plist
grep: platforms/ios/MiMovistar/Classes/AppDelegate.m: No such file or directory
Not there
/Users/rodrigopalmafanjul/appmimovistar/i+d/hooks/before_build/ios_before_build.sh: line 25: platforms/ios/MiMovistar/Classes/AppDelegate.m: No such file or directory
Delete: Entry, ":CFBundleLocalizations", Does Not Exist
File Doesn't Exist, Will Create: platforms/ios//-Info.plist
File Doesn't Exist, Will Create: platforms/ios//-Info.plist
File Doesn't Exist, Will Create: platforms/ios//-Info.plist
Running command: /Users/rodrigopalmafanjul/appmimovistar/i+d/hooks/after_prepare/010_add_platform_class.js /Users/rodrigopalmafanjul/appmimovistar/i+d
add to body class: platform-android
ANDROID_HOME=/usr/local/Cellar/android-sdk/24.4.1_1
JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home
Starting a new Gradle Daemon for this build (subsequent builds will be faster).
Incremental java compilation is an incubating feature.
:preBuild
UP-TO-DATE
:preDebugBuild
UP-TO-DATE
:checkDebugManifest
:CordovaLib:preBuild
UP-TO-DATE
:CordovaLib:preDebugBuild
UP-TO-DATE
:CordovaLib:compileDebugNdk
UP-TO-DATE
:CordovaLib:compileLint
:CordovaLib:copyDebugLint
UP-TO-DATE
:CordovaLib:mergeDebugProguardFiles
UP-TO-DATE
:CordovaLib:packageDebugRenderscript
UP-TO-DATE
:CordovaLib:checkDebugManifest
:CordovaLib:prepareDebugDependencies
:CordovaLib:compileDebugRenderscript
UP-TO-DATE
:CordovaLib:generateDebugResValues
UP-TO-DATE
:CordovaLib:generateDebugResources
UP-TO-DATE
:CordovaLib:packageDebugResources
UP-TO-DATE
:CordovaLib:compileDebugAidl
UP-TO-DATE
:CordovaLib:generateDebugBuildConfig
UP-TO-DATE
:CordovaLib:mergeDebugShaders
UP-TO-DATE
:CordovaLib:compileDebugShaders
UP-TO-DATE
:CordovaLib:generateDebugAssets
UP-TO-DATE
:CordovaLib:mergeDebugAssets
UP-TO-DATE
:CordovaLib:processDebugManifest
UP-TO-DATE
:CordovaLib:processDebugResources
UP-TO-DATE
:CordovaLib:generateDebugSources
UP-TO-DATE
:CordovaLib:incrementalDebugJavaCompilationSafeguard
UP-TO-DATE
:CordovaLib:compileDebugJavaWithJavac
UP-TO-DATE
:CordovaLib:processDebugJavaRes
UP-TO-DATE
:CordovaLib:transformResourcesWithMergeJavaResForDebug
UP-TO-DATE
:CordovaLib:transformClassesAndResourcesWithSyncLibJarsForDebug
UP-TO-DATE
:CordovaLib:mergeDebugJniLibFolders
UP-TO-DATE
:CordovaLib:transformNative_libsWithMergeJniLibsForDebug
UP-TO-DATE
:CordovaLib:transformNative_libsWithSyncJniLibsForDebug
UP-TO-DATE
:CordovaLib:bundleDebug
UP-TO-DATE
:prepareAndroidCordovaLibUnspecifiedDebugLibrary
UP-TO-DATE
:preReleaseBuild
UP-TO-DATE
:CordovaLib:preReleaseBuild
UP-TO-DATE
:CordovaLib:compileReleaseNdk
UP-TO-DATE
:CordovaLib:copyReleaseLint
UP-TO-DATE
:CordovaLib:mergeReleaseProguardFiles
UP-TO-DATE
:CordovaLib:packageReleaseRenderscript
UP-TO-DATE
:CordovaLib:checkReleaseManifest
:CordovaLib:prepareReleaseDependencies
:CordovaLib:compileReleaseRenderscript
UP-TO-DATE
:CordovaLib:generateReleaseResValues
UP-TO-DATE
:CordovaLib:generateReleaseResources
UP-TO-DATE
:CordovaLib:packageReleaseResources
UP-TO-DATE
:CordovaLib:compileReleaseAidl
UP-TO-DATE
:CordovaLib:generateReleaseBuildConfig
UP-TO-DATE
:CordovaLib:mergeReleaseShaders UP-TO-DATE
:CordovaLib:compileReleaseShaders UP-TO-DATE
:CordovaLib:generateReleaseAssets UP-TO-DATE
:CordovaLib:mergeReleaseAssets
UP-TO-DATE
:CordovaLib:processReleaseManifest
UP-TO-DATE
:CordovaLib:processReleaseResources
UP-TO-DATE
:CordovaLib:generateReleaseSources UP-TO-DATE
:CordovaLib:incrementalReleaseJavaCompilationSafeguard
UP-TO-DATE
:CordovaLib:compileReleaseJavaWithJavac
UP-TO-DATE
:CordovaLib:processReleaseJavaRes
UP-TO-DATE
:CordovaLib:transformResourcesWithMergeJavaResForRelease
UP-TO-DATE
:CordovaLib:transformClassesAndResourcesWithSyncLibJarsForRelease UP-TO-DATE
:CordovaLib:mergeReleaseJniLibFolders UP-TO-DATE
:CordovaLib:transformNative_libsWithMergeJniLibsForRelease
UP-TO-DATE
:CordovaLib:transformNative_libsWithSyncJniLibsForRelease
UP-TO-DATE
:CordovaLib:bundleRelease UP-TO-DATE
:prepareComAndroidSupportSupportCompat2421Library
UP-TO-DATE
:prepareComAndroidSupportSupportCoreUi2421Library
UP-TO-DATE
:prepareComAndroidSupportSupportCoreUtils2421Library
UP-TO-DATE
:prepareComAndroidSupportSupportFragment2421Library
UP-TO-DATE
:prepareComAndroidSupportSupportMediaCompat2421Library
UP-TO-DATE
:prepareComAndroidSupportSupportV42421Library
UP-TO-DATE
:prepareComGoogleAndroidGmsPlayServicesBase940Library
UP-TO-DATE
:prepareComGoogleAndroidGmsPlayServicesBasement940Library
UP-TO-DATE
:prepareComGoogleAndroidGmsPlayServicesLocation940Library
UP-TO-DATE
:prepareComGoogleAndroidGmsPlayServicesTasks940Library
UP-TO-DATE
:prepareDebugDependencies
:compileDebugAidl
:compileDebugRenderscript
:generateDebugBuildConfig
UP-TO-DATE
:mergeDebugShaders
UP-TO-DATE
:compileDebugShaders
UP-TO-DATE
:generateDebugAssets
UP-TO-DATE
:mergeDebugAssets
:generateDebugResValues
UP-TO-DATE
:generateDebugResources
:mergeDebugResources
:processDebugManifest
:processDebugResources
:generateDebugSources
:incrementalDebugJavaCompilationSafeguard
UP-TO-DATE
:compileDebugJavaWithJavac
Note: /Users/rodrigopalmafanjul/appmimovistar/i+d/platforms/android/src/org/apache/cordova/file/DirectoryManager.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: /Users/rodrigopalmafanjul/appmimovistar/i+d/platforms/android/src/org/apache/cordova/file/AssetFilesystem.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Incremental compilation of 23 classes completed in 1.621 secs.
:compileDebugNdk
UP-TO-DATE
:compileDebugSources
:prePackageMarkerForDebug
:transformClassesWithDexForDebug
To run dex in process, the Gradle daemon needs a larger heap.
It currently has approximately 910 MB.
For faster builds, increase the maximum heap size for the Gradle daemon to more than 2048 MB.
To do this set org.gradle.jvmargs=-Xmx2048M in the project gradle.properties.
For more information see https://docs.gradle.org/current/userguide/build_environment.html
:mergeDebugJniLibFolders
UP-TO-DATE
:transformNative_libsWithMergeJniLibsForDebug
UP-TO-DATE
:processDebugJavaRes
UP-TO-DATE
:transformResourcesWithMergeJavaResForDebug
UP-TO-DATE
:validateDebugSigning
:packageDebug
:zipalignDebug
:assembleDebug
:cdvBuildDebug
BUILD SUCCESSFUL
Total time: 1 mins 46.587 secs
Built the following apk(s):
/Users/rodrigopalmafanjul/appmimovistar/i+d/platforms/android/build/outputs/apk/android-debug.apk
Running on macOS Sierra 10.12
Cordova 6.3.1
Cordova Android 5.2.2

Thanks for the quick response, I managed to get another fork running for now.
But I will test your fork again with Cordova Android 5.2.2 .. maybe the problem only happens with 5.1.1

I need this as soon as we can get it :)

Thank you for the awesome work. We need it very urgently to be merged with master too.

Hi everyone, does this fails to build with some cordova version?

For newbies like me: I had some problem integrating this to my project, so I had a research on how to do it. You can find the way to do that here:
http://stackoverflow.com/questions/43575839/how-to-send-http-put-request-using-ionic2-native-http-plugin

any update on this?

Any update on this guys? :)

@eluzardo-simpro You can use my fork https://github.com/miki95/cordova-HTTP where is already implemented


Wrong type definition in generated migration

When working with a legacy database that has field like the following
  `field_name` decimal(10,3) DEFAULT '1.000' NOT NULL

propel generates a wrong migration omitting the 10,3 in the type definition
This can be reproduced using make test from https://github.com/kairichard/propel-issue-decimal-default where a second migration will be created because the schema does not match with the generated migration.

It looks like this can be closed as well since #898 has been merged.


Fix compile error, update to latest cargo. Based on Florian Merz’s fork.

Hi,
based on Florian Merz´s branch, fixed the openssl compile issue happening on your and Florian’s fork by updating to latest cargo and fixing the interface issue for query and the non-existing human import.
Hope for inclusion in the crates.io variant.
BR
Jochen

Thanks for PR. I just merged #14, it should fix compilation problems.


How to make curly braces show up on a different line ?

Basically what’s written in the topic; long story short is that I prefer writing the (opening) curly braces on separate lines from the block of code under which they belong.
i.e.
if (someBoolean)
{
//do things
}
NOT
if (someBoolean){
// do things
}
????

What language?  If Javascript, set Brace Style to expand.

But this is for Javascript only ? how about php ?

You'll need to create a configuration file for PHP CS Fixer with how you want braces configured.  See https://github.com/FriendsOfPHP/PHP-CS-Fixer

What about cpp?
How can i get the new line braces format in atom for cpp language?


Manage GraphBLAS version with GB_VERSION.txt file

This is a bit out of left field and might not be necessary, but food for thought. Feel free to just close.
Add GB_VERSION.txt that contains the version of GraphBLAS to build against.
As far as I can tell this is currently only specified in the git tag. That works well for tagged commits like releases, but sometimes introduces issues for development commits.  I've done a lot of dev with extra commits to hard-code refs since the automatic methods would result in v0.0.1.
This restores the DRY principle by having the places that need the GraphBLAS version number to read it from the file. If anything this at least makes upgrades a little bit easier.
Also adds:

a script that fetches the latest version number of upstream GraphBLAS.
a workflow that can create an update PR. If this workflow were scheduled it would satisfy #3. A bit overkill, but was a fun little exercise in automation.

The tag method is still kept for releases to ensure no accidental version mismatches on release.

LGTM! I understand the limitations of the previous approach, and I would also occasionally hard-code versions during development.
One thing... I would add a shebang line in latest_suitesparse_graphblas_version.py and make sure it's executable:
#!/usr/bin/env python

One thing... I would add a shebang line in latest_suitesparse_graphblas_version.py and make sure it's executable:
#!/usr/bin/env python

Good point. Added.

Food for thought: If the version stuff becomes an issue in the future it might be worth considering this pattern: https://setuptools-git-versioning.readthedocs.io/en/stable/schemas/file/dev_release_file.html
I think all that would be required is to convert GB_VERSION from a file pointing at a GraphBLAS version to one pointing at a python-suitesparse-graphblas version, i.e. the same thing but with a ".0" (or other integer) suffix. The scripting for that would be pretty simple.
A bit overkill at the moment though.

Also note that the workflow in this PR is probably overkill. Its greatest benefit is to serve as documentation for what parts of the repo need to be modified for each update. I mostly wrote and tested it as a sort of devops practice.
Since GraphBLAS updates are not too common, the manual method might be even easier: use GitHub's edit button on GB_VERSION.txt and let GitHub create the PR.

Its greatest benefit is to serve as documentation for what parts of the repo need to be modified for each update.

heh, good to know... and I think this is a great benefit!

Since GraphBLAS updates are not too common, the manual method might be even easier: use GitHub's edit button on GB_VERSION.txt and let GitHub create the PR.

Yeah, manually specifiying the GraphBLAS version and choosing the correct tag version hasn't really been a pain point, so I'm fine keeping this simple and manual for now. This PR does solve actual pain points.
Question: can we make this work with SuiteSparse:GraphBLAS beta releases such as 8.2.1.beta3? It's nice to be able to test out his betas, and maybe we could actually publish pre-releases using his betas too. I can test this out in a day or two--I think maybe we can add a check to suitesparse.sh to specifically handle betas.
Anyway, I think this PR is good to merge. If I find any issues when I get around to playing with it in a day or two, I'll let you know!

Question: can we make this work with SuiteSparse:GraphBLAS beta releases such as 8.2.1.beta3? It's nice to be able to test out his betas, and maybe we could actually publish pre-releases using his betas too. I can test this out in a day or two--I think maybe we can add a check to suitesparse.sh to specifically handle betas.

A quick glance suggests the main sticking point would be the version regex in suitesparse.sh, which only pulls out the first three digits (i.e. 8.2.1.beta3 would result in 8.2.1):
if [[ $1 =~ refs/tags/([0-9]*\.[0-9]*\.[0-9]*)\..*$ ]]; then
    VERSION=${BASH_REMATCH[1]}

Other than that, I'm not sure how the python version logic and PyPI would handle a fifth dot field. Upload to Test PyPI to verify.

Anyway, I think this PR is good to merge. If I find any issues when I get around to playing with it in a day or two, I'll let you know!

🎉


Define languages with macro

For #205
This macro works mostly, except for a few mismatches with some tokei language names. These mismatches could probably be resolved by making the macro rule a bit more complex to allow onefetch => tokei language mappings.
This macro now works.
The implementation of the macro itself can be changed if we use types from tokei instead of defining our own type. But this is the basic idea.
It can (and probably should) be adjusted to allow color implementations as well.

This looks great @spenserblack, macro_rules! indead :smile:
Maybe you could include get_all_language_types() inside the macro:
fn get_all_language_types() -> Vec<tokei::LanguageType> {
    vec![
        $( tokei::LanguageType::$name, )*
    ]
}

So that we can remove the big Vec! at the end of the file.

It can (and probably should) be adjusted to allow color implementations as well.

Sure! tell me if you need a hand :+1:

OK, this macro should be good to go.
I've included tests built by the macro to enforce width and height, but they are ignored by default for now so the CI won't fail. They can be explicitly tested with cargo test -- --ignored.


add time filter to events

Fixes #1783  , based on https://github.com/bluesky/bluesky/pull/1505
Description

adds an early return logic branch inside the blunders.py/RunBundler/monitor/emit_event
where if any of the readings is too old, the method returns early
Motivation and Context

How Has This Been Tested?

there are two plans that get into an error state because of this small change
the first one is test_monitor in tests/test_devices.py:

@pytest.mark.parametrize("ophyd", ["v1", "v2"])
@requires_ophyd
def test_monitor(RE, ophyd):
    docs = []

    def collect(name, doc):
        docs.append(doc)

    if ophyd == "v1":
        sig = A("", name="a").s1
    else:
        sig = SigNew(name="a_s1")

    def plan():
        yield Msg("open_run")
        yield Msg("monitor", sig)
        sig._run_subs(sub_type="value")
        yield Msg("close_run")

    RE(plan(), collect)
    assert len(docs) == 4
    assert docs[2] == {
        "data": {"a_s1": 0},
        "descriptor": ANY,
        "seq_num": 1,
        "time": pytest.approx(time(), rel=0.1),
        "timestamps": {"a_s1": pytest.approx(time(), rel=0.1)},
        "filled": ANY,
        "uid": ANY,
    }

the other one is quite hard to parse and is such:

@requires_ophyd
def test_ramp(RE):
    from ophyd import StatusBase
    from ophyd.positioner import SoftPositioner
    from ophyd.sim import SynGauss

    import bluesky.utils as bsu

    RE.msg_hook = bsu.ts_msg_hook

    tt = SoftPositioner(name="mot")
    tt.set(0)
    dd = SynGauss("det", tt, "mot", 0, 3)

    st = StatusBase()

    def kickoff():
        yield Msg("null")
        for j, v in enumerate(np.linspace(-5, 5, 10)):
            RE.loop.call_later(0.1 * j, lambda v=v: tt.set(v))
        RE.loop.call_later(1.2, st._finished)
        print("YOLO")
        return st

    first = True

    def inner_plan():
        nonlocal first
        if first:
            yield from declare_stream(dd, name="primary")
            first = False

        yield from trigger_and_read([dd])

    g = ramp_plan(kickoff(), tt, inner_plan, period=0.08)
    db = DocCollector()
    RE.subscribe(db.insert)
    rs = RE(g)
    if RE.call_returns_result:
        uid = rs.run_start_uids[0]
    else:
        uid = rs[0]
    assert db.start[0]["uid"] == uid
    assert len(db.descriptor[uid]) == 2
    descs = {d["name"]: d for d in db.descriptor[uid]}

    assert set(descs) == set(["primary", "mot_monitor"])  # noqa: C405

    primary_events = db.event[descs["primary"]["uid"]]
    assert len(primary_events) > 11

    monitor_events = db.event[descs["mot_monitor"]["uid"]]
    # the 10 from the updates, 1 from 'run at subscription time'
    assert len(monitor_events) == 11

parsing the test logic now

maybe it's a detection of a bug, or the new behavior is not correct

I just rebuilt the container and all the tests worked

compared to main, test_monitor produces 3 docs instead of 4, start descriptor stop with the event missing

It is worth generating a pathological Signal subclass that lies about its timestamps to test this without sleeps.


Branch ion-trap is failing

Trying to build ion-trap at commit 9f09fb3b2f440493eea8443055e499b7a9eb3c6d failed.
More info at: https://travis-ci.com/Qiskit/qiskit-terra/jobs/242999952

seems to be an API access error


Skirmish crash when trying to respawn with no teammates alive (i think)

consistently happens when im in a lobby with myself, doesnt happen with a teammate as far as i can tell

get friends

thanks ill work on that

should be fixed as of this commit


Training

Hi, are you planning to continue training the model for better results on more data in the future?

No


Missing ABI features required for struct-args and i128 on s390x

In implementing support for struct-args and i128 to enable rustc_codegen_cranelift on s390x, I ran into two features that are not currently available with the common abi_impl implementation.

Struct arguments using an explicit buffer address
Many commonly used ABIs have special provisions for (large) struct arguments that cannot be passed in registers.  However, there are two typical flavors of these, and abi_impl only supports one of them.
Both these flavors have in common that the actual argument is copied by the caller into a buffer on the stack; the callee will then read the argument from that buffer, and is even free to modify it.  However, the two flavors differ in how the address of that buffer is communicated from the caller to the callee:

Implicit buffer address.   With this flavor, that ABI implicitly defines the location of the buffer, usually by making the buffer part of the same area used for stack slots for overflow scalar arguments.  Both caller and callee can compute the location of the buffer at compile time simply based on the function call signature; the address is not materialized as part of the run-time call sequence.
Explicit buffer address.  Here, the ABI does not define the location of the buffer itself, but leaves the compiler free to place it anywhere.  The actual location of the buffer is passed at run-time from caller to callee, typically in a register (or overflow stack slot) as if we had a pointer at this place in the function call signature.

There are pros and cons to either approach, leading to different choices of method across platform ABIs.  In particular, x86_64 nearly always uses the implicit buffer address method, with the only exception of structures whose size is not a compile-time constant.  Many other platforms, including s390x, use the the explicit buffer address method always, which allows the opportunity to the compiler to avoid actually copying the structure, e.g. if the original copy can be shown to be unused after the call so it wouldn't matter if the callee clobbers its contents.
However, the current cranelift abi_impl only supports the implicit buffer address method.   This should mostly be OK for x64_64 - I'm not sure if variable-sized structures passed by value even exist with the rustc frontend.   However, even for aarch64 I believe the current implementation is already incorrect - as far as I can see, GCC and LLVM use the explicit buffer address method on this architecture.    This is definitely also the case on s390x.
This should be straightforward to fix, and I already have an implementation.   The main change is to the ABIArg data type, which currently holds either a vector of ABISlots (registers or overflow stack slots) or a StructArg identifying a buffer location on the stack.  I've changed this to optionally provide both these items - where the semantics of providing both means that the ABISlot identifies the register (or overflow stack slot) needed to hold the address of the buffer at run time.

Type legalization in the back-end
If a function signature contains an argument of a type that cannot be directly passed in registers or overflow stack slots, but needs to use the (explicit or implicit) struct-arg buffer method,  the function signature needs to be legalized: this consists of replacing the argument of struct type by a pointer to that struct type, and adding code around function call sites as well as prologue and epilogue to convert between the two (taking the address of the struct, or dereferencing the pointer, as appropriate).
Now, usually this legalization is already done by the language front-end (whether clang or rustc), and by the time the (LLVM or cranelift) IR gets to the back end, we already have a pointer type.  And in fact the current abi_impl assumes that exactly this is the case.
But this is unfortunately not universally true.  There are a few cases whether the front end does not perform any legalization, but rather relies on the (currently LLVM) back end for that.  This is the case in particular for the i128 data type on s390x.   This remains as a scalar integer type in function call signatures, but is actually implemented using a hidden buffer like the struct-arg case.   With LLVM, this legalization is performed in the back-end - and to implement i128 in cranelift, we probably need to do the same here.
Interestingly enough, the AbiParam data type in cranelift contains a legalized_to_pointer flag which would appear to address exactly this situation.  However, as far as I can tell this flag is currently never set or used in all of cranelift.   It is not clear to me how this was originally intended to be used, or where exactly this legalization would have been done.
I have implemented the functionality for now using yet another flavor of the ABIArg data type, but that seems a bit of a hack.  One problem is that when implementing the pointer dereferencing in the prologue, the ABIArg slots only carry type information for the pointer (usually I64), and possibly the size of the buffer, but not the type of the buffer, so it is a bit unclear which instruction to use to implement the dereferencing.
Another problem is that if the buffer address is passed explicitly, but in a stack slot, we require a temporary register to load that address in order to dereference it (since the into_regs will hold the final i128 value, which lives in the vector register set on s390x, it cannot be used as address).   It's not quite clear how to allocate a temp at this stage in the compilation process.  (I'm using the "stack limit register" right now, which appears to work, but I'm not sure this is guaranteed to be OK on all platforms.)
Finally, this legalization needs to be done not just for function arguments, but also return values.   Here, abi_impl already has code to allocate a buffer in the caller and pass its address to the callee so it can place the return value there.  This is exactly what we need on s390x for a i128 return value - except for one implementation detail: cranelift passes that implicit pointer as last argument (because this is expected by wasmtime when used for the multiple return value extension), while the platform ABI requires the implicit pointer as first argument.   To work around this, I'm now passing the return pointer first if the enable_llvm_abi_extensions flag is true, and last otherwise.  This seems to work with both wasmtime and rustc_codegen_cranelift, but also feels like a hack.

@cfallin these are the problems I mentioned in our last meeting.  I'd be happy to implement a solution, but I'd appreciate some guidance / discussion on what the correct approach should be.

For i128 in the C ABI, rustc already performs the pass-by-ref transformation itself, so Cranelift just sees a pointer. For the Rust ABI however you are right that the transformation needs to happen in the backend.

except for one implementation detail: cranelift passes that implicit pointer as last argument (because this is expected by wasmtime when used for the multiple return value extension), while the platform ABI requires the implicit pointer as first argument. To work around this, I'm now passing the return pointer first if the enable_llvm_abi_extensions flag is true, and last otherwise. This seems to work with both wasmtime and rustc_codegen_cranelift, but also feels like a hack.

Wasmtime has a separate call conv variant called WasmtimeSystemV which is documented as affecting the way values are returned. I think you should match on the call conv, not the enable_llvm_abi_extensions flag. Also i128 isn't used at all by wasmtime, so it is probably fine to simply error on i128 returns on WasmtimeSystemV.

Wasmtime has a separate call conv variant called WasmtimeSystemV which is documented as affecting the way values are returned. I think you should match on the call conv, not the enable_llvm_abi_extensions flag. Also i128 isn't used at all by wasmtime, so it is probably fine to simply error on i128 returns on WasmtimeSystemV.

This seems to be working fine for me, and makes sense.   Thanks for the suggestion!

Interestingly enough, the AbiParam data type in cranelift contains a legalized_to_pointer flag which would appear to address exactly this situation. However, as far as I can tell this flag is currently never set or used in all of cranelift. It is not clear to me how this was originally intended to be used, or where exactly this legalization would have been done.

Doing a bit of archaeology on this, this was used to implement ValueConversion::Pointer, which was introduced in https://github.com/bytecodealliance/wasmtime/pull/1670 and only ever used by the 32-bit x86 backend.  So after that backend went away, the only user was gone, and when old-style legalization was removed completely a bit later on, support for ValueConversion went away with it, and no equivalent feature was ever added to the new ABI code used by new-style backends.

@uweigand thanks for the detailed writeup on these issues! At this point you will have likely paged in much more of the option-space here than I have; I wrote much of this code but my mental L2 cache has been thrashed many times over. So I'd defer to your suggested approaches above, as they sound reasonable. If my understanding is right, this is kind of a dual to the struct-arg mechanism: for struct args, the IR value is a pointer, and the ABI code allocates the space (implicit location, as you say); while here the IR value is the value stored in memory. That seems like a reasonable option to support.
I do think that if we can manage it, avoiding a separate legalization pass is ideal; I'd prefer that we handle everything inline in abi_impl, as long as the complexity is not too bad. But it seems that we could consider the large-types-passed-as-pointers when computing the stackframe layout on the caller side, and allocate extra space, and generate the stores (or loads for returns); and on the callee side, load from the pointer (or store to it).
In any case, happy to look at PRs as a next step!

In particular, x86_64 nearly always uses the implicit buffer address method, with the only exception of structures whose size is not a compile-time constant. Many other platforms, including s390x, use the the explicit buffer address method always [...] However, the current cranelift abi_impl only supports the implicit buffer address method. This should mostly be OK for x64_64 - I'm not sure if variable-sized structures passed by value even exist with the rustc frontend. However, even for aarch64 I believe the current implementation is already incorrect - as far as I can see, GCC and LLVM use the explicit buffer address method on this architecture.

Interesting; I must admit that I am not really familiar with the relevant ABI code paths because I haven't worked at all on the cg_clif integration, but it looks like there is an issue there. The standard ABI is actually a bit more nuanced - structures (or composite types, as the specification calls them) that are 16 bytes or smaller are passed as parameters in registers and/or implicitly on the stack, and via an explicit buffer address otherwise (but I am simplifying things a bit).
I128 (integral quad-word) is a fundamental data type according to the ABI and as such never uses an explicit buffer address.

Finally, this legalization needs to be done not just for function arguments, but also return values. Here, abi_impl already has code to allocate a buffer in the caller and pass its address to the callee so it can place the return value there. This is exactly what we need on s390x for a i128 return value - except for one implementation detail: cranelift passes that implicit pointer as last argument (because this is expected by wasmtime when used for the multiple return value extension)...

Return values either fit fully into registers, or go through an explicit buffer address that is kept in a register, x8, that normally is not used for parameters and return values; it is in fact the next register after the last one used for that purpose.

In particular, x86_64 nearly always uses the implicit buffer address method, with the only exception of structures whose size is not a compile-time constant. Many other platforms, including s390x, use the the explicit buffer address method always [...] However, the current cranelift abi_impl only supports the implicit buffer address method. This should mostly be OK for x64_64 - I'm not sure if variable-sized structures passed by value even exist with the rustc frontend. However, even for aarch64 I believe the current implementation is already incorrect - as far as I can see, GCC and LLVM use the explicit buffer address method on this architecture.

Interesting; I must admit that I am not really familiar with the relevant ABI code paths because I haven't worked at all on the cg_clif integration, but it looks like there is an issue there. The standard ABI is actually a bit more nuanced - structures (or composite types, as the specification calls them) that are 16 bytes or smaller are passed as parameters in registers and/or implicitly on the stack, and via an explicit buffer address otherwise (but I am simplifying things a bit).

Yes, it's similar on most platforms.  The decision which structs are passed in registers/stack slots and which are passed via buffer is made by the front end: the front end decides to either represent the parameter as some scalar type (if passed in registers / stack slots), or as a pointer type with the StructArgument attribute if passed in a buffer.   The back end then simply has to implement that decision.
This is all unaffected by this particular issue: this issue is only about the case where the front end has already decided to use the StructArgument method.  The question raised here is whether the implementation of that method in the back end is actually correct, because even just this also differs between platforms.  Specifically, some platforms (notably Intel) define the address of the buffer implicitly in the ABI, so caller and callee automatically agree on it - while other platforms allow the caller to choose any address, which is then passed explicitly via register or stack slot to the callee.
The question I had about AArch64 was that it is currently (in Cranelift) using the implicit method like Intel, while looking at the GCC and LLVM implementation, I see those using the explicit method.  So I suspect this is actually incorrect in Cranelift, but as I have no AArch64 hardware to verify, I didn't implement any change.
Now that https://github.com/bytecodealliance/wasmtime/pull/4585 has landed, it should be straightforward to fix this in the AArch64 back end as well (if indeed something needs to be fixed).

I128 (integral quad-word) is a fundamental data type according to the ABI and as such never uses an explicit buffer address.

Yes, s390x is really special here, unfortunately.


Can't get this to work because it requires node core modules

Hi,
I'm trying to use this, but I get errors immediately.
If I include
import Dropbox from 'react-native-dropbox-sdk';
I get
Unable to resolve module 'url' from ./node_modules/node-fetch/index.js:  Module does not exist in the module map
'url' is a node core module, which is not usable inside React-Native.
How do you get around this?

Hi @jholton - Sorry I hadn't updated this project sooner, but I've moved away from trying to support dropbox in the React-Native side. There are too many low-level components Dropbox requires (UInt8 arrays, Buffers/Blobs etc.) that RN just doesn't have. I was not able to get this working correctly in every case, so I'll most likely add Dropbox to my existing project by adding Objective-C/Java code.

But to quickly address your issue - It looks like you installed react-native-dropbox-sdk, but perhaps you're not using rn-nodeify? I've used it several times to "hack in" several necessary core modules. I've used it here.
I apologise for misleading you..

Thanks for the info!


Can we build index on original data but prune by the distance between transformed vectors and queries?

In this way, we can conduct experiments more fairly.

Also, we can keep the original vectors. If we directly evict them from memory, it is more or less radical.

For the first comment, they are equivalent. This is because for IVF and HNSW, the construction depends only on the distance between vectors and the transformation does not distort the distances. For other indexes, yes, users can construct the indexes with the original vectors and transform the vectors afterwards.
For the second comment, it depends on the needs, but in general, I think it is unnecessary to keep them (as it will have double sized space consumption). If the users only target the IDs of the KNN objects, the original vectors will not be used. If the original vectors are required, the algorithm can also inversely transforming the vectors at the end of the algorithm.

For the first comment, they are equivalent. This is because for IVF and HNSW, the construction depends only on the distance between vectors and the transformation does not distort the distances. For other indexes, yes, users can construct the indexes with the original vectors and transform the vectors afterwards.
For the second comment, it depends on the needs, but in general, I think it is unnecessary to keep them (as it will have double sized space consumption). If the users only target the IDs of the KNN objects, the original vectors will not be used. If the original vectors are required, the algorithm can also inversely transforming the vectors at the end of the algorithm.

Very good.  Strong paper!

For the first comment, they are equivalent. This is because for IVF and HNSW, the construction depends only on the distance between vectors and the transformation does not distort the distances. For other indexes, yes, users can construct the indexes with the original vectors and transform the vectors afterwards.
For the second comment, it depends on the needs, but in general, I think it is unnecessary to keep them (as it will have double sized space consumption). If the users only target the IDs of the KNN objects, the original vectors will not be used. If the original vectors are required, the algorithm can also inversely transforming the vectors at the end of the algorithm.

Very good.
By the way, congrats on the acceptance of your wonderful work in Sigmod!

Thank you for your kind congratulations and thoughtful comments. Have a good day :)

PROGRESS: I have tried this idea, but the performance degrades. The reason might be the frequent memory access to find the transformed vectors of candidate during distance calculation.

It might be due to the implementation. The transformed vectors are supposed to be stored where the original vectors are (i.e., to replace the original vectors). In this case, it will not introduce extra costs of finding the transformed vectors.

Exactly.


Mixture of model prediction and update

Hello Ilya,
first, thanks for you amazing work. I would have one question about a way how you have designed an A3C training.
What you basically do is that you play N steps and you store all the values, log_probs and entropies during this phase and you store all these variables in lists. After that, you perform an update, where you use these variables.
Another approach would be to split this into two methods:

predict(state), which would return action values
update(states, actions, rewards), which would update the model

In this setup, you can use volatile=True in predict, but on the other hand, in update method you have to compute all these values, log_probs and entropies by forward pass on all the states.
My question is, did you consider this second approach? Why did you chose the first one? Is it more efficient or its just simpler to implement? I am trying to make my own A3C implementation and definitely I will try out myself which is more efficient, but I am curious about your opinion.
Thanks

The first approach is faster on CPU in practice (no need to do a second pass, no performance gain from using batches).
However, the second one can be faster for GPU training. I implemented the second approach here:
https://github.com/ikostrikov/pytorch-a2c-ppo-acktr

Thanks, that seems to be reasonable. Currently, I am using Macbook Pro with i7 processor with 4 cores and I don't have any GPU available so that's why I am trying experimenting with A3C. I will try to implement first approach. Also, is it possible to use PPO algorithm with this asynchronous settings? I didn't read about PPO much and just wonder if it is possible. Also do you think PyTorch is good fit for implementing CPU reinforcement algorithms or are there any better? I like the way how PyTorch works and is easy to use.

4 cores is fine for mujoco but not enough for atari (unless you are ready to wait for several days to get results).
Synchronous methods are much easier to debug and they usually perform just as well as asynchronous methods. See the OpenAI blogpost about a2c/acktr where they mention that.
In terms of the frameworks, the difference between all the frameworks nowadays is marginal (in terms of time you need to spend in different frameworks to implement exactly the same thing). It's usually +-1-2 hours between TensorFlow and PyTorch depending on a specific task. But PyTorch tends to work slightly faster than TensorFlow, especially on CPU. So if performance is the major concern then I would recommend PyTorch.

Thanks for you time, it helps a lot. I will not use Atari nor Mujoco. I am trying to implement my own environment based on descrete grid and experiment with it, so hopefully my processor is enough.


Bedrock production support

Describe the feature
We need a production-ready Bedrock SDK for Rust.
Use Case
We are building an ultra-fast LLM pipeline on Rust and, as a consequence, need a Bedrock SDK for Rust.
Proposed Solution
I'm not sure what makes the current crate non-production-ready, but getting it to production-ready state is what we need.
https://crates.io/crates/aws-sdk-bedrock
Other Information
No response
Acknowledgements

[ ] I may be able to implement this feature request
[ ] This feature might incur a breaking change

A note for the community

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue, please leave a comment

Hi @prattcmp - I believe we can make this easier
I’m the maintainer of LiteLLM - we allow you to deploy an LLM proxy to call 100+ LLMs in 1 format - Bedrock, OpenAI, Anthropic etc https://github.com/BerriAI/litellm/tree/main/openai-proxy.
If this looks useful (we're used in production)- please let me know how we can help.

If this looks useful (we're used in production)- please let me know how we can help.
Usage
Bedrock request

curl http://<IP_ADDRESS>:8000/v1/chat/completions \

  -H "Content-Type: application/json" \

  -d '{

     "model": "bedrock/anthropic.claude-instant-v1",

     "messages": [{"role": "user", "content": "Say this is a test!"}],

     "temperature": 0.7

   }'

gpt-3.5-turbo request

curl http://<IP_ADDRESS>:8000/v1/chat/completions \

  -H "Content-Type: application/json" \

  -d '{

     "model": "gpt-3.5-turbo",

     "messages": [{"role": "user", "content": "Say this is a test!"}],

     "temperature": 0.7

   }'

claude-2 request

curl http://<IP_ADDRESS>:8000/v1/chat/completions \

  -H "Content-Type: application/json" \

  -d '{

     "model": "claude-2",

     "messages": [{"role": "user", "content": "Say this is a test!"}],

     "temperature": 0.7

   }'

I like this idea, but we don't want to add another layer of cost or latency.

@prattcmp thanks for the feedback -if we could minimize latency for the proxy would you use it ?

@prattcmp thanks for the feedback -if we could minimize latency for the proxy would you use it ?

We wouldn't pay for it, but if it were a well-maintained, open-source SDK, we might consider using it

Hi @prattcmp, as you can tell, the AWS Rust SDK (and all of the client crates) are currently in Developer Preview. For anyone not sure what we mean when we say Developer Preview, check out https://docs.aws.amazon.com/sdkref/latest/guide/maint-policy.html
What makes this Developer Preview and not GA is we're still developing this SDK. When we release things as GA, we are making certain backward compatibility guarantees that we don't have with Developer Preview. GA products are expected to be generally stable at both runtime and compile time. We're not ready for that with Rust yet.
I'm closing this issue because we won't have any more information to add here until we're ready to release the SDK.

So right now the Rust SDK doesn't support AWS Bedrock invocations?

bedrock is fully supported

bedrock is fully supported

Sorry my bad, I was using the aws-sdk-bedrock crate instead of the aws-sdk-bedrockruntime crate. Perhaps some examples should be made to clarify this?


DllNotFoundException on Apple Silicon chips

Hi!
I've tried to get list of all MIDI devices on my MacBook m1 with .NET 6 but InputDevice.GetAll() throws
Unhandled exception. System.DllNotFoundException: Unable to load shared library 'Melanchall_DryWetMidi_Native64' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: dlopen(libMelanchall_DryWetMidi_Native64, 0x0001): tried: 'libMelanchall_DryWetMidi_Native64' (no such file), '/usr/local/lib/libMelanchall_DryWetMidi_Native64' (no such file), '/usr/lib/libMelanchall_DryWetMidi_Native64' (no such file), '/Users/dvorobey/RiderProjects/Piano/Piano/bin/Debug/net6.0/libMelanchall_DryWetMidi_Native64' (no such file)    at Melanchall.DryWetMidi.Multimedia.CommonApi64.GetApiType()    at Melanchall.DryWetMidi.Multimedia.CommonApi64.Api_GetApiType()    at Melanchall.DryWetMidi.Multimedia.MidiDevicesSession.GetSessionHandle()    at Melanchall.DryWetMidi.Multimedia.MidiDevice.EnsureSessionIsCreated()    at Melanchall.DryWetMidi.Multimedia.InputDevice.GetAll()    at InputDeviceExample.Program.Main(String[] args) in /Users/dvorobey/RiderProjects/Piano/Piano/Program.cs:line 11
Here is a code that I tried to launch
static void Main(string[] args)         {             var devices = InputDevice.GetAll();          }
What I doing wrong?

Hi,
Hmm, can you please show me content of the folder where your program built? Can you see Melanchall_DryWetMidi_Native64.dylib there?
Also maybe it's caused by M1 since dylib was built on virtual machine and I suppose it doesn't emulate M1 CPU. Well, right now let's start from the output folder as I stated at the start of the comment.

Do you have any ideas?

Unfortunately no :( I need to think how we can diagnose the problem.

Can you please say you have .NET 6 installed for x64 or for Arm64?

Well, do you work in Visual Studio for Mac?

I'm using JetBrains Rider and I've installed Melanchall.DryWetMidi from the NuGet gallery

Well, it's a kind of mystic for me. So I really need your help. So please let's start from simple steps.

Your .NET version is 6.0.301. I suggest to try to update the version. Can you please completely uninstall .NET 6 and install again from https://dotnet.microsoft.com/en-us/download/dotnet/6.0:
. Does your program work now?
Take dylib from this archive libMelanchall_DryWetMidi_Native64.zip (I've rebuilt the file with new name, maybe simple renaming doesn't work) and put it to the folder with your csproj. In Rider set properties for the file: Content, Copy always to out folder. Rebuild the project. Does your program work now?
What if you put the fole to /usr/lib/? Does your program work now?

If steps above didn't lead to the program working, let's try to start from simple apps. I've prepared three different solutions. Please try each one and tell whether something works or not:

CheckDllImport.zip
CheckDllImportLibPrefix.zip
CheckDllImportLibPrefixAndExtension.zip

@ThaddeusChristopher Thanks a lot for the info!

Ah cross platform mysteries, so much fun :-)

Yes...
Well, I need your help a little more thus. I've prepared super simple solution with simple native lib built on this code:
int Foo() { return 123; }

Here the solution: SimpleNativeDllImport.zip. Please try it and let me know if DllNotFoundException still occur.
If you have the same error, we need to go further. Unfortunatelu I have no possibility to build dylib on M1, so I need your help again.
Please open Xcode, create new Library/Framework (if I remember a template name correctly). choose Plain C/C++ language, paste the code above into the editor and build a dylib. Then put it near the solution (rename it to test.dylib if needed or change library name in the DllImport inside C# code). Does SimpleNativeDllImport works fine now?

I've just built dylib on macOS 12 VM image and it seems it has different binary content. Please check with it too: test.zip. Maybe miracle will happen :)

No joy from from either zip above sadly. Bummer!
The new dylib extracted from "test.zip" just posted gets the same error: System.DllNotFoundException: Unable to load shared library 'test' or one of its dependencies.
Its dylib file size is half that of the others though, that's interesting.

I used Xcode to build the requested dylib file. Note: The Xcode Library template comes with no files, so I made a main.cpp and a matching main.h. Apparently when they build the dylib goes to some far-away folder (can find where this is like so: Product >> Show Build Folder).
Some interesting result:

Sounds like permissions related trouble?
Actually this does tell us one interesting thing: The fact that Apple prevents the dylib file from running proves they at least know that it is there :-)
Not much useful info on file properties (file is not "Locked" at least):

@ThaddeusChristopher Many many thanks!
By the way, you can simplify your Xcode project. You don't need to use C++ there, you can create just single C file (for example, main.c) and write this code in it:
int Foo() { return 123; }

Can you please try to create dylib in this way so we can be sure that all works fine? Also can you please the version of your Xcode?
I really appreciate that you're spending your time to help me!

Yes, I tried the "main.c" approach in a new project and had the experience above (no c++ involved, sorry for not specifying).
Alright, I guess this must be the place to make those changes:

TestDylib2.zip
See zip attached above. I have utterly no idea what to put in those fields :-)

And here you can see where Xcode drops the dylib. Seems needlessly complicated but hey, it's Apple town...

Thanks!
Another one experiment from me, heh. Can you please check the new file: test.zip?

I think that did it! Well done! This must be the "universal" dylib format?
qadept@QAs-Mac-mini ~ % /Users/qadept/Downloads/SimpleNativeDllImport/SimpleNativeDllImport/bin/Debug/net6.0/SimpleNativeDllImport ; exit;
Foo called: 123
Everything fine! Press any key to exit the program...

@ThaddeusChristopher Thanks again!

This must be the "universal" dylib format?

No, this is just arm64 version (I suppose :)).
Well, to close the subject about dylib format I've prepared three new solutions:

CheckArm64Dylib.zip
CheckUniversalDylib.zip
CheckX86_64Dylib.zip

On VMs I have access to programs 2 and 3 are executed fine printing the correct number (which I've changed to eliminare possible copy/paste errors). The program 1 reasonably failed since VMs are Intel-based.
Please run all three solutions on your M1 chip. We expect that programs 1 and 2 will be executed fine and the program 3 will fail on your machine.

No problem!
All went as expected.
CheckArm64Dylib.zip
Builds and runs without issue
CheckUniversalDylib.zip
Builds and runs without issue
CheckX86_64Dylib.zip
Fails to build (DLL not found)

I've switched building of macOS binaries to universal format, all my tests are run successfully. But it's for Intel-based VMs, so it's unfortunately says nothing about how it will work on Apple Silicon. I suppose runtime issues are possible.
So if you agree to run unit tests on your machine, it would be great. But I should to note that complete tests run takes noticeable amount of time. Tests are logically divided into two categories: Core and Multimedia. Each category runs for about 20-25 minutes.
Core tests can be run with your regular work on computer in parallel. But it's desired for Multimedia tests to be run without any user activity on a machine. It's because of the nature of those tests. Many of them checks that specific events occurred within a small amount of time (like 30 or even 15 ms), so any other activity can affect tests and they will fail.

Sure I can let those sit uninterrupted on this station for a bit. Do you have a link to the updated solution I can download and run test for?

@ThaddeusChristopher Thanks for responding! I'll try to give as detailed instruction as possible.
Well, first of all we need to clone the library repo on your machine. Suppose you're in Terminal in folder X.  At this point we need to place the files from Native.zip archive in the X/drywetmidi/DryWetMidi folder.
Then
dotnet build DryWetMidi.Tests/Melanchall.DryWetMidi.Tests.csproj --configuration ReleaseTest --framework net6.0

Core tests
If everything builds OK, we're now ready to run core tests. Supposing you're in X/drywetmidi in Terminal:
dotnet test DryWetMidi.Tests/Melanchall.DryWetMidi.Tests.csproj --blame --no-build --configuration ReleaseTest --filter "(FullyQualifiedName~Melanchall.DryWetMidi.Tests.Core|FullyQualifiedName~Melanchall.DryWetMidi.Tests.Common|FullyQualifiedName~Melanchall.DryWetMidi.Tests.Composing|FullyQualifiedName~Melanchall.DryWetMidi.Tests.Interaction|FullyQualifiedName~Melanchall.DryWetMidi.Tests.MusicTheory|FullyQualifiedName~Melanchall.DryWetMidi.Tests.Standards|FullyQualifiedName~Melanchall.DryWetMidi.Tests.Tools)" --framework net6.0 --verbosity normal

I expect the core set should pass. Since tests there are relying on .NET itself, not on native binaries prepared by me.
Multimedia tests
It's time to have some fun :) First of all we need to place dylib from the SendTestData.zip archive to X/DryWetMidi.Tests/bin/ReleaseTest/net6.0.
Then take please the app from the LoopbackDevice.zip archive. Now we need to open separate Terminal window and run the LoopbackDevice program there:
./LoopbackDevice "MIDI A" "MIDI B" "MIDI C" &

Don't close this Terminal instance, we now have three virtual MIDI ports running within the app.
And now we're ready to run multimedia tests (in the first Terminal window of course):
dotnet test --blame --no-build --configuration ReleaseTest --filter "(FullyQualifiedName~.Multimedia)" --framework net6.0 --verbosity normal

Thank you very much!

Core tests failing with stack overflow. This could be because the "dotnet" terminal command seems to be preferring Visual Studio Preview over the non-preview version also installed. Hence when I ran the "build" function it built using VS for Mac Preview. At a glance I'm uncertain how to re-map the "dotnet" command to the non-preview VS, assuming that is what the issue here is.

qadept@QAs-Mac-mini drywetmidi % dotnet test DryWetMidi.Tests/Melanchall.DryWetMidi.Tests.csproj --blame --no-build --configuration ReleaseTest --filter "(FullyQualifiedName~Melanchall.DryWetMidi.Tests.Core|FullyQualifiedName~Melanchall.DryWetMidi.Tests.Common|FullyQualifiedName~Melanchall.DryWetMidi.Tests.Composing|FullyQualifiedName~Melanchall.DryWetMidi.Tests.Interaction|FullyQualifiedName~Melanchall.DryWetMidi.Tests.MusicTheory|FullyQualifiedName~Melanchall.DryWetMidi.Tests.Standards|FullyQualifiedName~Melanchall.DryWetMidi.Tests.Tools)" --framework net6.0 --verbosity normal
Build started 7/25/2022 11:29:43 AM.
Test run for /Users/qadept/Documents/drywetmidi/drywetmidi/DryWetMidi.Tests/bin/ReleaseTest/net6.0/Melanchall.DryWetMidi.Tests.dll (.NETCoreApp,Version=v6.0)
Microsoft (R) Test Execution Command Line Tool Version 17.3.0 (arm64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
The active test run was aborted. Reason: Test host process crashed : Stack overflow.
   at Submission#0+<<Initialize>>d__0.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.__Canon, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.__Canon ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[System.__Canon, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.__Canon ByRef)
   at Submission#0.<Initialize>()
   at Submission#0.<Factory>(System.Object[])
   at Microsoft.CodeAnalysis.Scripting.ScriptExecutionState+<RunSubmissionsAsync>d__9`1[[System.__Canon, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.CodeAnalysis.Scripting.ScriptExecutionState+<RunSubmissionsAsync>d__9`1[[System.__Canon, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], Microsoft.CodeAnalysis.Scripting, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<RunSubmissionsAsync>d__9`1<System.__Canon> ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Microsoft.CodeAnalysis.Scripting.ScriptExecutionState+<RunSubmissionsAsync>d__9`1[[System.__Canon, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], Microsoft.CodeAnalysis.Scripting, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<RunSubmissionsAsync>d__9`1<System.__Canon> ByRef)
   at Microsoft.CodeAnalysis.Scripting.ScriptExecutionState.RunSubmissionsAsync[[System.__Canon, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.Collections.Immutable.ImmutableArray`1<System.Func`2<System.Object[],System.Threading.Tasks.Task>>, System.Func`2<System.Object[],System.Threading.Tasks.Task>, System.Runtime.CompilerServices.StrongBox`1<System.Exception>, System.Func`2<System.Exception,Boolean>, System.Threading.CancellationToken)
   at Microsoft.CodeAnalysis.Scripting.Script`1+<RunSubmissionsAsync>d__21[[System.__Canon, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.CodeAnalysis.Scripting.Script`1+<RunSubmissionsAsync>d__21[[System.__Canon, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], Microsoft.CodeAnalysis.Scripting, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<RunSubmissionsAsync>d__21<System.__Canon> ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Microsoft.CodeAnalysis.Scripting.Script`1+<RunSubmissionsAsync>d__21[[System.__Canon, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], Microsoft.CodeAnalysis.Scripting, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<RunSubmissionsAsync>d__21<System.__Canon> ByRef)
   at Microsoft.CodeAnalysis.Scripting.Script`1[[System.__Canon, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].RunSubmissionsAsync(Microsoft.CodeAnalysis.Scripting.ScriptExecutionState, System.Collections.Immutable.ImmutableArray`1<System.Func`2<System.Object[],System.Threading.Tasks.Task>>, System.Func`2<System.Object[],System.Threading.Tasks.Task>, System.Func`2<System.Exception,Boolean>, System.Threading.CancellationToken)
   at Microsoft.CodeAnalysis.Scripting.Script`1[[System.__Canon, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].RunAsync(System.Object, System.Func`2<System.Exception,Boolean>, System.Threading.CancellationToken)
   at Microsoft.CodeAnalysis.Scripting.Script`1[[System.__Canon, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].RunAsync(System.Object, System.Threading.CancellationToken)
   at Microsoft.CodeAnalysis.CSharp.Scripting.CSharpScript.RunAsync[[System.__Canon, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.String, Microsoft.CodeAnalysis.Scripting.ScriptOptions, System.Object, System.Type, System.Threading.CancellationToken)
   at Microsoft.CodeAnalysis.CSharp.Scripting.CSharpScript.EvaluateAsync[[System.__Canon, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.String, Microsoft.CodeAnalysis.Scripting.ScriptOptions, System.Object, System.Type, System.Threading.CancellationToken)
   at Melanchall.DryWetMidi.Tests.Common.TestFilesProvider.GetValidFileReference(System.String, Boolean ByRef)
   at Melanchall.DryWetMidi.Tests.Core.MidiFileTests.CheckValidFilesReadingByReferences()
   at System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Span`1<System.Object> ByRef, System.Signature, Boolean, Boolean)
   at System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
   at NUnit.Framework.Internal.Reflect.InvokeMethod(System.Reflection.MethodInfo, System.Object, System.Object[])
   at NUnit.Framework.Internal.MethodWrapper.Invoke(System.Object, System.Object[])
   at NUnit.Framework.Internal.Commands.TestMethodCommand.InvokeTestMethod(NUnit.Framework.Internal.TestExecutionContext)
   at NUnit.Framework.Internal.Commands.TestMethodCommand.RunTestMethod(NUnit.Framework.Internal.TestExecutionContext)
   at NUnit.Framework.Internal.Commands.TestMethodCommand.Execute(NUnit.Framework.Internal.TestExecutionContext)
   at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand+<>c__DisplayClass1_0.<Execute>b__0()
   at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.RunTestMethodInThreadAbortSafeZone(NUnit.Framework.Internal.TestExecutionContext, System.Action)
   at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.Execute(NUnit.Framework.Internal.TestExecutionContext)
   at NUnit.Framework.Internal.Execution.SimpleWorkItem+<>c__DisplayClass4_0.<PerformWork>b__0()
   at NUnit.Framework.Internal.ContextUtils+<>c__DisplayClass1_0`1[[System.__Canon, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<DoIsolated>b__0(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at NUnit.Framework.Internal.ContextUtils.DoIsolated(System.Threading.ContextCallback, System.Object)
   at NUnit.Framework.Internal.ContextUtils.DoIsolated[[System.__Canon, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.Func`1<System.__Canon>)
   at NUnit.Framework.Internal.Execution.SimpleWorkItem.PerformWork()
   at NUnit.Framework.Internal.Execution.WorkItem.RunOnCurrentThread()
   at NUnit.Framework.Internal.Execution.WorkItem.Execute()
   at NUnit.Framework.Internal.Execution.ParallelWorkItemDispatcher.Dispatch(NUnit.Framework.Internal.Execution.WorkItem, NUnit.Framework.Internal.Execution.ParallelExecutionStrategy)
   at NUnit.Framework.Internal.Execution.ParallelWorkItemDispatcher.Dispatch(NUnit.Framework.Internal.Execution.WorkItem)
   at NUnit.Framework.Internal.Execution.CompositeWorkItem.RunChildren()
   at NUnit.Framework.Internal.Execution.CompositeWorkItem.PerformWork()
   at NUnit.Framework.Internal.Execution.WorkItem.RunOnCurrentThread()
   at NUnit.Framework.Internal.Execution.WorkItem.Execute()
   at NUnit.Framework.Internal.Execution.TestWorker.TestWorkerThreadProc()
   at System.Threading.Thread+StartHelper.Callback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback,

The active Test Run was aborted because the host process exited unexpectedly. Please inspect the call stack above, if available, to get more information about where the exception originated from.
The test running when the crash occurred:
Melanchall.DryWetMidi.Tests.Core.MidiFileTests.CheckValidFilesReadingByReferences

This test may, or may not be the source of the crash.

Attachments:
  /Users/qadept/Documents/drywetmidi/drywetmidi/DryWetMidi.Tests/TestResults/fa6ea368-4ba8-4e25-83ed-59ce05fdbdd7/Sequence_348532e4fad74c1eb3f2a9cd6efd392c.xml
Passed!  - Failed:     0, Passed:   283, Skipped:     0, Total:   283, Duration: 10 s - /Users/qadept/Documents/drywetmidi/drywetmidi/DryWetMidi.Tests/bin/ReleaseTest/net6.0/Melanchall.DryWetMidi.Tests.dll (net6.0)
Test Run Aborted with error System.Exception: One or more errors occurred.
 ---> System.Exception: Unable to read beyond the end of the stream.
   at System.IO.BinaryReader.Read7BitEncodedInt()
   at System.IO.BinaryReader.ReadString()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.LengthPrefixCommunicationChannel.NotifyDataAvailable()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TcpClientExtensions.MessageLoopAsync(TcpClient client, ICommunicationChannel channel, Action`1 errorHandler, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---.

Build FAILED.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:27.26

Sequence_348532e4fad74c1eb3f2a9cd6efd392c.xml.zip
According to the XML it failed on "CheckValidFilesReadingByReferences"

Oh sorry, I gave you command for Windows. Here the correct one:
dotnet test DryWetMidi.Tests/Melanchall.DryWetMidi.Tests.csproj --blame --no-build --configuration ReleaseTest --filter "((FullyQualifiedName~Melanchall.DryWetMidi.Tests.Core|FullyQualifiedName~Melanchall.DryWetMidi.Tests.Common|FullyQualifiedName~Melanchall.DryWetMidi.Tests.Composing|FullyQualifiedName~Melanchall.DryWetMidi.Tests.Interaction|FullyQualifiedName~Melanchall.DryWetMidi.Tests.MusicTheory|FullyQualifiedName~Melanchall.DryWetMidi.Tests.Standards|FullyQualifiedName~Melanchall.DryWetMidi.Tests.Tools)&Name!=CheckValidFilesReadingByReferences)" --framework net6.0 --verbosity normal

Glad it was an easy thing, ha. Looks like core tests run fine:
qadept@QAs-Mac-mini drywetmidi % dotnet test DryWetMidi.Tests/Melanchall.DryWetMidi.Tests.csproj --blame --no-build --configuration ReleaseTest --filter "((FullyQualifiedName~Melanchall.DryWetMidi.Tests.Core|FullyQualifiedName~Melanchall.DryWetMidi.Tests.Common|FullyQualifiedName~Melanchall.DryWetMidi.Tests.Composing|FullyQualifiedName~Melanchall.DryWetMidi.Tests.Interaction|FullyQualifiedName~Melanchall.DryWetMidi.Tests.MusicTheory|FullyQualifiedName~Melanchall.DryWetMidi.Tests.Standards|FullyQualifiedName~Melanchall.DryWetMidi.Tests.Tools)&Name!=CheckValidFilesReadingByReferences)" --framework net6.0 --verbosity normal
Build started 7/25/2022 1:19:58 PM.
Test run for /Users/qadept/Documents/drywetmidi/drywetmidi/DryWetMidi.Tests/bin/ReleaseTest/net6.0/Melanchall.DryWetMidi.Tests.dll (.NETCoreApp,Version=v6.0)
Microsoft (R) Test Execution Command Line Tool Version 17.3.0 (arm64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.

Passed!  - Failed:     0, Passed:  6317, Skipped:     0, Total:  6317, Duration: 2 m 57 s - /Users/qadept/Documents/drywetmidi/drywetmidi/DryWetMidi.Tests/bin/ReleaseTest/net6.0/Melanchall.DryWetMidi.Tests.dll (net6.0)

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:03:12.18

OK, the loopback app has this suspicious output when sudo-run (it also returns to terminal after suspending):
qadept@QAs-Mac-mini ~ % sudo ./LoopbackDevice "MIDI A" "MIDI B" "MIDI C" &
[1] 27808
qadept@QAs-Mac-mini ~ %
[1]  + suspended (tty output)  sudo ./LoopbackDevice "MIDI A" "MIDI B" "MIDI C"
qadept@QAs-Mac-mini ~ %

When I then proceed to run the Multimedia tests they all fail seemingly from being unable to find the virtual MIDI devices in question:
Error Message:
   System.ArgumentException : There is no output MIDI device 'MIDI A'. (Parameter 'name')

Hmm... What if we remove & at the end of the command?

Without ampersand "command not found":
qadept@QAs-Mac-mini ~ % sudo ./LoopbackDevice "MIDI A" "MIDI B" "MIDI C"
Password:
sudo: ./LoopbackDevice: command not found
qadept@QAs-Mac-mini ~ %

With the ampersand it also says that, but looks different (displays a number, maybe process id? and "exit"):
qadept@QAs-Mac-mini ~ % sudo ./LoopbackDevice "MIDI A" "MIDI B" "MIDI C" &
[1] 859
qadept@QAs-Mac-mini ~ % sudo: ./LoopbackDevice: command not found
[1]  + exit 1     sudo ./LoopbackDevice "MIDI A" "MIDI B" "MIDI C"
qadept@QAs-Mac-mini ~ %

And without "sudo" it states "permission denied", which does suggest it can actually see it:
qadept@QAs-Mac-mini ~ % ./LoopbackDevice "MIDI A" "MIDI B" "MIDI C" &
[1] 884
zsh: permission denied: ./LoopbackDevice
[1]  + exit 126   ./LoopbackDevice "MIDI A" "MIDI B" "MIDI C"

Is there a particular directory I should run LoopbackDevice from maybe? Tried user folder, downloads, documents.
I double-checked if the Apple Systems >> Security app was blocking anything. Did not state that it was.

The actual error when running the test is:
A total of 1 test files matched the specified pattern.
  Failed CheckEventsReceivingOnConnectedDevices [813 ms]
  Error Message:
   System.ArgumentException : There is no output MIDI device 'MIDI A'. (Parameter 'name')
  Stack Trace:
     at Melanchall.DryWetMidi.Multimedia.OutputDevice.GetByName(String name) in /Users/qadept/Documents/drywetmidi/drywetmidi/DryWetMidi/Multimedia/OutputDevice/OutputDevice.cs:line 448
   at Melanchall.DryWetMidi.Tests.Multimedia.DevicesConnectorTests.CheckEventsReceiving(IReadOnlyList`1 eventsToSend) in /Users/qadept/Documents/drywetmidi/drywetmidi/DryWetMidi.Tests/Multimedia/DevicesConnector/DevicesConnectorTests.cs:line 147
   at Melanchall.DryWetMidi.Tests.Multimedia.DevicesConnectorTests.CheckEventsReceivingOnConnectedDevices() in /Users/qadept/Documents/drywetmidi/drywetmidi/DryWetMidi.Tests/Multimedia/DevicesConnector/DevicesConnectorTests.cs:line 32

@ThaddeusChristopher Are you sure you're currently in the folder with LoopbackDevice in Terminal? ./LoopbackDevice means "to run LoopbackDevice placed in the current directory".
So can you please cd to the folder where you've downloaded LoopbackDevice to (or open Terminal from that folder via right-click menu) and try again (I think we don't need ampersand)?

Perhaps that app you compiled on your non-ARM64 processor can't run on this ARM64 processor? Man, Apple really shot themselves in the foot with this new architecture haha.
I feel your pain, but like I said I copied the LoopbackDevice app (extracted from the zip) to various folders and ran it with the results posted above. I can "ls" inside said folders and definitely do see the app present in its extracted form. Further, running without "sudo" denies permission to run it, acknowledging that it is there.
I suspect either (1) it's another ARM64 thing, or (2) Apple is blocking its execution in some way, something they like doing a little too much haha. Or possibly this dev station lacks some tools needed to run it, such as the right shared libraries, etc...
That said, compiling it inside Xcode on this ARM64 processor does appear able to work. We're getting there :-)
So what I was able to do was get it running via Terminal from Xcode's Products >> Build Output folder, like so:
qadept@QAs-Mac-mini Debug % sudo ./midiLoopbackTest1 "MIDI A" "MIDI B" "MIDI C"
Creating client...
Creating port 'MIDI A'...
    creating source...
    creating destination...
OK
Creating port 'MIDI B'...
    creating source...
    creating destination...
OK
Creating port 'MIDI C'...
    creating source...
    creating destination...
OK
Waiting for data...

So... yes, now I can run the Multimedia Tests. Results will be forthcoming...

Results:
Build started 7/26/2022 12:53:51 PM.
Test run for /Users/qadept/Documents/drywetmidi/drywetmidi/DryWetMidi.Tests/bin/ReleaseTest/net6.0/Melanchall.DryWetMidi.Tests.dll (.NETCoreApp,Version=v6.0)
Microsoft (R) Test Execution Command Line Tool Version 17.3.0 (arm64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
    - Failed:     0, Passed:   516, Skipped:     0, Total:   516, Duration: 21 m 34 s - /Users/qadept/Documents/drywetmidi/drywetmidi/DryWetMidi.Tests/bin/ReleaseTest/net6.0/Melanchall.DryWetMidi.Tests.dll (net6.0)

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:21:36.32

There is a TestResults folder but its output for this run seems to be empty. I'll take it that's a good sign?

I have a proper word for the results – AWESOME!  Counts of passed tests are correct:

Core: 6317
Multimedia: 516

As for LoopbackDevice built by me, I built it using universal format, merging x86_64 and arm64 architectures. But maybe it works fine for dylibs, but need a bit different steps for apps.

Man, Apple really shot themselves in the foot with this new architecture haha.

Absolutely agreed with you :)
Well, I'll build a prerelease NuGet package and let you know.
And many many many thank you, @ThaddeusChristopher. Support of Apple Silicon chips is a critical task in my opinion and I really don't know how it could be solved without you.

Thank you for building such a high quality library in your spare time! And if you ever feel like adding Android/iOS support for modern Net 6 MAUI projects that'd sure be great ;-)

There is a chance that the library will work on iOS due to macOS support since both platforms has similar API. That's just a theory, I didn't tested it.
As for Android, it's much more difficult since I even don't know how to setup test environment within my CI process.
But of course I have such plans :)

@ThaddeusChristopher @vorobeyDmitriy @dmitryvorobey The fix is available in a prerelease version of the DryWetMIDI on NuGet – 6.1.2-prerelease3.

@melanchall the nativeless version of this project works on Android and iOS great ;)

:rocket: 6.1.2 version is released now!
Prerelease NuGet packages will be unlisted soon, so please update the package references to the new version.
Thanks for using the library!


[Doc]: Clarify project language

Describe your documentation update

At the main page of the github repository, it detects major program language as Jupyter notebook instead of Python because demo notebooks being counted. To clarify that the package is built using Python, demo directory needs to be excluded when github detects used languages.

Relevant info:
https://stackoverflow.com/questions/29932342/is-there-a-way-to-ignore-calculating-language-statistics-for-a-directory-on-gith

Okay, it is now addressed by the last PR. Closing this issue.

See https://github.com/PCMDI/pcmdi_metrics/blob/main/.gitattributes


Migrate project to Kotlin

Description
Migrate all entire project to Kotlin 1.3.50, but keeping the integration concept. These are some of the things that have been done:

Migrate entire sample to kotlin
Make the library more modular by structure better the files / code
Add a project logo, make more easy to identify it (It is just a proposal 😄 , please share you point of view)
Add severals tools like ktlint, detekt, spotless for static analysis code adding them to Travis
Migrate all the tests to kotlin, for guaranty the coverage for new code
Modify readme specify the minim API version, and increase the version to 3.1.0 (please review this part if you consider to make major change)
Simplify logic about Rationale/Settings dialogs displaying an AlertDialog
Eradicate LowAPiPermissions for avoid throw exception to integrator on their code

Overview

Architecture:

Coverage:

Next Steps
Obviously with these changes, we only migrate the project to kotlin with little changes or simplifications. It's a good point to continue in this line and try to simplify the integration of the library and not only also adding more coverage tests, code documentation, etc..

@VMadalin wow!  This is a massive and very well thought out contribution.  First of all thank you for taking the time to do this.  In general it's good to file an issue before sending a PR like this so you don't have to do any unnecessary work though.
Without looking too deeply here are some things I like:

The logo is awesome!  Let's definitely use that.
The code reorganization is a good idea.  Although we shouldn't move any classes which were part of the publicly documented API before, organization isn't worth breaking people's code.
I like moving the sample to Kotlin, shows off more modern practices.

Here are some things that scare me though:

Moving to Kotlin is definitely a breaking change and this would be 4.0.0.  I am worried that doing this is mostly valuable to us and only causes churn for our developers.  Is there anything about the API that gets much nicer in Kotlin?  Should we consider just doing an easypermissions-ktx instead?
If we are going to do a breaking change, I would like to first fix as many of the open issues as we can in the Java version and release a 3.0.1 so that people can get a stable version of the library without adopting Kotlin.

cc @SUPERCILEX who always knows what the Android community is up to.  Would you say that it's "in fashion" for Java libraries to move to Kotlin or is that considered a hostile move?

@samtstern Thanks for your answer. I'm absolutely agree to focus effort trying to fix first the open issues, and upload a fix with these changes. Making more robust the current version. It's important also to migrate these fixes in the kotlin version.
Regarding the version, and repo. In my opinion it's not bad to think to create a new repository easypermissions-ktx or a new branch but the only problem what I see is to maintenance, both projects separately. Although while decide the best option maybe we should to create a specific branch called kotlin or similar and merge there all PR related and giving people the opportunity to contribute with next refactor version of the library

The logo is awesome! Let's definitely use that.
The code reorganization is a good idea. Although we shouldn't move any classes which were part of the publicly documented API before, organization isn't worth breaking people's code.
I like moving the sample to Kotlin, shows off more modern practices.

👆 I agree with all of this!
Some thoughts:

Moving to Kotlin shouldn't be a breaking change. In theory, we can make the APIs exactly the same as before to keep supporting our Java users.
The main benefits of Kotlin are NPE avoidance, faster development cycles, and more pleasant/modern language to use that makes life easier.

I would argue that we see none of those benefits. EasyPermissions is essentially a feature-complete library and thus in maintenance mode. That means we aren't actively developing it so a more enjoyable language that makes development faster is irrelevant. As for the NPEs we seen, they're mostly due to OEM bugs which means it would still be broken for Kotlin.

Kotlin presents a 1MB+ size increase for devs not using it previously. For something as fundamental as permissions, I feel like we'd want to minimize bloat and keep everybody happy (including Java people).
Major libraries that have switched to Kotlin are doing it because they can offer radically superior APIs (think Jetpack Compose, coroutines, operator + extension functions, property delegation, etc), but even that's sometimes unnecessary since you can usually just add extension functions in a companion lib to do the same thing (think KTX).
@samtstern There's also some confidential information I shared with you.

TL;DR: I would wholeheartedly support a KTX version of EasyPermissions, but I'm a little skeptical of having the whole library switched to Kotlin. Until 70% of the top 100K apps with updates in the last 6 months use Kotlin, I think libraries should generally stick to Java. (Or if JetPack releases a core library in Kotlin.)

@SUPERCILEX I'm absolutely agree with all points that you comment. But obviously depends a lot of project factors. For example at Userzoom, we decided to migrated the SDK to Kotlin recently but the distribution of it is in .jar file, for keeping the compatibility with different frameworks/platforms like react-native, phonegap, xamarin and flutter. The tendencies in a long time is to migrate all projects but this is a big task and require time, and for the moment the tendency is to have a hybrid project (specially for big projects).
@samtstern I consider that isn't a good practice to have tests in kotlin and the library in java that create a lot of confusion on development process, specially if you use TDD or BDD. Maybe the only part what we could migrate is the sample, keeping or not the both samples. The project structure is also valuable base to start to discuss how it would be better to organise the project, facilitating its maintenance.
However I see as good to pursue easypermissions-ktx to start discuss the new API thinking in simplify the integration, for example it's a good start point compare another notification permission like: https://github.com/permissions-dispatcher/PermissionsDispatcher with a very interest integration method.

Is there any progress in moving to Kotlin and publishing 4.X version?

@MadRatSRP no progress, I think the conversation above stands.  I'd be open to converting the sample app and (maybe) the tests to Kotlin.  I'd also be open to a KTX library.  Converting the core library to Kotlin has no user benefit so I don't think it's a good investment.

@MadRatSRP no progress, I think the conversation above stands. I'd be open to converting the sample app and (maybe) the tests to Kotlin. I'd also be open to a KTX library. Converting the core library to Kotlin has no additional user benefit so I don't think it's a good investment.

Actually, it has a lof of the additional user benefit. As you know, current Android development flows around Kotlin (including Kotlin Native) and Flutter so if somebody will convert the whole library into Kotlin, that will make it much more popular. And your code will help someone to learn something new about Kotlin and other things

@MadRatSRP that's true, but all of that can be accomplished via an easypermissions-ktx library, there's no need to convert the core library to Kotlin.

@VMadalin hey it looks like you're back! Are you still planning to pursue this PR?

@VMadalin hey it looks like you're back! Are you still planning to pursue this PR?

Hi @samtstern, I was involved on other things this year and I decided to dedicate some time to move forward the work I did. My intention was to continue this under https://github.com/VMadalin/easypermissions-ktx, of course, keeping google licence.
I'm not sure if is possible to add under googlesample/easypermissions-ktx keeping the consistency with the java repo?

@VMadalin that sounds like a good plan.  We can't add your repo under googlesamples (that's for Google-owned code only) but if you want I am happy to link to https://github.com/VMadalin/easypermissions-ktx from the README of this repo.

That sounds amazing @samtstern, I let you know when I'm done ;)

@VMadalin I'm going to close this PR since at this point we both agree we're not going to merge all of this.  When you're ready please send a new PR to add a link to the README, or just send me an email (samstern at google dot com) and I'll do it!


Lolipop plot with custom data instead of count

Describe the issue
Currently the lolipop plot's y-axis can only plot the count of the number of times a certain mutation occurs in the data. Is it possible to make this y-axis value customizable (e.g. variant allele frequency)?
This is a follow-up on a previous issue: #746
Also including an example I found from another package:

which was taken from this package (https://bioconductor.org/packages/release/bioc/vignettes/trackViewer/inst/doc/lollipopPlot.html#Change_the_height_of_a_lollipop_plot)

Hi,
Sorry for the late reply. I have added an argument data to lollipopPlot which should do the job.
Input data should be a two-column data frame with position and the corresponding counts. Optionally you can add a thrid column Variant_Classification which will be used for color coding the dots.
Minimal example:
my_data = data.frame(pos = sample.int(912, 15, replace = TRUE), count = sample.int(30, 15, replace = TRUE))
head(my_data)
  pos count
1 710    29
2 816    11
3 763    28
4 354    28
5 607    26
6 137    11
lollipopPlot(data = my_data, gene = "DNMT3A")

Hope this helps.


Fix passing 'server_insecure' to 'redhat_subscription'

The role parameter 'rhc_server.insecure' is a boolean parameter, whereas the configuration value for subscription-manager is a string with the values '0' (== secure) or '1' (== insecure). Thus, passing the boolean value of the role parameter as-is to 'redhat_subscription' is not correct.
Use a slightly more complex approach for it:

if 'rhc_server.insecure' is not passed or it was passed as "omit" value, then omit it
if 'rhc_server.insecure' is true, pass '1'
if 'rhc_server.insecure' is false, pass '0'

[citest]


README.md - Tutorials links not working

The links in the README.md to the tutorials are not working.
Instead of a tutorial a HTTP error code 404 is returned.
Examples of the not working links from the README.md:

Querying JPA
Querying SQL
Querying Mongodb
Querying Lucene
Querying Collections
Querydsl Spatial
Querying JDO

Oh. Okay.
@timowest was there anything lost in the previous outage?

I reinstated a link for latest. Turns out that was all that was needed.
Thanks for reporting!


iOS version

Suggestion
iOS has analogue of MediaStore that can hook into iTunes library
This could be theoretically implemented with this plugin
https://pub.dev/packages/playify

Good suggestion 👍.
I was thinking about this a lot already, because it's a problem we need to tackle for Desktop support as well.
We already have relatively good interface to the platform media store with the SweyerPlugin class. If we allow the app to use a different "plugin" (doesn't have to be an actual Flutter plugin) that wraps the playify package, iTunes support should be relatively easy to implement.
In the future we could then allow for more than one active plugin, then we could have other media stores like YouTube, Spotify and maybe even stuff like online radio. We can even have a plugin that exposes temporary paths like the one we get when the app is opened with a file.


Consider adding a deprecated + on collection.Set/Map

The fact that collection.Set and mutable.Set no longer have a + method can lead to very error confusing messages or even confusing compiling code. I think this could cause quite some headaches for users during the migration.
scala> Set(1) + 2
                ^
       error: type mismatch;
        found   : Int(2)
        required: String

User: It's clearly a set of Ints, why does it want a String?
scala> Set("a") + "b"
res14: String = HashSet(a)b

User: Going to file a bug, there's something wrong with HashSet.toString.
Same for Map (Map(1 -> "a") + (2 -> "b")).
One could argue we have similar issues with 2.12 collections, for example
scala> List(1) + 2
<console>:15: error: type mismatch;
 found   : Int(2)
 required: String
       List(1) + 2
                 ^

scala> List("a") + "b"
res10: String = List(a)b

But the difference is, IMO, that + is a common operation on (immutable) Sets and Maps, but not on other types. So it will happen more often that someone will try to call + on a collection.Set/Map.

Even if we didn't have any2stringadd, I'd be for adding + to improve compatibility.
We should also discourage the use of collection.X factories. Or is there a good reason to ever use collection.Set("hi") over collection.immutable.Set("hi")?


Md-select dropdown does not move when page is scrolled

Bug, feature request, or proposal:
When Md-select control is pressed, the md-select-panel opened up. It shows the list of possible ms-options. However, if the user scrolls the page, the rest of the elements on the page scroll but the md-select-panel retains its original position.
What is the expected behavior?
The relative position of md-select and md-select panel should remain the same.
What is the current behavior?
Relative position of md-select and md-select-panel does not remain the same. Md-select scrolls with the page but md-select-panel's position remains static.
What are the steps to reproduce?
Providing a Plunker (or similar) is the best way to get the team to see your issue.
Plunker template: http://plnkr.co/edit/o077B6uEiiIgkC0S06dd
What is the use-case or motivation for changing an existing behavior?
Which versions of Angular, Material, OS, browsers are affected?
2.0.0-beta.1, all os.s
Is there anything else we should know?

known missing feature https://github.com/angular/material2/issues/1430

Closing this as dupe of #1430.


Allow buttons to trigger scripts.

Currently, you can assign a button to open a URL, but no other actions are possible.
Example:  I'd like to assign infobutton to be used to trigger a script at the user's discretion:  For instance...
--infobuttonaction "sh /path/to/script"
Use case:
When using a Command File to initiate a captive process that the user cannot exit until certain conditions are met...
To prevent a condition where the primary script has somehow failed or broken, leaving the user stuck in a captive state, I'd like to create a "recheck the conditions" button that essentially opens a side process to double check the current condition of the device.
if [ $conditionsMet == true ]; then gracefully allow the user to exit the primary process that must be broken.  These actions would be managed using the secondary process (script) triggered by the info button.

You can program any actions to be taken as a result of button clicks by capturing the exit code of the dialog window and then using an if or case statement to take action dependent on that exit code.
There is a list of exit codes here:
https://github.com/swiftDialog/swiftDialog/wiki/Exit-codes
Here is a simplified example of this:
https://github.com/SecondSonConsulting/swiftDialogExamples/blob/main/captureButtonOutput.sh
This can be trickier if you've sent the SwiftDialog window to the background using & as you describe in your example, but still totally possible:
https://bigmacadmin.wordpress.com/2023/09/29/swiftdialog-get-the-exit-code-of-a-background-process/

Excellent write-up on this, BigMacAdmin.  I'll try to incorporate this into my workflow, and see if it will work for my use case.  It does seem promising.  I very much appreciate you sharing your knowledge for all of us.
I still think this is a valid feature request which would simplify this process significantly.

I understand that desire, and I do not speak for the author of swiftDialog, but I think the issue is more complex than you assume.
swiftDialog runs as the logged in user. Even if you're executing it from a script as root, the Dialog process itself gets launched as the logged in user. This is necessary for GUI apps to work properly on macOS.
If that user process then launches a script, it's a script thats running as the logged in user (not root) and it would be "owned" by SwiftDialog, meaning it would not have Privacy Protection Policy privileges (like Full Disk Access, or Documents/Downloads folder access, or app sandbox access.) Then we're in a place where SwiftDialog maybe requires a PPPC profile to work with particular projects/scripts?
What seems like a simple request is, I think, a big can of worms. SwiftDialog is meant to display messages. The logic of what you want to do based on user interaction is up to the admin.

Understood, and that makes a lot of sense.  I've actually got swiftDialog being launched via a Launch Daemon, which I had hoped would resolve any issues with running as the logged in user.  It seems I may be mistaken.
I'm working to get your (very slick) solution to work.  My trouble currently is that I specifically need swiftDialog to remain active (not exit) when the infobutton is pressed by the user.  Monitoring for an exit code breaks the process as I designed it (admittedly, with an incorrect understanding of how swiftDialog works -- thats on me).
Ultimately, I think I need to completely rethink how this works from the ground up.   Live and learn!


Error: Can't locate revision identified by 'e5ef6828ac4e'

When running the superset_init Docker image, which runs the command superset db update, the following error occurs:
ERROR [root] Error: Can't locate revision identified by 'e5ef6828ac4e'
superset_init exited with code 1

This may be preventing subsequent steps in the initialization process, such as creating the admin user.
Expected results
All database migrations would run without error.
Actual results
Some database migrations seem to be missing.
How to reproduce the bug

run docker-compose up > docker.log
open the docker.log file
look for lines related to superset_init
See error

Environment
(please complete the following information):

superset version: 0.37.1rc1

Checklist
Make sure these boxes are checked before submitting your issue - thank you!

[x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[x] I have reproduced the issue with at least the latest released version of superset.
[x] I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context
I found this error when troubleshooting issue #10711
As an open-source project, it is really important that end-users, and not just developers, can actually run the software.

I just realized this may be due to running migrations on the master branch before switching to the 0.37.1rc1 tag.


Draft: Sharpness-Aware Minimisation module

Summary
Draft for SAM optimisator https://arxiv.org/pdf/2010.01412v2.pdf

Sorry that this never had much of a public review at the time.  I'm going to close this as stale now, in part because it's mostly touching the opflow gradients, which are just on the cusp of becoming deprecated.  If you're still interested in doing something like this, then I'd point you towards the Qiskit Ecosystem first, as in to package this up as its own thing, and then look to move from Terra to there if appropriate.