FR: Add control to integration

It would be great if it would be possible to initiate a charge with a given charge rate via the integration.  As described here, I could then use HA to see when my solar production is above my home's use and send that excess power into the Rivian at a specified charge rate.
I aware that Rivian is currently limiting the vehicle/account to only two phones.  For me, having HA as one of those is acceptable, although I think Rivian needs to increase this limit.
Sample automation:
Set the charge rate to house_power - pv_power and start charging.  It's likely that rate would either need to be rounded to the nearest 1kw or nearest setpoint.

Hi @poldim
Thanks for the FR.  Right now however, we're not wanting to add anything that communicates to the truck to handle control.  There is a lot of security and safety concerns around that that we do not want to be liable for.
I am going to keep this open however, but it is not currently on our roadmap to handle.

Hey @bretterer I just found this issue. It seems like since this FR you added some ability to control the truck... but am I correct in understanding that it can only be done using Bluetooth from my HA machine? Meaning the truck would have to be at home, and in range of the server to work?
I get the hesitation to add control functionality via internet but I was hoping to use this integration to wake my truck when it is in the parking lot at work. It frequently takes me many, many attempts to wake the truck using the Rivian app and I often am unsuccessful in getting it to wake up and heat before it is time to leave. I was going to set a script to just keep trying until it wakes and then notify me when it does. Sounds like that won't be possible using this? Even JUST waking would be enough functionality, I wouldn't need any other controls since I can use the app once it is indeed awake.
Thank you for your time.

Hey @bretterer I just found this issue. It seems like since this FR you added some ability to control the truck... but am I correct in understanding that it can only be done using Bluetooth from my HA machine? Meaning the truck would have to be at home, and in range of the server to work?
I get the hesitation to add control functionality via internet but I was hoping to use this integration to wake my truck when it is in the parking lot at work. It frequently takes me many, many attempts to wake the truck using the Rivian app and I often am unsuccessful in getting it to wake up and heat before it is time to leave. I was going to set a script to just keep trying until it wakes and then notify me when it does. Sounds like that won't be possible using this? Even JUST waking would be enough functionality, I wouldn't need any other controls since I can use the app once it is indeed awake.
Thank you for your time.

The actual commands are run on the cloud (not Bluetooth), but to enable them, you have to fully pair your device with the vehicle which requires a one-time Bluetooth connection to set up. This is part of security mechanism that Rivian has implemented.

Hey @bretterer I just found this issue. It seems like since this FR you added some ability to control the truck... but am I correct in understanding that it can only be done using Bluetooth from my HA machine? Meaning the truck would have to be at home, and in range of the server to work?
I get the hesitation to add control functionality via internet but I was hoping to use this integration to wake my truck when it is in the parking lot at work. It frequently takes me many, many attempts to wake the truck using the Rivian app and I often am unsuccessful in getting it to wake up and heat before it is time to leave. I was going to set a script to just keep trying until it wakes and then notify me when it does. Sounds like that won't be possible using this? Even JUST waking would be enough functionality, I wouldn't need any other controls since I can use the app once it is indeed awake.
Thank you for your time.

The actual commands are run on the cloud (not Bluetooth), but to enable them, you have to fully pair your device with the vehicle which requires a one-time Bluetooth connection to set up. This is part of security mechanism that Rivian has implemented.

Oooh okay thanks a lot. I misunderstood the instructions! Appreciate it

Hey @bretterer I just found this issue. It seems like since this FR you added some ability to control the truck... but am I correct in understanding that it can only be done using Bluetooth from my HA machine? Meaning the truck would have to be at home, and in range of the server to work?
I get the hesitation to add control functionality via internet but I was hoping to use this integration to wake my truck when it is in the parking lot at work. It frequently takes me many, many attempts to wake the truck using the Rivian app and I often am unsuccessful in getting it to wake up and heat before it is time to leave. I was going to set a script to just keep trying until it wakes and then notify me when it does. Sounds like that won't be possible using this? Even JUST waking would be enough functionality, I wouldn't need any other controls since I can use the app once it is indeed awake.
Thank you for your time.

The actual commands are run on the cloud (not Bluetooth), but to enable them, you have to fully pair your device with the vehicle which requires a one-time Bluetooth connection to set up. This is part of security mechanism that Rivian has implemented.

Any tips for getting this to work? I could not get the vehicle to pair with HA, I even used a long active USB cable to take my HA machine's Bluetooth adapter all the way inside the truck. No matter what I do it fails to pair... I have BT set up correctly as I already use it to control several devices.

It was verified with HA on RPI, Macbook,  and ESPHome BT Proxy. We didn't
have a chance to fix a BT adapter on a virtual machine. I've just
instructed others to get an ESP32 and set up a BT proxy if they have
problems.
--
J.R. Gutierrez
On Thu, Dec 21, 2023 at 9:10 PM ksheyman @.***> wrote:

Hey @bretterer https://github.com/bretterer I just found this issue. It
seems like since this FR you added some ability to control the truck... but
am I correct in understanding that it can only be done using Bluetooth from
my HA machine? Meaning the truck would have to be at home, and in range of
the server to work?
I get the hesitation to add control functionality via internet but I was
hoping to use this integration to wake my truck when it is in the parking
lot at work. It frequently takes me many, many attempts to wake the truck
using the Rivian app and I often am unsuccessful in getting it to wake up
and heat before it is time to leave. I was going to set a script to just
keep trying until it wakes and then notify me when it does. Sounds like
that won't be possible using this? Even JUST waking would be enough
functionality, I wouldn't need any other controls since I can use the app
once it is indeed awake.
Thank you for your time.
The actual commands are run on the cloud (not Bluetooth), but to enable
them, you have to fully pair your device with the vehicle which requires a
one-time Bluetooth connection to set up. This is part of security mechanism
that Rivian has implemented.
Any tips for getting this to work? I could not get the vehicle to pair
with HA, I even used a long active USB cable to take my HA machine's
Bluetooth adapter all the way inside the truck. No matter what I do it
fails to pair... I have BT set up correctly as I already use it to control
several devices.
—
Reply to this email directly, view it on GitHub
https://github.com/bretterer/home-assistant-rivian/issues/105#issuecomment-1867179571,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACTMO6O3WA2ZM7MADSQBV3YKT237AVCNFSM6AAAAAAZCOH7ASVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRXGE3TSNJXGE
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

It was verified with HA on RPI, Macbook, and ESPHome BT Proxy. We didn't have a chance to fix a BT adapter on a virtual machine. I've just instructed others to get an ESP32 and set up a BT proxy if they have problems.
…
-- J.R. Gutierrez
On Thu, Dec 21, 2023 at 9:10 PM ksheyman @.> wrote: Hey @bretterer https://github.com/bretterer I just found this issue. It seems like since this FR you added some ability to control the truck... but am I correct in understanding that it can only be done using Bluetooth from my HA machine? Meaning the truck would have to be at home, and in range of the server to work? I get the hesitation to add control functionality via internet but I was hoping to use this integration to wake my truck when it is in the parking lot at work. It frequently takes me many, many attempts to wake the truck using the Rivian app and I often am unsuccessful in getting it to wake up and heat before it is time to leave. I was going to set a script to just keep trying until it wakes and then notify me when it does. Sounds like that won't be possible using this? Even JUST waking would be enough functionality, I wouldn't need any other controls since I can use the app once it is indeed awake. Thank you for your time. The actual commands are run on the cloud (not Bluetooth), but to enable them, you have to fully pair your device with the vehicle which requires a one-time Bluetooth connection to set up. This is part of security mechanism that Rivian has implemented. Any tips for getting this to work? I could not get the vehicle to pair with HA, I even used a long active USB cable to take my HA machine's Bluetooth adapter all the way inside the truck. No matter what I do it fails to pair... I have BT set up correctly as I already use it to control several devices. — Reply to this email directly, view it on GitHub <#105 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACTMO6O3WA2ZM7MADSQBV3YKT237AVCNFSM6AAAAAAZCOH7ASVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRXGE3TSNJXGE . You are receiving this because you are subscribed to this thread.Message ID: @.>

Alright thanks. I'll use the BT proxy option if I can't get it to work.

Also, it would be beneficial to enable debug logging to see where it gets in the process. Another thing of note, BT adapters have limited slots available for connections. If you have multiple devices already connecting via BT, it's possible they are taking up those slots and preventing a successful pairing. So it might be helpful to try disabling those other devices temporarily.

Also to note. The BT connection for HA is only required during the initial key exchange, after that all commands are sent over WiFi over the public Internet to the Rivian cloud servers and not the truck directly


HDD Temperature

Greetings and thank you for the dashboard!
I was wondering how are you retrieving HDD temperatures from FreeNAS? I am also using graphite as in your guide but don't see HDD temperatures in the database.
I assume you must be using a script?

Hi,
Very sorry about the late response.
I'm not using any custom scripts, I've just configured the remote graphite server in freenas.
Might be that your disks is not providing temperature readings? Do you see them in the freenas ui?

I don't see that information on my reporting tab in FreeNAS. Its either my disks not reporting or the fact that I am still on 11.1-U7?

I'm on 11.2 now, but I believe I had access to that info on previous versions as well.
Can you see the temp readings in the reporting section in your FreeNAS UI?

I cannot see it on the UI either. So is it up to the drive to report that information?

What kind of disks do you have?

I have four Seagate 2TB ST2000DM008.

Ok, I'm not familiar with those. I use WD in my setup.
Not sure, but might be that you have to enable S.M.A.R.T tests for your drives if that's not already been done.
If that doesn't help I would try to get some help in the FreeNAS forum


feat: Support custom targets

Progress:

[x] Add support for custom targets to target-spec (https://github.com/guppy-rs/guppy/pull/74)
[x] Looser triple validation (https://github.com/guppy-rs/guppy/commit/738a0c8cb995ddc32be01ffe9a32724f36bf280d)
[x] Release new target-spec version
[x] Release new target-spec-miette version
[x] Release new guppy version
[x] Update nextest-runner with support: https://github.com/nextest-rs/nextest/pull/891

[x] Add support for custom targets as --target, CARGO_BUILD_TARGET, build.target etc (in @sunshowers's local clone) -- includes both directly specified targets and RUST_TARGET_PATH.
[x] Add tests (mostly done, just need to add tests for RUST_TARGET_PATH).
[x] Add support for resolving custom targets in platform overrides (done in a different manner via https://github.com/guppy-rs/guppy/commit/738a0c8cb995ddc32be01ffe9a32724f36bf280d)

[x] Publish new nextest-runner version

Hmm, is nextest not handling custom targets correctly? In addition to the following warning, all dependencies are recompiled when doctest is built.
warning: could not determine target triple: error deserializing target triple from --target option
caused by: unknown platform triple
caused by: unknown triple string: target-specs/i686-unknown-linux-gnu.json
caused by: Unrecognized architecture: target

https://github.com/tokio-rs/tokio/actions/runs/5244982846/jobs/9472222567#step:7:1274
Although it probably wouldn't be difficult for nextest to add support for custom targets (probably basically just porting the logic around this to nextest-runner).

Originally posted by @taiki-e in https://github.com/tokio-rs/tokio/issues/5691#issuecomment-1587999784

Yes, nextest doesn't currently support custom targets. I'd have to add support for them to target-spec, because nextest does some evaluation of targets for per-test overrides. It's just some busy work, probably 1-2 days.

This is mostly done code-wise, just need to get new versions of various libraries uploaded.

I was close to releasing a version of nextest today, but realized that I wanted to do target validation somewhat differently. https://github.com/guppy-rs/guppy/commit/738a0c8cb995ddc32be01ffe9a32724f36bf280d has the new approach, and my hope is to get a new version of nextest out by this weekend.

(I think I'd like to spend some more time thinking about that change before fully committing to it.)

@NobodyXu finished this up, should be out as cargo-nextest 0.9.54 within the next 20-30 minutes.

Thank you @sunshowers !


Expand extensions list for dist folder icon

Changes proposed:

[x] Add
[ ] Prepare
[ ] Delete
[ ] Fix

Things I've done:

[x] My pull request fixes an issue, I referenced the issue.

This PR partially addresses one of the goals in #650 by adding an icon for lib folders. If this isn't acceptable it can be dropped.
Reason: lib and release are common patterns for distribution folders; as an example, Angular UI Router has used both: release, lib
Ambiguities: Some libraries use lib in different ways; for instance, as a source folder or a side-by-side resource with dist in the published library. However, the common case (as I understand it) is for lib to be some kind of resource meant to be consumed by an end user, mildly analogous to node_modules.

@JimiC I've updated the PR to remove the association, pending approval from @robertohuertasm.

I agree with @JimiC. Merging this wihout the lib association.


Update NOTICE

Issue : Expired Date
I changed the the Copyright from 2018 to 2021, hope that helps
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Thank you very much for the Merging of the branch


Not creating a new Encrypted DB? (LibVer 1.8.116)

Hello, I have been chasing my tail because I thought my problems were elsewhere, but it looks like my code is not even generating an encrypted database.
This is the function to create the DB, it executes in the path I expect, creating the DB with the Password, but the DB is never encrypted.
The code to create the database is this
    /// <summary>
    /// Create async db connection.
    /// </summary>
    /// <returns>
    /// A <see cref="Task{TResult}" /> representing the result of the asynchronous operation.
    /// </returns>
    public static SQLiteAsyncConnection CreateDBConnectionAsync(string dbName = "", string Password = null)
    {
        try
        {
            string databasePath = GetDBPath(dbName);

            if (AppSettingsFunctions.GetBoolSetting("settings_DBIsEncrypted")) //&& CoreGlobals.IsAuthenticated)
                Password = CoreGlobals.appPassword;

            SQLiteConnectionString connectionString;

            if (!CoreGlobals.bDBEncrypted)
                connectionString = new SQLiteConnectionString(databasePath,
                                                                true);
            else
                connectionString = new SQLiteConnectionString(databasePath,
                                                                true,
                                                                key: Password,
                                                                postKeyAction: c =>
                                                                {
                                                                    c.Execute("PRAGMA cipher_compatibility = 3");
                                                                });

            var connection = new SQLiteAsyncConnection(connectionString);

            return connection;
        }
        catch (SQLite.SQLiteException ex)
        {
            DebugTools.LogException(ex);
        }
        return null;
    }

public static bool CheckDBType(string dbPath)
{
bool dbEncrypted = false;
try
{
if (File.Exists(dbPath))
{
String s = "SQLite format 3";
byte[] b = new byte[s.Length];
int bytesRead = new FileStream(dbPath, FileMode.Open).Read(b, 0, s.Length);
if (bytesRead > 0)
{
if (bytesRead == s.Length && s.Equals(Encoding.UTF8.GetString(b)))
{
// Unencrypted
dbEncrypted = false;
}
else
{
// Encrypted
dbEncrypted = true;
}
}
}
}
catch (IOException e)
{
DebugTools.LogException(e);
}
return dbEncrypted;
}
Any ideas on why I am not getting an encrypted DB?
Cheers!

Hi @Devilcow
It would be best to confirm your application is properly loading SQLCipher at runtime. The best way to do that is to execute the following PRAGMA statement from your connection once opened:
PRAGMA cipher_version;

This should return a string value if your application is linking SQLCipher, otherwise a plaintext SQLite library will not return a value.

Thanks! That was it, The PCL project was using the sqlCipher lib, but the Android was building with the non cipher version.
One problem I am having now is that when the program is running, I can encrypt the database, and it gets flagged as encrypted (Checking the header of the file), but once I close then re-open the app, even though I enter the proper password, it still gives me the "File Not a Database".
I will browse the other posts, however, if you know of any starting points to check?


Potential bug in parmec changeset b723702

Hi @tkoziara
I think you may have introduced a bug in parmec changeset b723702 since the large multi-layer array models I have been able to simulate in changeset dd103c0 no longer produce results (nan values in output database).
Let me know if you need an example model. I haven't tested this with smaller models to see if it's a consistent issue across all the parmec models we've developed. Please let me know if you think there is a useful test case for me to run.
Many thanks,
Ben
(mention @stevebrasier)

Yes; thank you for reporting: I have looked into the change set and spotted a little bug; this also fixed a broken test for me so perhaps this is it; I am working on #10 now and I will commit all changes by the end of the day:)

Please try current commit:)

Hi @tkoziara,
This can now be closed thank you :)


issue under dropdown values under modules

[x] sales person cannot be created from sales modules
steps to reproduce-

click on create new sales order
click on sales person field and click on create sales person
enter details and click on submit button
submission is occuring but no sales person is created

[x] challan type is not displaying any dropdown values

issue fixed !


Fixes CVE-2023-48795

The crypto is pulled from go/compute which is pulled by go/storage
https://github.com/googleapis/google-cloud-go/tree/main/storage
Fixes CVE-2023-48795 - golang.org/x/crypto Authentication Bypass by Capture-replay

Not sure how important it is, but running go mod tidy would remove a few lines from go.sum.

Could you run a go mod tidy and commit those changes as well? @spolti

Not sure how important it is, but running go mod tidy would remove a few lines from go.sum.

It is, I always forget to run it, thanks for the reminder.

/lgtm

/assign @yuzisun

/approve


Support for S3 storage in BackupMariaDB and RestoreMariaDB

Be able to take and restore backups using AWS S3 as storage.

its will support only AWS or any storage with S3 API, like azure blob, monio and others ?

Hey @ox1depl !
The intention is supporting any S3 compatible API by levaraging this library:

https://github.com/minio/minio-go

Let me know if you have other suggestions.

At the moment, S3 backups can be achieved by installing the following CSI driver:

https://github.com/yandex-cloud/k8s-csi-s3

and specify its storageClassName in the Backup/Restore resources:
apiVersion: mariadb.mmontes.io/v1alpha1
kind: Backup
metadata:
  name: backup
spec:
  mariaDbRef:
    name: mariadb
  storage:
    persistentVolumeClaim:
      storageClassName: csi-s3
      resources:
        requests:
          storage: 100Mi
      accessModes:
        - ReadWriteOnce
  args:
    - --single-transaction
    - --all-databases

The S3 support is a priority for the next releases though


[BugFix]Fixed metric overriding issue

Signed-off-by: iamabhishek-dubey<EMAIL_ADDRESS>

@iamabhishek-dubey whats the status here with the open pull request?
I would suggest to rename "taskId" just to "id" as it's not always an taskId (for query/time its the queryId).
In general it would be very useful to have (as default) the shared common set of fields, and in addition an optional parameter to extend them.
Example: query/time
There is the "type" field which indicates the type of query. Very useful information which is lost at the moment.


[ASTGen] Use 'PURE' bridging mode to improve editor experience

'PURE' briding mode avoids importing things from swift:: or llvm:: namespaces. Since it significantly reduces the number of decls to import, editor experience is improved.

@swift-ci Please smoke test

@swift-ci Please smoke test Windows


Coverity 1520485: logically dead code

This is a false positive, but changing the condition so the code appears live seems safer than removing the check.
Fixes Coverity 1520485

I just wanted to merge this, but then noticed that the 2nd part of the commit message
    Coverity 1520485: logically dead code

    This is a false positive, but changing the condition so the code appears live
    seems safer than removing the check.

should be updated.

I'll update the message when merging.

Merged to master, 3.1, and 3.0 branches. Thank you.


"New File" templates

"New Jekyll Markdown" file should insert blank (?) YAML front matter
Other than 'blank markdown file', are there other templates that need to be considered?

Have implemented a simple version of this in the Issue3 branch.
Feel free to thumbs up/down it or run with your own approach.

Paul merged it while I fought with an XP PC


Compiler complains object is possibly undefined after assigning a value

TypeScript Version:  3.5.0-dev.20190420
Search Terms: Object is possibly 'undefined'
Code
interface SomeObject {
    prop: string
}

interface SomeComplexObject {
    prop: SomeObject
}

export class SomeClass {
    private prop: String;

    public constructor(someParam?: SomeObject, someOtherParam?: SomeComplexObject) {
        if (someParam === undefined && someOtherParam === undefined) {
            // <1>
            throw new Error("One of the params must be provided.");
        } else if (someOtherParam !== undefined) {
            // <2>
            someParam = someOtherParam.prop;
        }

        // If someParam === undefined, someOtherParam === undefined, it should goes to <1> and throw;
        // If someParam === undefined, it should goes to <2>, then someParam will be assigned with a non-undefined value, then goes to <3>;
        // If someParam !== undefined, it should goes to <3> directly and should not generate any error.
        // <3>
        this.prop = someParam.prop;
        //              ↑ Object is possibly 'undefined'.ts(2532)
    }
}

Expected behavior:
someParam won't be undefined and its props can be accessed
Actual behavior:
Compiler complains Object is possibly 'undefined'
Playground Link: Playground (Need to check strictNullCheck)
Related Issues: Not yet

Our flow control analysis isn't built to detect this kind of complementarity.
The equivalent form
        if (someParam === undefined) {
            if (someOtherParam === undefined) throw new Error("One of the params must be provided.");
            someParam = someOtherParam.prop;
        }

doesn't produce an error.

In fact, below snippet won't produce an error too:
interface SomeObject {
    prop: string
}

interface SomeComplexObject {
    prop: SomeObject
}

export class SomeClass {
    private prop: String;

    public constructor(someParam?: SomeObject, someOtherParam?: SomeComplexObject) {
        if (someParam !== undefined) {
            // <1>
            // No op
        } else if (someOtherParam !== undefined) {
            // <2>
            someParam = someOtherParam.prop;
        } else {
            throw new Error("One of the params must be provided.");
        }

        // If someParam === undefined, someOtherParam === undefined, it should goes to <1> and throw;
        // If someParam === undefined, it should goes to <2>, then someParam will be assigned with a non-undefined value, then goes to <3>;
        // If someParam !== undefined, it should goes to <3> directly and should not generate any error.
        // <3>
        this.prop = someParam.prop;
        // More statements using someParam...
    }
}

Since I have ~5-6 statements using someParam and I prefer not to put them in an if-else block, I will use this one to workaround.

why does the above check not prevent this error?

why does the above check not prevent this error?
ah workaround tx.btc_volume!

Guess the compiler is complaining about tx.btc_volume might be undefined, not tx.

@yhvicey yes you were right. If i put the bang in the right place it shuts up tx.btc_volume!

I guess this will be a TS difference from the new JS feature with tx?.btc_volume? to fail out and not have to keep checking nulls. like coffeescript had many years ago...

In your tsconfig.js
"strictNullChecks": false,
"strict": false,
https://github.com/Microsoft/TypeScript/issues/31060

look i have an angular code like this
this.titleService.setTitle(this.SinglePost[0]['title']['rendered']);
and its sowing me an error like
ERROR in src/app/blog/blog.component.ts(93,40): error TS2532: Object is possibly 'undefined'.
src/app/blog/blog.component.ts(93,56): error TS2339: Property '0' does not exist on type '[]'.
so i got a solution of it
var index =0;
      this.titleService.setTitle(this.SinglePost[index]['title']['rendered']);

and now its working fine

Also the issue exists when we have the following code:
function someChecking(value?: {}) {
   const rules  = { key1: "v1", key2: "v2"}
   if (!value) {
       return false;
    }

   // allright this is nested function
    function isRuleFailed(ruleKey: string): any {
       // I get 'possible undefined' here - but it's wrong because function fired after checking value
       return value[ruleKey];
    }

    return isRuleFailed('required') || Object.keys(value).find(isRuleFailed)

}

I have the same issue as @Yegorich555.
if (progress === undefined) return null;
// Object is possibly 'undefined'.  TS2532
if (progress.total === 0) return something;

This is not the thread to post every "I got an 'object is possibly undefined' error".
Note that narrowing doesn't occur in unreachable code; if your code cannot be reached by any execution path, then you may see errors as a result.
Please post self-contained bugs if you feel you've encountered an error you shouldn't have, or use Stack Overflow to help understand why you're seeing a particular error. Thanks!


Revert 'awaited' type

The new awaited type is currently causing too many breaks to be considered stable for TS 3.9. This PR reverts the awaited type for this release while we continue to investigate the impact of the change.
Closes #37511
Closes #37512
Closes #37526
Closes #37528
Closes #37534
Closes #37551
Closes #37556
Closes #37564
Closes #37566
Closes #37567
Closes #37572
Closes #37592
Closes #37602
Closes #37603

@typescript-bot perf test
@typescript-bot run dt
@typescript-bot test this
@typescript-bot user test this

I found a few more stragglers, will update shortly.

@typescript-bot run dt
@typescript-bot test this
@typescript-bot user test this

@typescript-bot run dt
@typescript-bot test this
@typescript-bot user test this

RWC changes look good
user/docker changes look good except for some errors in vscode that look to be due to https://github.com/microsoft/TypeScript/pull/36539, not this PR.
DT changes look good

@typescript-bot cherry-pick this to release-3.9

oops... I was relying on this solution because I have a Promise<MaybeList<Promise<ISymbolSearchResult[]>>>
where MaybeList is an array abstraction so I am using promise.all to remove the inner Promise I don't know what to do now to avoid using as unknown as MaybeList<ISymbolSearchResult[]> inside of my async function (which should be returning the Promise<MaybeList<ISymbolSearchResult[]>> instead of the Promise<MaybeList<Promise<ISymbolSearchResult[]>>>)


Failed to load resource due to Error: token cannot be null

I'm trying to use the Stream Analytics VS Code plugin to create Stream Analytics code as advised in the portal.
Unfortunately, I get this error: [Error] Failed to load resource due to Error: token cannot be null..
I have tried different tenants and different Stream Analytics Jobs, but they all give the same error.
In VS Code it looks like this:

Hi @Agazoth , which version of Azure Account extension are you using? and could you authentication library are you using in the Azure Account settings? e.g.,

MSAL

Thanks! Switching to ADAL did the trick :-)
Are there any plans for switching to MSAL

Hi @Agazoth , we plan to support MSAL in Oct. release. Thank you for raising this issue!

Hi @Agazoth, we have supported MSAL in 1.5.0, could you please have a try? Thanks!


An error ocurred while starting the kernel

Issue Report Checklist

[x] Searched the issues page for similar reports
[ ] Read the relevant sections of the Spyder Troubleshooting Guide and followed its advice
[ ] Reproduced the issue after updating with conda update spyder (or pip, if not using Anaconda)
[ ] Could not reproduce inside jupyter qtconsole (if console-related)
[x] Tried basic troubleshooting (if a bug/error)

[ ] Restarted Spyder
[ ] Reset preferences with spyder --reset
[x] Reinstalled the latest version of Anaconda
[ ] Tried the other applicable steps from the Troubleshooting Guide

[ ] Completed the Problem Description, Steps to Reproduce and Version sections below

Problem Description
I am not able to perform coding in Spyder because of the error coming in the IPython console that says -"An error ocurred while starting the kernel
What steps reproduce the problem?

Restarting the kernel

What is the expected output? What do you see instead?
Paste Traceback/Error Below (if applicable)

Traceback (most recent call last):
File "C:\Users\ANGEL\Anaconda3\lib\runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "C:\Users\ANGEL\Anaconda3\lib\runpy.py", line 85, in run_code
exec(code, run_globals)
File "C:\Users\ANGEL\Anaconda3\lib\site‑packages\spyder_kernels\console_main.py", line 11, in
start.main()
File "C:\Users\ANGEL\Anaconda3\lib\site‑packages\spyder_kernels\console\start.py", line 279, in main
import_spydercustomize()
File "C:\Users\ANGEL\Anaconda3\lib\site‑packages\spyder_kernels\console\start.py", line 39, in import_spydercustomize
import spydercustomize
File "C:\Users\ANGEL\Anaconda3\lib\site‑packages\spyder_kernels\customize\spydercustomize.py", line 19, in
import pdb
File "C:\Users\ANGEL\Anaconda3\lib\pdb.py", line 73, in
import cmd
File "C:\Users\ANGEL\Anaconda3\lib\cmd.py", line 45, in
import string, sys
File "C:\Users\ANGEL\string.py", line 37, in
pancard_number[2]="A" #This line will result in an error, i.e., string is immutable
TypeError: 'str' object does not support item assignment

PASTE TRACEBACK HERE

Versions

Spyder version:
Python version: Python 3.7
Qt version:
PyQt version:
Operating System name/version:

Dependencies

PASTE DEPENDENCIES HERE

IPython >=4.0     :  7.4.0 (OK)
cython >=0.21     :  0.29.6 (OK)
jedi >=0.9.0      :  0.13.3 (OK)
matplotlib >=2.0.0:  3.0.3 (OK)
nbconvert >=4.0   :  5.4.1 (OK)
numpy >=1.7       :  1.16.2 (OK)
pandas >=0.13.1   :  0.24.2 (OK)
pycodestyle >=2.3 :  2.5.0 (OK)
pyflakes >=0.6.0  :  2.1.1 (OK)
pygments >=2.0    :  2.3.1 (OK)
pylint >=0.25     :  2.3.1 (OK)
qtconsole >=4.2.0 :  4.4.3 (OK)
rope >=0.9.4      :  0.12.0 (OK)
sphinx >=0.6.6    :  1.8.5 (OK)
sympy >=0.7.3     :  1.3 (OK)

C:\Users\ANGEL\string.py

You need to remove or rename this file for this to work correctly. The thing is that file has the same name as an internal Python module, and that's what breaks things internally.


ExternalInterface not working with multiple HTML5 embeds

I use lime.embed to load two instances of an app in the same page. The problem is that if the second instance calls ExternalInterface.addCallback, then the callback is added to the root element of the first instance.
The attached project (test.tar.gz) reproduces the issue.

When calling div1.sayHi the second instance's handler actually gets called (cause it has overwritten div1.sayHi).
When calling div2.sayHi we get an error cause there's no handler installer on div2

(index):61 calling sayHi on test1
Test.js:3131 Main.hx:25: hi from,test2

(index):64 calling sayHi on test2
(index):65 Uncaught TypeError: document.getElementById(...).sayHi is not a function
    at callExternalInterface ((index):65)
    at HTMLInputElement.onclick ((index):54)

This is a problem with static method calls. In Javascript, all common code is shared, so two calls to the same static method will execute the same code.
There are no namespaces, like in ActionScript. I wonder if there is some way we can derive which embed instance is calling the static method

Perhaps we should make lime.embed include the JavaScript file at that point. That will mean a delay before the script begins to run, but that would allow the scripts to remain encapsulated, and keep static methods (like this one) isolated so that they don't interact improperly

The standard way for creating namespaces in javascript is to wrap code in a wrapper function, so that globals become local variables, and export whatever you need. Typically the wrapper is an anonymous function that is called immediately once the file is loaded, creating a single version of each exported class, like this:
<script> // SomeExternalFile.js
(function($export) {
    var A = $export.A = function() {
        console.log("num: "+A.num++);
    }
    A.num = 1; // static
})(window);
</script>
<script>
    new A();    // num: 1
    new A();    // num: 2
</script>

If we want to have multiple isolated versions of each class, we can wrap them in a namespace function which is called later, whenever we want to create a new namespace, like this:
<script> // SomeExternalFile.js
(function($export) {
    // top-level, only export a wrapper "namespace" function
    $export.my_namespace = function($export) {
        var A = $export.A = function() {
            console.log("num: "+A.num++);
        }
        A.num = 1; // static
    }
})(window);
</script>
<script>
var ns1 = {}; my_namespace(ns1);
var ns2 = {}; my_namespace(ns2);
new ns1.A();    // num: 1
new ns2.A();    // num: 1 (separate static var)
</script>

I hacked a proof of concept showing the use of separate namespaces for each lime embed. I just wrapped the whole .js file produced by openfl build in a namespace function, exporting a tiny lime.embed which creates a new namespace ns and delegates the call to ns.lime.embed.
(function (console, $hx_exports, $global) { "use strict";

$hx_exports.lime = $hx_exports.lime || {};
$hx_exports.lime.embed = function() {
    var ns = {};                            // classes exported by limeNS go here
    limeNS(console, ns, $global);           // create the "namespace"
    ns.lime.embed.apply(ns.lime, arguments);// delegate the call to ns.lime.embed
};
var limeNS =
//// start of original js
(function (console, $hx_exports, $global) { "use strict";
....
});
//// end of original js (with only the last line modified to avoid calling the wrapper function)

})(typeof console != "undefined" ? console : {log:function(){}}, typeof window != "undefined" ? window : exports, typeof window != "undefined" ? window : typeof global != "undefined" ? global : typeof self != "undefined" ? self : this);

The proof of concept is attached (test.tar.gz), the modified code is in the html5-modified dir. Index.html creates two embeds that are perfectly isolated, each getting separate static variables. This also makes ExternalInterface work like a charm.
I guess it wouldn't be hard to create a proper solution along those lines, having a global exported lime (shared by all scripts loaded in the same page) and make __registerEntryPoint store a wrapper namespace function, called by lime.embed, or something like that.

Just to add that stage.loaderInfo is also broken for multiple embeds cause get_loaderInfo() uses the static openfl.Lib.current. And I'm sure there will be other similar cases, it's not a minor issue.

Hej guys,
Is there any solution for that ?
I'm not able to embed many instances of the same app in a single HTML page : http://community.openfl.org/t/multiple-instances-of-html-target/8559/16?u=michal
Regards,

I just added support for multiple namespaces -- one for each embed, based on @chatziko's code :grin: (thank you!)
Each lime.embed call uses it's own $hx_exports context. This resolves static variable conflicts between multiple instances of the same or different applications being embedded.
However, in order to help with @:expose methods, we are copying fields from the $hx_exports context when they do not exist in the broader context yet.
Let's say I have two projects, "TestProject1" and "TestProject2". Let's pretend both projects expose a method called "testMethod":
var exports = lime.embed ("TestProject1", "openfl-content1", 200, 200, { parameters: {} });
var exports2 = lime.embed ("TestProject2", "openfl-content2", 200, 200, { parameters: {} });
exports.testMethod (); // executes from TestProject1
exports2.testMethod (); // executes from TestProject2
testMethod (); // executes from TestProject1, where it was defined first

This also works for repeated embeds of the same project on the page. Exposed methods would prefer the context of the first time it was embedded
Please give me feedback if you think this can be improved, but I'm already VERY happy to see statics properly being isolated :smile:


OSRM json format

The mapbox SDK seems to be frequently used and is open source (as far as I can see for several languages) and so we should consider implementing an alternative json format of /route via a different end point like /route-osrm or parameter like format=osrm
http://project-osrm.org/docs/v5.5.1/api/
Dependency: #1108

Would suggest a new endpoint, which then emulates the OSRM API both ways.

Ok. What do you mean with "both ways"?

That the endpoint's requests are also like OSRMs. Not only the responses, like you were hinting at with format=osrm.

Ah, makes sense. Now the endpoint makes also more sense, yes.

When we merge #1422, should we create smaller issues per feature that we want to add to this endpoint and close this issue?

Yes, sure.

This will go into a separate repository https://github.com/graphhopper/graphhopper-navigation
And we won't support OSRM but only mapbox (they seem to be incompatible and OSRM seems not to work with the navigation SDK)


feat: process config in azion/config

Generates a manifest file for the Azion Edge Application.
Example Process Config
import { processConfig } from 'azion';

const config = {...};

const manifest = processConfig(config);

console.log(manifest);

import { AzionConfig, processConfig } from 'azion';

const config: AzionConfig = {...};

const manifest = processConfig(config);

console.log(manifest);

:tada: This PR is included in version 1.7.0-stage.12 :tada:
The release is available on:

npm package (@stage dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 1.7.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Wrong Library and DLL Reference in MFCreateAVIMediaSink

Describe the bug
The Library and DLL Reference are wrong in MFCreateAVIMediaSink:
The symbol is actually located in mfsrcsnk.dll but not mf.dll

@GrantMeStrength Can you please take a look at this issue?

Thanks for reporting this issue. I've verified that the API is in mfsrcsnk.dll and I've updated the docs. You should see the update on the live site within 24 hours.


[BUG] com.azure.cosmos.spark infer schema doesn't work correctly

Hi everybody.
There is bug in driver com.azure.cosmos.spark.
When i extract data from Azure Cosmos DB with Azure Databricks and enable inferring of schema, field that contains array in cosmos db displayed as empty.
Could you please look into this and make it work as it was in previous version of driver.
Thanks in advance.
Regards,
Daniil.

Hi @em-daniil-terentyev, someone from the cosmos team will follow up with you shortly.

@moderakh  - please route this appropriately.

@em-daniil-terentyev Could you please provide a sample repro:

document content
spark code
spark connector version

Might be solved by https://github.com/Azure/azure-sdk-for-java/pull/21933, this added collision resolution also for Array types (which was present in Spark 2 Connector ). If the Arrays had different element Type or if the property was an Array in one document and null in the other.

In com.azure.cosmos.spark:azure-cosmos-spark_3-1_2-12:4.2.0 it's fixed


No such file or directory: '0.traj'

When i am running merger.py  ,some error like this: FileNotFoundError: [Errno 2] No such file or directory: '0.traj'
I have readed your other Issues ,can you show some example  of 0.traj file ,please .
Thank you very mach

Hi htthYjh,
First of all, I strongly suggest looking at the demo file first. The demo file shows the fundamental steps of trajectory clustering. Hopefully, it should be enough. Or, you can take a look at the report as well.
I was not very experienced with trajectory clustering and python in general when I was doing this project. So, the original code is quite messy, undocumented, and hard to follow. I don't remember the format of the .traj files exactly. However, from other modules, I infer that it is a dictionary of a list of tuples. I give an example:
{
000001: {'path':[(36, 42, 1, 5), (36.1, 42, 1, 5), (36.2, 42, 1, 5)]}
000002: {'path':[(36, 42, 1, 5), (36, 42.1, 1, 5), (36, 42.2, 1, 5)]}
}

each path has the following format [(latitude_1, longitude_1, speed_1, azimuth_1), (latitude_2, longitude_2, speed_2, azimuth_2), ...]. I don't quite remember the units of the fields.
I hope, this answer solves your issue. I'm planning to write a blog post by rewriting some parts of the blog post and giving more detail for different situations since the original code creates a considerable amount of confusion.


Cannot choose merge requests if there are multiple merge requests with the same source branches

Prerequsities

[x] I'm on the latest version of the plugin
[x] I've installed the required dependencies
[x] I've run :h gitlab.nvim.troubleshooting and followed the steps there

Setup Configuration and Environment
{
    debug = { go_request = false, go_response = false },
    popup = {
      exit = "gq",
      backup_register = "+", -- NOTE: Default backup to system register, may hinder performance in WSL
    }
  }

Bug Description
Cannot choose merge requests if there are multiple merge requests with the same source branches.
When executing :lua require('gitlab').review, gitlab.nvim will show the following error:
gitlab.nvim: Multiple MRs found: please call gitlab.choose_merge_request()

And executing :lua require('gitlab').choose_merge_request() and choose the MR, it still show the above error.
It occurs when the source branch of a merge request is deleted upon merging, and another MR is created with a newly created source branch that has the same name as the original.
I think the solution to this is to check if the merge request is merged or not and only check if the open merge requests is more than one in https://github.com/harrisoncramer/gitlab.nvim/blob/38bde8a0e4a982329ab4dc4f214f3f06dfc4e4fd/cmd/app/middleware.go#L123
Reproduction Steps

Create a MR with branch example.
Merge with the option "delete source branch" checked.
Create another MR with branch example.
Checkout example branch.
Open Neovim
Execute :lua require('gitlab').review() or :lua require('gitlab').choose_merge_request() and select example branch.

Screenshots

Hey thanks for the bug report, I was able to replicate the problem here.
Just so I understand correctly, you are creating an MR with a branch name (say some-branch) then merging and deleting that branch on the remote, and then re-creating another MR with the same branch name?
This is a pretty strange workflow! However, I think the fix here is to pass along the internal Id (IID) of the merge request to the Go server, rather than relying on the source and target branch, when a selection is made.
Please let me know if this solves your issue: 388-cannot-choose-merge-requests-if-multiple

Thank you! I'll try if the new commit fix the bug and update here.


the meta url for pull request is the api URL not the URL to the github pull request

fix this!

The API attribute is html_url not url.
This was fixed on this commit.


The pirateweather integration is using deprecated forecast (2023.9.0b0)

HA will remove the forecast attribute, you will have to move to a new method to include the forecast
see https://developers.home-assistant.io/blog/2023/08/07/weather_entity_forecast_types/

Thanks for the heads up here! I've had my eye on the changes to the source, so great to see everything documented. The current approach leaves much to be desired, and so I'm really happy with these new updates

Any update here now that this is GA?

@Intecpsp it's only a warning for now, and won't be a real problem until March 2024 (if I understand correctly).  I assume @alexander0042 will prioritize/schedule this issue to be fixed in a new release some time before then.
Attaching a screenshot of the message I got about this on my HA instance.

Apologies for the repeat, but I mentioned this on #137 but it doesn't seem like that's been noticed, so adding it here.
You might be interested to know that AstroWeather recently updated their integration to support the new service mode.

Just a request, please leave the forecasts in until 2024.3 like the core integrations do as well.
Another custom weather integration removed it already, and I still rely on the forecast attribute for a template weather entity.

Any update on this?  Been a few months at this point since the change went live.

Apologies for the radio silence, but I've got good news here! I'm going to post a longer update as a project, but in short, I have a immediate solution to this specific issue that should address this. It keeps 95% of the same code and structure, with a little translation layer to fit it into the new Home Assistant format. I'm going to leave it running on my machine for a couple days to make sure it's working, but then I'll push it out as an update!
In the process of fixing this, I ended up spending a fair bit of time thinking about the underlying structure, and I think that it makes the most sense to restructure the integration using all the new tools. Going to push this as a branch, so will be available for discussion but won't come out as an update. This should fix the occasional update issue, remove the legacy Dark Sky dependency, and allow for it to eventually make it into the default. However, it will likely break compatibility with the existing integration, hence the fix to this one first!

Will you include the forecast attribute until 2024.3 or will you immediately remove it?

Apologies for the radio silence, but I've got good news here! I'm going to post a longer update as a project, but in short, I have a immediate solution to this specific issue that should address this. It keeps 95% of the same code and structure, with a little translation layer to fit it into the new Home Assistant format. I'm going to leave it running on my machine for a couple days to make sure it's working, but then I'll push it out as an update!
In the process of fixing this, I ended up spending a fair bit of time thinking about the underlying structure, and I think that it makes the most sense to restructure the integration using all the new tools. Going to push this as a branch, so will be available for discussion but won't come out as an update. This should fix the occasional update issue, remove the legacy Dark Sky dependency, and allow for it to eventually make it into the default. However, it will likely break compatibility with the existing integration, hence the fix to this one first!

You could always publish both the immediate and long term fixes as branches on the project and release beta versions for people to try as you can download beta versions in HACS. Could always be helpful for the immediate fix to test on other setups besides your own.

@TheFes, the plan is to keep this repo around so long as there's no major incompatibility within Home Assistant that makes it impossible. The key reason to transition to a new codebase to get added to the built in HA install is because of the YAML support. I want to keep this one compatible with legacy yaml, but that really isn't possible in order to be added as a new HA integration.
@cloneofghosts, love this plan, and is exactly what I'm planning on doing. You'll see a "beta" branch today that is the existing integration, along with a (broken) "future" branch that I'll use to put together a fully HA compliant version going forward.

OK, and pre-release is live! Looks like anyone using HACS should see it if they enabled "show pre-release" when it was installed, otherwise you need to remove and re-add the Pirate Weather repo in HACS

OK, and pre-release is live! Looks like anyone using HACS should see it if they enabled "show pre-release" when it was installed, otherwise you need to remove and re-add the Pirate Weather repo in HACS

Great.  I'll check it out shortly.  I integrate Pirate Weather forecast attribute data into a lot of my stuff, including this LED Matrix information panel, so I'll let you know how things integrate with the update.

Outside of the time sensor issue everything has been running smoothly on my end. Just wanted to note that if you're using a custom weather card it might not work correctly with the update. They'll need to update the card on their end to support the forecast attribute.
Maybe it's something that @alexander0042 could mention in the release notes or the repositories readme file.

Now that I've tested the version control (which would have been bad), I agree we're ready to push to main, and would appreciate if you could do that! I just pushed a new commit reverting my testing changes, so I think you'll have to make a new tag and mark that one as the official release

Version released. Thank you to everyone who tested the pre-release version.


db hint

you should not use a transaction when executing just one query.
transactions are blocking operations, and thus affect performance and scalability, should only be used where necessary ;)

Thanks for the hint. I've just remove redundant transactions.


Moving timeline playhead

First of all, thank you very much for this repo, I am enjoying it a lot.
I wanted to ask, is there a way to move the playhead? If I add to the timeline a list of MediaPoolItems, they get added one after another; I would like to add a video and an audio file "at the same time". Is that possible? If not, is there a way to set the time location of a MediaPoolItem?

I have done it:
timeline.settings._obj.GetCurrentTimecode() to get the current playhead position
timeline.settings._obj.SetCurrentTimecode('01:00:20:00') to move it to the specified position

My bad, by using mediapool.append_to_timeline() the new clips get added at the end nonetheless. }
Is there a way to modify the time location of a MediaPoolItem?

Hey @MarcoP91! Pydavinci doesn't fully support all of Resolve's API yet, but unfortunately I don't think even Resolve's API supports inserting clips into timeline yet...  I haven't got Resolve handy atm, but you can check the scripting README in your copy of Resolve to see what's supported.

Ok, thank you 👍
I created a workaround with osascript, I'll leave it here:
`query = f"""
osascript -e '
tell application "Davinci Resolve"
activate
delay 0.1
tell application "System Events"
key code 123
delay 0.1
key code 123
delay 0.1
keystroke "y"
delay 0.1
keystroke "d" using command down
delay 0.1
keystroke "{seconds}"
delay 0.1
keystroke return
end tell
end tell
'
"""
os.system(query)`

Apparently AppendToTimeline() has some new parameters in DR 18 that allows for this. See this forum thread for details:

Apparently AppendToTimeline() has some new parameters in DR 18 that allows for this. See this forum thread for details: https://forum.blackmagicdesign.com/viewtopic.php?f=21&t=113040

cool!

Fixed in by #28
Closed


Cherry Pick PR to handles NPE when wrong UUID is specified in the vsphere.conf file

We should cherry pick this PR - https://github.com/kubernetes/kubernetes/pull/44661 to release 1.5 and release 1.6 to handle NPE.
Recently a customer reported this issue, and we required to go through stack trace to root cause the problem.
with this PR, the correct error message is printed, instead of NPE stack trace.
cc: @tusharnt @BaluDontu @luomiao

Rohit will look into this with the highest priority.

release-1.5: https://github.com/kubernetes/kubernetes/pull/49310
release-1.6: https://github.com/kubernetes/kubernetes/pull/49311

Closing this issue. both PR are approved on kubernetes.


Binary attributes

Binary attributes are supported by Joi, but not by jsonapi.
Example resource:
var jsonApi = require("jsonapi-server");

jsonApi.define({
    resource: "img",
    handlers: new jsonApi.MemoryHandler(),
    attributes: {
        bin: jsonApi.Joi.binary(),
        mimetype: jsonApi.Joi.string(),
        width: jsonApi.Joi.number().min(1).max(10000).precision(0),
        height: jsonApi.Joi.number().min(1).max(10000).precision(0),
    }
});

Results in the following error when using npm start:
node_modules/jsonapi-server/lib/graphQl/joiConverter.js:38
    throw new Error('Unable to parse Joi type, got ' + JSON.stringify(joiScheme))
    ^

Error: Unable to parse Joi type, got {"isJoi":true,"_type":"binary","_settings":null,"_valids":{"_set":[]},"_invalids":{"_set":[]},"_tests":[],"_refs":[],"_flags":{},"_description":null,"_unit":null,"_notes":[],"_tags":[],"_examples":[],"_meta":[],"_inner":{}}

What changes to the code are needed to make this work?

JSON and GraphQL don't support the binary type, so serialisation support for binary values would have to be implemented. We should have a better error during validation for unsupported types, like binary, though.

Nah, I'm good now. I changed my implementation to a URL string which points to a remote file.

Thanks for the update @Benny-. Closing this.


MSID: 91716 Version: 1 DOI: 2<IP_ADDRESS>2140

"msas": "Neuroscience"
"msid": "91716"
"preprintDoi": "10.1101/2<IP_ADDRESS>2140"
"Reviewed Preprint posted": "2023-11-01"
"Sent for peer review": "2023-08-21"
"Posted to bioRxiv": "2023-08-07" (link: "Go to bioRxiv": "https://www.biorxiv.org/content/10.1101/2<IP_ADDRESS>2140v1/")
[PLACE PDF URL HERE WHEN AVAILABLE] See step 7
Step 1. Inform bioRxiv
Who can help: @QueenKraken, @nlisgo, @scottaubrey

[x] Manuscript is in data hub index (https://data-hub-api.elifesciences.org/enhanced-preprints/docmaps/v2/by-publisher/elife/get-by-manuscript-id?manuscript_id=91716)

Step 2. Create preview of manuscript
Who can help: @fred-atherden, @nlisgo, @scottaubrey

[ ] Manuscript is available for preview (https://prod--epp.elifesciences.org/preview/91716/v1)

[ ] Trigger import for staging: https://prod--epp.elifesciences.org/import
[ ] Sync data folder with s3://prod-elife-epp-data/data
[ ] Trigger import for staging: https://staging--epp.elifesciences.org/import
[ ] Sync data folder with s3://staging-elife-epp-data/data
[ ] Pull request created to enhanced-preprint-data
[ ] Post link to pull request below

Pull request: [PLACE LINK TO PULL REQUEST HERE]
Detailed instructions: https://github.com/elifesciences/enhanced-preprints-data#add-a-manuscript
Step 3: Awaiting public reviews and QC
Who can help: Production team

[ ] QC passed or QC not needed

Who can help: Editorial team

[x] Public reviews available on sciety (https://sciety.org/articles/activity/10.1101/2<IP_ADDRESS>2140)
[x] Manuscript and editor details at the top of this issue (Supplied by Editorial team)
[ ] Reviews returned by EPP - https://prod--epp.elifesciences.org/api/reviewed-preprints/91716/v1/reviews

Example
"msas": "Genetics and Genomics", "Neuroscience"
"msid": "84628"
"version": "1"
"preprintDoi": "10.1101/2<IP_ADDRESS>4241"
"articleType": "Reviewed Preprint"
"status": "Published from the original preprint after peer review and assessment by eLife."

"Reviewed Preprint posted": "2023-01-02"
"Sent for peer review": "2022-10-28"
"Posted to bioRxiv": "2022-11-21" (link: "Go to bioRxiv": "https://www.biorxiv.org/content/10.1101/2<IP_ADDRESS>4241v1")

Editors:
Reviewing Editor
Michael B Eisen
University of California, Berkeley, United States
Senior Editor
Michael B Eisen
University of California, Berkeley, United States

Step 4: Modify manuscripts.json (no PDF)

[ ] Reviewed preprint is published to EPP (https://elifesciences.org/reviewed-preprints/91716)

[ ] Pull request created to enhanced-preprint-client
[ ] Pull request switched to "Ready for review"
[ ] Post link to pull request below

[ ] Request that a doi be registered

[ ] Post link to #enhanced-preprint comment below

Pull request: [PLACE LINK TO PULL REQUEST HERE]
#enhanced-preprint comment thread: [PLACE LINK TO COMMENT HERE]

Instructions to modify manuscripts.json

Visit: https://github.com/elifesciences/enhanced-preprints-client/actions/workflows/publish-manuscript.yaml
Click: Run workflow
Complete the form and click "Run workflow"
A successful run should result in a new pull request at https://github.com/elifesciences/enhanced-preprints-client/pulls
Open the pull request and click the "Ready for review" button to trigger tests
Once the tests pass and you are happy with the changes the PR can be merged

Example pull request: https://github.com/elifesciences/enhanced-preprints-client/pull/334/files
Once the pull request is merged in it should be available a few minutes later.

Request that a doi
Post the following in #enhanced-preprint:
@Fred can you register a doi for https://elifesciences.org/reviewed-preprints/91716

Step 5: Awaiting search reindex

[ ] Reviewed preprint is avaliable on journal homepage (https://elifesciences.org)

The search reindex is triggered once an hour. We need the reviewed preprint to be indexed as the search application serves the journal homepage.

Additional info
If needed, the jenkins pipeline to reindex search can be triggered sooner.
https://alfred.elifesciences.org/job/process/job/process-reindex-reviewed-preprints/

Step 6: Published! Request PDF generation

[ ] Post the link to the PDF url at the top of the issue

[ ] Post link to #enhanced-preprint comment below

#enhanced-preprint comment thread: [PLACE LINK TO COMMENT HERE]
Post the following to the #enhanced-preprint on slack:
@Ryan Dix-Peek please can you generate a PDF for https://elifesciences.org/reviewed-preprints/91716

Step 7: Introduce PDF to data folder and git repo

[ ] PDF is avaliable at https://github.com/elifesciences/enhanced-preprints-data/raw/master/data/91716/v1/91716-v1.pdf

Detailed instructions: https://github.com/elifesciences/enhanced-preprints-data#add-a-pdf
Step 8: Add PDF url to manuscripts.json

[ ] Reviewed preprint PDF is available for download (https://prod--epp.elifesciences.org/reviewed-preprints/91716)

[ ] Pull request created to enhanced-preprint-client
[ ] Pull request switched to "Ready for review"
[ ] Post link to pull request below

[PLACE LINK TO PULL REQUEST HERE]

Instructions to add PDF url to manuscripts.json

Visit: https://github.com/elifesciences/enhanced-preprints-client/actions/workflows/add-pdf-url-to-manuscript.yaml
Click: Run workflow
Complete the form and click "Run workflow"
A successful run should result in a new pull request at https://github.com/elifesciences/enhanced-preprints-client/pulls
Open the pull request and click the "Ready for review" button to trigger tests
Once the tests pass and you are happy with the changes the PR can be merged

Example pull request: https://github.com/elifesciences/enhanced-preprints-client/pull/397/files
Once the pull request is merged in it should be available a few minutes later.

Step 9: Done!

[ ] Kettle is on!

elocations missing from references.


No way to modify antenna gain.

This would be extremely useful for situations when cards need to be read from greater than 1cm. I know the MFRC522-Python library this is based on supports this.

I'm going to look into this shortly, and likely submit a pull request if I have a good solution. I think I already found what needs changing.


Invoker: Added HTML5 Web Messaging communication channel support

Based on FTN7 v1.3

Implemented and tested according to FTN7 v1.3


panic: resource temporarily unavailable on SunOS Elle 5.11 tribblix-m35 i86pc i386 i86pc

A GoToSocial user is trying some wacky deployment types and is encountering some interesting issues trying to run GtS on the abovementioned operating system. They managed to get some logs out of it which seem pretty interesting!
panic: resource temporarily unavailable

goroutine 6 [running]:
github.com/ncruces/go-sqlite3/internal/alloc.Virtual(0x3?, 0x100000000)
        github.com/ncruces/go-sqlite3@v0.19.0/internal/alloc/alloc_unix.go:27 +0xd2
github.com/tetratelabs/wazero/experimental.MemoryAllocatorFunc.Allocate(0x260e660?, 0x40df01?, 0xc0336c3250?)
        github.com/tetratelabs/wazero@v1.8.1/experimental/memory.go:28 +0x1c
github.com/tetratelabs/wazero/internal/wasm.NewMemoryInstance(0xc001cc8110, {0x2f33a60?, 0x2ab5e78?}, {0x2f4dce8, 0xc0334c76c0})
        github.com/tetratelabs/wazero@v1.8.1/internal/wasm/memory.go:78 +0xa8
github.com/tetratelabs/wazero/internal/wasm.(*ModuleInstance).buildMemory(0xc0337a0600, 0xc0016fa4e0, {0x2f33a60?, 0x2ab5e78?})
        github.com/tetratelabs/wazero@v1.8.1/internal/wasm/module.go:653 +0x4c
github.com/tetratelabs/wazero/internal/wasm.(*Store).instantiate(0xc000116fc0, {0x2f3f738, 0xc0337dbd40}, 0xc0016fa4e0, {0x0, 0x0}, 0xc033b8e2c0, {0xc001f0c000, 0x3a, 0x3a})
        github.com/tetratelabs/wazero@v1.8.1/internal/wasm/store.go:370 +0x365
github.com/tetratelabs/wazero/internal/wasm.(*Store).Instantiate(0xc000116fc0, {0x2f3f738, 0xc0337dbd40}, 0x30?, {0x0?, 0x30?}, 0xc00006e908?, {0xc001f0c000, 0x3a, 0x3a})
        github.com/tetratelabs/wazero@v1.8.1/internal/wasm/store.go:327 +0x5a
github.com/tetratelabs/wazero.(*runtime).InstantiateModule(0xc0016f8720, {0x2f3f738, 0xc0337dbd40}, {0x2f48bd0, 0xc001f0a040}, {0x2f56a40, 0xc0051e8d20})
        github.com/tetratelabs/wazero@v1.8.1/runtime.go:318 +0x1e5
github.com/ncruces/go-sqlite3.instantiateSQLite()
        github.com/ncruces/go-sqlite3@v0.19.0/sqlite.go:99 +0x2be
github.com/ncruces/go-sqlite3.newConn({0x2f3f700, 0x60d8460}, {0xc000dc25a0, 0x9b}, 0x46)
        github.com/ncruces/go-sqlite3@v0.19.0/conn.go:76 +0xb4
github.com/ncruces/go-sqlite3.OpenContext(...)
        github.com/ncruces/go-sqlite3@v0.19.0/conn.go:49
github.com/ncruces/go-sqlite3/driver.(*connector).Connect(0xc0016bae10, {0x2f3f700, 0x60d8460})
        github.com/ncruces/go-sqlite3@v0.19.0/driver/driver.go:212 +0xeb
github.com/superseriousbusiness/gotosocial/internal/db/sqlite.(*sqliteConnector).Connect(0x2e5337dbad0?, {0x2f3f700?, 0x60d8460?})
        github.com/superseriousbusiness/gotosocial/internal/db/sqlite/driver.go:64 +0x2a
database/sql.(*DB).conn(0xc0016d8c30, {0x2f3f700, 0x60d8460}, 0x1)
        database/sql/sql.go:1415 +0x71e
database/sql.(*DB).query(0xc0016d8c30, {0x2f3f700, 0x60d8460}, {0xc033b9a000, 0x19}, {0x0, 0x0, 0x0}, 0x0?)
        database/sql/sql.go:1749 +0x57
database/sql.(*DB).QueryContext.func1(0xb0?)
        database/sql/sql.go:1732 +0x4f
database/sql.(*DB).retry(0x411c01?, 0xc0336c3b90)
        database/sql/sql.go:1566 +0x42
database/sql.(*DB).QueryContext(0x0?, {0x2f3f700?, 0x60d8460?}, {0xc033b9a000?, 0x0?}, {0x0?, 0x2f3f700?, 0x60d8460?})
        database/sql/sql.go:1731 +0xc5
github.com/uptrace/bun.(*baseQuery).scan(0xc033ab83c0, {0x2f3f700?, 0x60d8460?}, {0x2f40128?, 0xc033ab83c0?}, {0xc033b9a000, 0x19}, {0x2f368a0, 0xc033753720}, 0x1)
        github.com/uptrace/bun@v1.2.1/query_base.go:562 +0x10c
github.com/uptrace/bun.(*SelectQuery).Scan(0xc033ab83c0, {0x2f3f700, 0x60d8460}, {0xc03379b150?, 0x25aca40?, 0x58?})
        github.com/uptrace/bun@v1.2.1/query_select.go:885 +0x165
github.com/superseriousbusiness/gotosocial/internal/db/bundb.(*applicationDB).GetAllTokens(0xc0053c7370, {0x2f3f700, 0x60d8460})
        github.com/superseriousbusiness/gotosocial/internal/db/bundb/application.go:142 +0x42d
github.com/superseriousbusiness/gotosocial/internal/oauth.(*tokenStore).sweep(0xc0000558a0, {0x2f3f700, 0x60d8460})
        github.com/superseriousbusiness/gotosocial/internal/oauth/tokenstore.go:71 +0x3b
github.com/superseriousbusiness/gotosocial/internal/oauth.newTokenStore.func1({0x2f3f700, 0x60d8460}, 0xc0000558a0)
        github.com/superseriousbusiness/gotosocial/internal/oauth/tokenstore.go:58 +0xfb
created by github.com/superseriousbusiness/gotosocial/internal/oauth.newTokenStore in goroutine 1
        github.com/superseriousbusiness/gotosocial/internal/oauth/tokenstore.go:49 +0xf1

Do you have any idea what might be causing this? Is there maybe some config change they could make?

commenting here so i get notified for this thread, and because it's a frankly fascinating setup :p
though i would not be surprised if wazero ended up being the limiting factor in this situation, rather than SQLite

we're the user doing terrifying things with Solaris! Yeah, it seemed to work for a while, then eventually something went horribly wrong and everything detonated. Also got "slow database query" warnings in the GTS logs.

So is this a Solaris 11, or an illumos OS?
The problem with Solaris, is that:

it doesn't support OFD locks (to be frank, only Linux does, and on macOS they're a private API)
their BSD flock “compatiblity” implementation is broken.

Given that I'm personally uninterested in implementing POSIX locks, the result will always be rather limited. I've tried to come up with some possible simplifications, but POSIX locks simply defy all logic.
That said this doesn't appear to be a locking issue, rather an mmap issue.
How much memory can a process map? The default SQLite configuration tries to mmap 4GB per connection, which is probably excessive. It's uncommitted memory, so just reserving address space, which works out great for Linux 64-bit, (even with overcommit disabled), but maybe it breaks other Unix OSes.
That said, given the locking issues above, if this is Solaris, you can't have more than one connection. I don't know what configuration knobs GoToSocial makes available.

If I understand this correctly, this is trying to allocate 256MB, and mmap is returning EAGAIN:
github.com/ncruces/go-sqlite3/internal/alloc.Virtual(0x3?, 0x100000000)

So there's already some configuration in place (otherwise we would be allocating the full 4GB).
I'm not sure there's anything I can do here, besides maybe disabling the use of mmap on Solaris (it seems to be of little benefit).

I was wrong, this is trying to allocate the full 4GB. Still not sure if I can/should do something "auto-magically".
By default, SQLite asks us to allocate the minimum (hardcoded to 320KB) resizable to a maximum (4GB). This is per connection. Something else could be configured by GoToSocial.
This would set the maximum to 64MB (a page is 64KB, 1024 pages are 64MB):
https://github.com/ncruces/go-sqlite3/blob/368c900db80c44cea3cc26361d4e57719857b6b4/sqlite_test.go#L14
This would set both minimum and maximum to 64MB:
https://github.com/ncruces/go-sqlite3/blob/368c900db80c44cea3cc26361d4e57719857b6b4/internal/testcfg/testcfg.go#L19-L21
SQLite can probably live with 64MB in 99% of cases (the default cache size is 2MB, so 8MB would cover a lot). But degenerate cases may need a lot more (e.g. inserting large BLOBs, huge WALs, etc).
The initial call reserves a lot of address space, but tries not to commit it to memory (which is how it works on Linux, but Solaris may think differently):
https://github.com/ncruces/go-sqlite3/blob/368c900db80c44cea3cc26361d4e57719857b6b4/internal/alloc/alloc_unix.go#L23-L28
This ensures we don't need to move memory later, and it allows us to then remap portions of this for shared memory WAL. Not using mmap means we can't use shared memory WAL, as does moving the memory. Figuring out a better strategy would allow (e.g.) Linux 32-bit to support shared memory WAL (fixing many warnings here, @tsmethurst, @NyaaaWhatsUpDoc).
As for Solaris, this is mostly irrelevant, because it doesn't support decent file locks, and so can't use shared memory WAL. On thing you can try @Here-Be-Saoirse is building with the sqlite3_nosys tag and report back.

@ncruces This is on Tribblix, an Illumos distro. Apologies, we should have been more clear; we use the term SOlaris for that whole thing

@ncruces This is on Tribblix, an Illumos distro. Apologies, we should have been more clear; we use the term SOlaris for that whole thing

Oh, that makes a lot more sense.
Unlike Solaris, illumos actually has working BSD locks, and they're even compatible with POSIX locks, just like a BSD. They even claim to have OFD locks, which would be better than the BSDs, but it's a bit deceiving, since they span the entire file (so are simply equivalent to, and likely implemented on top of, flock).
So I assume the issue is just that illumos (or the VM it's running on, etc) simply doesn't like the idea of reserving 4GB of address space for each connection. Which, tbh, isn't all that surprising. There's a reason I run my parallel tests with the 64MB limit: GitHub Actions doesn't love me having dozens of connections each eating 4GB of continuous address space (even on Linux, which is unusually liberal about this).
This actually validades one of wazero's controversial design decisions (implement sandboxing with bounds checks instead of mapping even more memory and using page faults to sandbox, which is what browsers do).
So I guess the fix here might be for me to change the default amount of memory to reserve, for example:

256MB on 64-bit platforms;
32MB on 32-bit platforms.

Then enable this by default, and enable shared memory WAL on 32-bit platforms.
That's a rather simple change I can commit to the repo, if it makes it easier for y'all to test, but I'd rather release it only after SQLite 3.47 comes out, as both are “major” changes (this should be in a couple of weeks, there's a release candidate, which passes all my tests).

@ncruces That should work. We'd love to test this on actual Solaris, specifically, SPARC64 D Solaris 10u11, but.... golang. Golang is incredibly unfriendly to ports to as-yet unsupported CPU architectures. There doesn't seem to be a frjom-zero bootstrap process, you seem to need golang to build golang, which...... is certainly a choice.  The 'D' there was unintended

I'll get a gotosocial pr up in a few :)

I'll get a gotosocial pr up in a few :)

To configure WithMemoryLimitPages()? That's a good idea; I'll probably do it in go-sqlite3 too (as I wrote above), but that's more immediate for you and I'll love the feedback.
What that won't do is solve WAL on 32-bit platforms for you. Unless you want me to fast track that? I'd rather avoid release notes mentioning breaking compatibility every week. 😬

SPARC64 Solaris 10u11, even. The 'D' there was unintended

Solaris proper, even if it had a working toolchain, has the file locking issue.
It's really insane, IMO, that just opening and closing a file releases locks acquired through another file descriptor. As if I controlled what files other parts of the process open.

@ncruces That's janky as hell.

I'll get a gotosocial pr up in a few :)

To configure WithMemoryLimitPages()? That's a good idea; I'll probably do it in go-sqlite3 too (as I wrote above), but that's more immediate for you and I'll love the feedback.
What that won't do is solve WAL on 32-bit platforms for you. Unless you want me to fast track that? I'd rather avoid release notes mentioning breaking compatibility every week. 😬

here we are! https://github.com/superseriousbusiness/gotosocial/pull/3441
hopefully we'll be able to sneak in another release-candidate of gotosocial so we can test this final change before release of v0.17.0.
don't worry about fast-tracking 32bit WAL support. i'm not even sure it's a platform that we're going to officially support given we only officially support our WebAssembly embedded ffmpeg, and the performance for that in interpreted mode of wazero is not very servicable :')

@ncruces That's janky as hell.

That's a actually working as specified for POSIX locks. If you have time to waste:
https://www.samba.org/samba/news/articles/low_point/tale_two_stds_os2.html
I finally tracked down why this insane behavior was standardized by the POSIX committee by talking to long-time BSD hacker and POSIX standards committee member Kirk McKusick (he of the BSD daemon artwork). As he recalls, AT&T brought the current behavior to the standards committee as a proposal for byte-range locking, as this was how their current code implementation worked. The committee asked other ISVs if this was how locking should be done. The ISVs who cared about byte range locking were the large database vendors such as Oracle, Sybase and Informix (at the time). All of these companies did their own byte range locking within their own applications, none of them depended on or needed the underlying operating system to provide locking services for them. So their unanimous answer was "we don't care". In the absence of any strong negative feedback on a proposal, the committee added it "as-is", and took as the desired behavior the specifics of the first implementation, the brain-dead one from AT&T.

oh goooods that's jank. That's...... bad. that's extremely not good.

Very interesting reading, thanks for the investigations! Kim knows way more about the way we use wazero in GtS so I'll defer to her :) But yeah as she said, please don't fast track anything for our sake, it's not necessary!

One thing that could be interesting to test here is changing wazero to support the compiler on illumos.
The requirements for the compiler aren't that big. It hasn't been opened up to other platforms more for lack of testing than it being hard. It's probably a matter of checking out the code, looking for places that have a freebsd build tag, adding illumos and testing.
I bet that OpenBSD wouldn't work (because “security” that's going to place restrictions on JIT, and us not wanting to deal with that), but other BSDs and illumos might.
I can't say I'm sure I'd be able to convince the other wazero maintainers to merge something like that, but “I tested this in actual hardware, it works, and I'll keep testing it” can't hurt.

SPARC64 Solaris 10u11, even. The 'D' there was unintended

If you ever get Go running, you can try #179 (and, maybe later, #180) on Solaris.
The only dependency for sqlite3_dotlk is an atomic Mkdir, and it may eventually support single process “shared memory” WAL.

This is released, and tested in CI.


Add trigger authentication via JWKS or similar

How can we add JWKS authentication to the media server. For example maybe a new trigger called JWKS_AUTH can be triggered on server load, that can then call an jwks endpoint and then for every subsequent trigger call, the jws token can be send through
I don't know, do we currently have any way to add auth to triggers??
Thank you

Hello!
We do have a plan of adding better/proper JWT/JWKS support. However you can already use this with the current triggers, it just requires work.
Full support is planned for the next API, which honestly we want done yesterday, but haven't been able to make as much progress on due to higher priority projects.
To do this currently:
This would require creating a (local) application or script that verifies the JWTs and is accessed by MistServer.
If you set this script/application to be called by the USER_NEW trigger it should be able to do all necessary processing.
The advantage of this method is that the token need to only be verified once per session, as MistServer will remember the yes/no answer for the duration of the session.

Okay, it's kinda what I am doing now by timestamp signing and veryfing the request origin in my server. I will keep this open until something similar exists

Yeah, that's fine, hopefully we won't disappoint with the wait time 😄


Update how-to-troubleshoot-online-endpoints.md

Add ImageBuildFailure section in TSG.

@frcai : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@Court72 : Thanks for your contribution! The author(s) have been notified to review your proposed change.


[BUILD] Bump Maven from 3.8.8 to 3.9.9

:mag: Description
Issue References 🔗

This pull request fixes #
Describe Your Solution 🔧

bump Maven from 3.8.x to 3.9.x
Maven 3.9 release notes:

3.9.0: https://maven.apache.org/docs/3.9.0/release-notes.html
3.9.9: https://maven.apache.org/docs/3.9.9/release-notes.html

Types of changes :bookmark:

[ ] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪
Behavior Without This Pull Request :coffin:
Behavior With This Pull Request :tada:
Related Unit Tests

Checklist 📝

[x] This patch was not authored or co-authored using Generative Tooling

Be nice. Be informative.

Other popular projects are now using Maven 3.9, e.g. Spark , and Celeborn.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 0.00%. Comparing base (dff40e8) to head (074b7ba).
Report is 2 commits behind head on master.

Additional details and impacted files
@@          Coverage Diff           @@
##           master   #6664   +/-   ##
======================================
  Coverage    0.00%   0.00%
======================================
  Files         683     683
  Lines       42205   42205
  Branches     5756    5756
======================================
  Misses      42205   42205

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Thanks, merged to master (1.10.0).

Seems we have used a different variable in GitHub workflows to gain the same purpose, it would be great to migrate to the official approch.
https://maven.apache.org/configure.html#maven_args-environment-variable

MAVEN_ARGS environment variable:
Starting with Maven 3.9.0, this variable contains arguments passed to Maven before CLI arguments. E.g., options and goals could be defined with the value -B -V checkstyle:checkstyle.

Seems we have used a different variable in GitHub workflows to gain the same purpose, it would be great to migrate to the official approch.
https://maven.apache.org/configure.html#maven_args-environment-variable

MAVEN_ARGS environment variable:
Starting with Maven 3.9.0, this variable contains arguments passed to Maven before CLI arguments. E.g., options and goals could be defined with the value -B -V checkstyle:checkstyle.

Cool. Sounds that MVN_OPT custom env var could be replaced by MAVEN_ARGS in some way.


Support for Background Update Notification in iOS

Hi, is there any current solution to handle silent messages correctly using cloud messaging plugin ?
Native way is to use:
func application(_ application: UIApplication, didReceiveRemoteNotification userInfo: [AnyHashable : Any], fetchCompletionHandler completionHandler: @escaping (UIBackgroundFetchResult) -> Void)
Perform action and call completionHandler.
But plugin does not offer this solution, only wait till app is back in foreground (and then catch it in onLaunch/onResume) or use Notification

https://developer.apple.com/library/archive/documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/CreatingtheNotificationPayload.html#//apple_ref/doc/uid/TP40008194-CH10-SW1
->Configuring a Background Update Notification

How it should be done from dart/flutter perspective
https://github.com/transistorsoft/flutter_background_geolocation/blob/master/lib/background_geolocation.dart

Looks like a  dup of #22305, #22197

OK I see that there is not much will, so I will create PR to support headless event as suggested in other threads following pattern here: https://github.com/transistorsoft/flutter_background_geolocation

there is not much will

Please don't make such arbitrary assumptions


feat(#44): Add tests for Optimization

Add tests for Optimization.
Closes: #44

PR-Codex overview
This PR focuses on adding unit tests for the Optimization class.
Detailed summary

Added unit tests for applying optimization to an empty folder.
Added unit test for applying optimization to a folder with garbage files.
Added unit test for throwing an exception when the folder does not exist.

✨ Ask PR-Codex anything about this PR by commenting with /codex {your question}

@rultor merge

@rultor merge

@volodya-lombrozo OK, I'll try to merge now. You can check the progress of the merge here

@rultor merge

@volodya-lombrozo Done! FYI, the full log is here (took me 3min)


convert_2_magic.tdt

What command was run:
convert_2_magic.tdt(input_dir_path='Lamont 71 x specimens', samp_name_con=2,samp_name_chars=1,,site_name_con=3, site_name_chars=3))
What happened:
the site_name_con and site_name_chars options throw an error.
What I expected to happen:
It should make tables with the desired names
Data file(s):
Lamont 71 x specimens.zip
Screenshot(s):
PmagPy version:
pmagpy-4.2.85

This problem is fixed in pmagpy-4.2.97


Return an error when a opcode does not have a stdlib implementation

We can get into a situation where the backend does not support a particular blackbox function call and the stdlib has not implemented a fallback either. For this, we should return an error to the caller.

Completed as of 16fab1310094d198026b1ee2fec29ba351272328


Trailing line spaces being ignored by word- or subword-jumps

This patch resolves the issue reported by @Neko-Box-Coder in issue (#2665) that trailing spaces are being ignored by left and right word jumps and sub-word jumps.

I have cloned and tested it. Works great. Thanks.
This is the text I used for testing.
test
    test
test

test

Thanks guys!


Can I read options from Environment variables?

First of all thanks for creating this nice package. Been using it for a lot of projects and it's working fine as always :)
I was wondering if it's possible to read options from environment variables as well as from command params?
When I'm running my app in a docker container I would like to be able to edit my env variables in the container and then reflect that to my startup options.
In the old repo, I found this issue https://github.com/gsscoder/commandline/issues/58 but can't seem to find the implementation in the new repo. Was it never implemented or planned as a feature or are they other ways to resolve my "problem"?
If not I can always just read the env variables myself and update the startup options if needed.

I think the consensus in that thread was "Separation of concerns means this library should stick to commandline parsing only", and any implementation of environment variable defaults should be left up to the users who need it, and/or another library if it's a common need.
BTW, if you're implementing this yourself you'll need to decide whether command line args should override environment variables, or environment variables override command line args, in cases when both are present. IMHO command line args should always override environment variables. The simplest way to achieve that is probably to do things like this in your options class:
[Option('f', "--foo", DefaultValue=System.Environment.GetEnvironment("MYPROG_FOO") ?? "")
string Foo { get; set; }

That way --foo=bar overrides MYPROG_FOO=quux, but if no --foo option is given on the command line then the value will be "quux". (And if neither is specified, it will be the empty string rather than null, so you're safe from NullReferenceErrors).

Actually, that wouldn't work since attribute properties must be constants that can be known at compile time (see https://github.com/commandlineparser/commandline/issues/524#issuecomment-544639778). But it might be possible if a DefaultFactory attribute was added similar to the idea I sketched out in https://github.com/commandlineparser/commandline/issues/530#issuecomment-676019847 - would that feature, if implemented, meet your needs?

any movement on this, not being able to use env as defaults is a pretty big thing, because it prevents the tooling from being useful, I would be okay with something like DefaultFunc = () => Environment.GetEnvironmentVariable("ENV")

Yes exactly, the order of precedence makes this problematic. This requirement of constant value for Default makes this really cumbersome to implement.

tumbleweed blowing by

I have worked around this by pre-processing the string[] args array.

Integration is a single line of code. Call args = AppendEnvironmentVariables(args); before processing.
Precedence is preserved, so command-line args override environment variables which in turn override defaults.
Environment variables names are assumed to be {ASSEMBLY}_{VERB}_{OPTION}.

It works by:

Getting the list of options associated with the verb.
Filtering out any options that were provided as command-line args.
Doing an environment variable lookup for the remaining unused options.
If the environment variable is found, appending it to the command-line args.

Here's the code, which is totally untested beyond my use case, so YMMV:
static void Main(string[] args)
{
    args = AppendEnvironmentVariables(args);
    ...
}

private static string[] AppendEnvironmentVariables(string[] args)
{
    if (args.Length == 0)
    {
        return args;
    }

    var verb = args[0];
    if (!TryGetOptions(verb, out var options))
    {
        return args;
    }

    var newArgs = new List<string>(args);
    foreach (var unusedOption in FilterOptions(args, options))
    {
        var value = Environment.GetEnvironmentVariable($"{Assembly.GetExecutingAssembly().GetName().Name.ToUpperInvariant()}_{verb.ToUpperInvariant()}_{unusedOption.LongName.ToUpperInvariant()}");
        if (value != null)
        {
            newArgs.Add($"--{unusedOption.LongName}={value}");
        }
    }
    return newArgs.ToArray();
}

private static bool TryGetOptions(string verb, out OptionAttribute[] options)
{
    foreach (var type in Assembly.GetExecutingAssembly().GetTypes().Where(type => !type.IsAbstract))
    {
        var verbAttribute = type.GetCustomAttributes<VerbAttribute>().FirstOrDefault();
        if (verbAttribute != null)
        {
            if (verbAttribute.Name == verb)
            {
                options = type.GetProperties()
                    .Select(property => property.GetCustomAttributes<OptionAttribute>().FirstOrDefault())
                    .Where(option => option != null).ToArray();
                return true;
            }
        }
    }

    options = null;
    return false;
}

private static OptionAttribute[] FilterOptions(string[] args, OptionAttribute[] options)
{
    var usedLongNames = new HashSet<string>();
    var usedShortNames = new HashSet<string>();

    foreach (var arg in args)
    {
        if (arg.StartsWith("--"))
        {
            var longName = arg.Substring(2);
            if (longName.Contains('='))
            {
                longName = longName.Substring(0, longName.IndexOf('='));
            }
            usedLongNames.Add(longName);
        }
        else if (arg.StartsWith("-"))
        {
            var shortName = arg.Substring(1);
            if (shortName.Contains('='))
            {
                shortName = shortName.Substring(0, shortName.IndexOf('='));
            }
            usedShortNames.Add(shortName);
        }
    }

    return options.Where(option => !usedLongNames.Contains(option.LongName) && !usedShortNames.Contains(option.ShortName)).ToArray();
}

Hope this helps someone else!

I have worked around this by pre-processing the string[] args array.

Integration is a single line of code. Call args = AppendEnvironmentVariables(args); before processing.
Precedence is preserved, so command-line args override environment variables which in turn override defaults.
Environment variables names are assumed to be {ASSEMBLY}_{VERB}_{OPTION}.

It works by:

Getting the list of options associated with the verb.
Filtering out any options that were provided as command-line args.
Doing an environment variable lookup for the remaining unused options.
If the environment variable is found, appending it to the command-line args.

Here's the code, which is totally untested beyond my use case, so YMMV:
static void Main(string[] args)
{
    args = AppendEnvironmentVariables(args);
    ...
}

private static string[] AppendEnvironmentVariables(string[] args)
{
    if (args.Length == 0)
    {
        return args;
    }

    var verb = args[0];
    if (!TryGetOptions(verb, out var options))
    {
        return args;
    }

    var newArgs = new List<string>(args);
    foreach (var unusedOption in FilterOptions(args, options))
    {
        var value = Environment.GetEnvironmentVariable($"{Assembly.GetExecutingAssembly().GetName().Name.ToUpperInvariant()}_{verb.ToUpperInvariant()}_{unusedOption.LongName.ToUpperInvariant()}");
        if (value != null)
        {
            newArgs.Add($"--{unusedOption.LongName}={value}");
        }
    }
    return newArgs.ToArray();
}

private static bool TryGetOptions(string verb, out OptionAttribute[] options)
{
    foreach (var type in Assembly.GetExecutingAssembly().GetTypes().Where(type => !type.IsAbstract))
    {
        var verbAttribute = type.GetCustomAttributes<VerbAttribute>().FirstOrDefault();
        if (verbAttribute != null)
        {
            if (verbAttribute.Name == verb)
            {
                options = type.GetProperties()
                    .Select(property => property.GetCustomAttributes<OptionAttribute>().FirstOrDefault())
                    .Where(option => option != null).ToArray();
                return true;
            }
        }
    }

    options = null;
    return false;
}

private static OptionAttribute[] FilterOptions(string[] args, OptionAttribute[] options)
{
    var usedLongNames = new HashSet<string>();
    var usedShortNames = new HashSet<string>();

    foreach (var arg in args)
    {
        if (arg.StartsWith("--"))
        {
            var longName = arg.Substring(2);
            if (longName.Contains('='))
            {
                longName = longName.Substring(0, longName.IndexOf('='));
            }
            usedLongNames.Add(longName);
        }
        else if (arg.StartsWith("-"))
        {
            var shortName = arg.Substring(1);
            if (shortName.Contains('='))
            {
                shortName = shortName.Substring(0, shortName.IndexOf('='));
            }
            usedShortNames.Add(shortName);
        }
    }

    return options.Where(option => !usedLongNames.Contains(option.LongName) && !usedShortNames.Contains(option.ShortName)).ToArray();
}

Hope this helps someone else!

@avenema that's amazing love it! :D,
however I am a bit curious what's the reasoning about naming spacing it by the executing assembly? :thinking:

@avenema that's amazing love it! :D,
however I am a bit curious what's the reasoning about naming spacing it by the executing assembly? :thinking:

@jannickj good question. I did it that way to avoid collisions with other projects. It's clear from the environment variable name that the value is specific to a particular executable and verb.
I'm using this approach in these three projects:

https://github.com/liveswitch/liveswitch-connect
https://github.com/liveswitch/liveswitch-hammer
https://github.com/liveswitch/liveswitch-mux

... and there's a possibility that at some point a verb/option pair might collide.

@jannickj good question. I did it that way to avoid collisions with other projects. It's clear from the environment variable name that the value is specific to a particular executable and verb.
I'm using this approach in these three projects:

https://github.com/liveswitch/liveswitch-connect
https://github.com/liveswitch/liveswitch-hammer
https://github.com/liveswitch/liveswitch-mux

... and there's a possibility that at some point a verb/option pair might collide.

@avenema hehe fair enough, and I guess this is sort of an exercise left to anyone wanting to improve it however I think in general when it comes to namespacing it's preferable to have it be optional I.e. the AppendEnvironmentVariables could have an optional argument string? namespace = null

@avenema hehe fair enough, and I guess this is sort of an exercise left to anyone wanting to improve it however I think in general when it comes to namespacing it's preferable to have it be optional I.e. the AppendEnvironmentVariables could have an optional argument string? namespace = null


2.0.6 TSServer protocol.d.ts: new features are missing documentation

TypeScript Version:  2.0.6-Insider
All new features are missing documentation. Hard to use without knowing how to :-)

Can you guys please merge this into 2.0.6 as well. I pulled today and it is not there. Looks like it is only available in master.


Attachments / multi line messages

Create some sort of support for attachment / multi line messages. Now it's just and empty message and the sender will be rendered as "unknown".

@erroneousboat this issue actual in 0.4.1 version too. Without attachments – this client is not so useful 😢

@lamartire this is a very old issue and referenced certain integrations not showing enough information. I'm not sure what you're getting at specifically, but on the master branch there has been made a change that adds more information for files. (#179)

@erroneousboat oh, thank you for fast response! 🙌
I'm kalmed down now 🌈


Please contribute a driver to the Linux kernel

Userspace hacks like this are very frustrating for professional developers. Please do the right thing and contribute a driver to the Linux kernel itself.

thank you for you input. We are well aware of the difference between user space and kernel drivers.
We however recommend to use our drivers from user space and don't consider this as a hack.
Of course we are also very happy if our sensors are supported in the linux kernel. Currently we however don't have any plans to develop kernel drivers our self.


[ariaDownload.py] add progress bar

This PR includes the addition of a progress bar to ariaDownload.py.

Hello @ehavazli! Thanks for opening this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file tools/bin/ariaDownload.py:

Line 324:65: E502 the backslash is redundant between brackets
Line 327:65: E502 the backslash is redundant between brackets
Line 385:80: E501 line too long (83 > 79 characters)

I tested downloading 15 products over California.
ariaDownload.py -t 64 -b '34.5 35.7 -116.5 -114.2' -s 20180101 -e 20180201 -o download

The download and progress bar worked fine, but download took more than 17 minutes (> 1 min per product).
Downloading without progress bar took ~3 min.
A second attempt using the progress bar took only 5 min 42 sec.

I tested downloading 15 products over California.
ariaDownload.py -t 64 -b '34.5 35.7 -116.5 -114.2' -s 20180101 -e 20180201 -o download

The download and progress bar worked fine, but download took more than 17 minutes (> 1 min per product).
Downloading without progress bar took ~3 min.
A second attempt using the progress bar took only 5 min 42 sec.

It took me about 19 seconds
2024-12-18 12:03:55,408 - ariaDownload.py - INFO - Downloading 15 products...
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:17<00:00,  1.19s/file]
2024-12-18 12:04:13,336 - ariaDownload.py - INFO - Download complete. Wrote -- 15 -- products to: /Users/havazli/Desktop/ARIA_tests/products
ariaDownload.py -t 64 -b '34.5 35.7 -116.5 -114.2' -s 20180101 -e 20180201 -o  2.53s user 2.37s system 26% cpu 18.830 total
(ARIA-tools) havazli@MT-315227 ARIA_tests %


Clear and refetch?

What would be the recommended way to clear + refetch a single particular query? I'm seeing refetch() and queryCache.clear(), but the former doesn't clear the cache for the query, and the latter presumably clears the cache for all queries. Thanks.

Worked it out with:
queryCache.removeQueries(myQueryKey);
myQuery.refetch();

But would be nice if useQuery() returned a clear function or maybe refetch() accepted a shouldClear option.


New package: Aqua v0.1.0

Registering package: Aqua
Repository: https://github.com/tkf/Aqua.jl
Created by: @tkf
Version: v0.1.0
Commit: 3ef8e0b791690c75fd60156062860694dfbc00f1
Reviewed by: @tkf
Reference: https://github.com/tkf/Aqua.jl/issues/1

https://github.com/JuliaRegistries/General/pull/993#issuecomment-496305767

I added julia = "1.0" in [compat].