Refactor InputTransform and DataModule

What does this PR do?
Resolves discussion from #1166
At present, the InputTransform for every DataModule is being passed for every stage even though InputTransform has specific methods to differentiate the stage at which the certain transform runs. This also creates 4 different instances of the class in each input (train, val, test, predict), with different methods to be run.
This PR aims to change the aforementioned by making the DataModule as the owner of the InputTransform because all the class does it to generate collate_fn for the dataloaders and the implementation for the on_after_batch_transfer callback.
Thus a single instance of the InputTransform class, present in the DataModule, can resolve the required Callables for every stage and the appropriate dataloader collate_fn and on_after_batch_transfer functions are created in the DataLoader's __init__ method.
This also relieves the Input class from having to take care of the InputTransform.
TL;DR
Previous API
dm = XYZTask_DataModule.from_xyz(
    train_file=train_file,
    val_file=val_file,
    test_file=test_file,
    predict_file=predict_file,
    train_transform=InputTransform,
    val_transform=InputTransform,
    test_transform=InputTransform,
    predict_transform=InputTransform,
    transform_kwargs=transform_kwargs,
)

# Implementation
class XYZTask_DataModule(DataModule):

    @classmethod
    def from_xyz(
        cls,
        train_file=train_file,
        val_file=val_file,
        test_file=test_file,
        predict_file=predict_file,
        train_transform=InputTransform,
        val_transform=InputTransform,
        test_transform=InputTransform,
        predict_transform=InputTransform,
        transform_kwargs=transform_kwargs,
        input_cls=Input,
    ):
        return cls(
            input_cls(RunningStage.TRAINING, train_file, transform=train_transform, **transform_kwargs),
            input_cls(RunningStage.VALIDATING, val_file, transform=val_transform, **transform_kwargs),
            input_cls(RunningStage.TESTING, test_file, transform=test_transform, **transform_kwargs),
            input_cls(RunningStage.PREDICTING, predict_file, transform=predict_transform, **transform_kwargs),
        )

New API
dm = XYZTask_DataModule.from_xyz(
    train_file=train_file,
    val_file=val_file,
    test_file=test_file,
    predict_file=predict_file,
    transform=InputTransform,
    transform_kwargs=transform_kwargs,
)

# Implementation
class XYZTask_DataModule(DataModule):

    @classmethod
    def from_xyz(
        cls,
        train_file=train_file,
        val_file=val_file,
        test_file=test_file,
        predict_file=predict_file,
        transform=InputTransform,
        transform_kwargs=transform_kwargs,
        input_cls=Input,
    ):
        return cls(
            input_cls(RunningStage.TRAINING, train_file),
            input_cls(RunningStage.TRAINING, val_file),
            input_cls(RunningStage.TRAINING, test_file),
            input_cls(RunningStage.TRAINING, predict_file),
            transform=transform,
            **transform_kwargs
        )

Before submitting

[x] Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
[x] Did you read the contributor guideline, Pull Request section?
[x] Did you make sure your PR does only one thing, instead of bundling different changes together?
[x] Did you make sure to update the documentation with your changes?
[ ] Did you write any new necessary tests? [not needed for typos/docs]
[ ] Did you verify new and existing tests pass locally with your changes?
[ ] If you made a notable change (that affects users), did you update the CHANGELOG?

PR review

[x] Is this pull request ready for review? (if not, please submit in draft mode)

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
Did you have fun?
Make sure you had fun coding 🙃

Hello all,
I have updated the PR for almost all tasks.
I need some help for:

ServeInput and serve related parts.
Pointcloud tasks retrieve their collate_fn when model is instantiated. The updated _collate_fn method in InputTransform class accepts an extra argument (stage: RunningStage) and I wasn't sure of how to set that up in a cleaner manner.
Same issue as above with the image.ObjectDetectionTask which fails for the test_init_train testing function. The extra argument is causing an error. Need some help in updating/changing this test.

Thanks

Awesome work @karthikrangasai !


add ability to request all todos for a given project and todolist

only added one function styled after similar requests in the api

Thanks!


is_night remaining true during day

Sorry to be a bother, but this issue seems to have returned.
Sunrise was 7:25am this morning where I am and at 8:30am, the time I am writing this now, the "is_night" flag is still remaining as true.

Okay.  Maybe I was just not patient this morning.
1 hour and 15 mins after sunrise it switched to is_night=false.
Is this a normal time frame to expect?

Hi @austwhite
As the data is only refreshed every 30 minutes, there might be some delay, but 1hour and 15min seems too long. I just recently discovered that Weatherbit actually supplies a data value called POD (Part of Day) and that shows d for day and n for night. That might be more accurate than my calculation based on Timezone and Observation Time. I will monitor this for the next few days, and if this better, it will be an easy fix to implement this.

That would be amazing.  That could make it function more like Dark Sky use (which I have to totally abandon as it is a unavailable yet again, even though it is not officially turned off yet)

I could use the sun.sun but that would only work if HA and the location your get Weather data for, are in the same place. If you add a location in Denmark and one in the US, and your HA installation is in the US, the Icon for Denmark would be wrong. That is why I try to get the data from the observation time for the location in the Weatherbit data.
If you want to test it out right away, go to the custom_components/weatherbit folder, and open manifest. json and change the below line so that is says 0.10 instead of 0.9 - That uses the POD parameter. If it does not work, just change back to 0.9. Remember to restart HA after the change.
"requirements": [       "weatherbitpypi==0.10"     ],

Everything working smoothly on my end.  I'll close this now.  Thanks for your help as always.


Resolve prettier config relative to current file

Use prettier.resolveConfig to load .prettierrc relative to the open file being formatted.
This allows settings like
{
  files: '*.java',
  tabWidth: 4
}

defined in project workspace to be respected by the extension.

Thanks! Looks good so I've merged in. Will deploy to the marketplace in due course.


Adding support for linux/darwin for Powershell function

Issue : https://github.com/masterzen/winrm/issues/117

Scratch that....you're right.


Improve logic for placing inline image previews within messages (#7537)

The image inlining within is as follows:

If image links appear in a paragraph, add inline image previews for every link to the end of the paragraph in order of appearance.

If a paragraph consists only of an image link, replace the paragraph consisting of only an image link with an inline image preview

If image links appear in a bullet list item, add inline image previews for every link to the end of the item in which they appear

If an item consists only of an image link, replace the image link with an inline image preview

In the case of blockquotes, the same logic as for paragraphs is followed since blockquotes contain paragraphs within.  The inline image previews will always be inside the blockquotes their respective links appear in, not after.

Also, if an image link is named, e.g. [Best Image in The World](https://bestimages.com/amazing.png), then even if a paragraph or list item consists only of this link, it is not deleted, since I make the assumption that if you gave your link a name you wish for the link to persist.
The modified code in this pull request is currently working and properly covered with tests;  However, I want to give it a day and look at it again later to be sure everything is up to our code standards c:

This will be great to have, thanks @PentaHiggs :)

I think my code is ready for a final review!  And I'll try to remember to title the pull request in that manner next time, @rishig , sorry about that.

Now that I've used it for a bit, I'm thinking it might make sense to make this a bit more aggressive; i.e. put things after the next \n, rather than the next paragraph break. I feel like it's pretty common to do something like
text text text
[pasted image]
more text

expecting the image to be "inline".
@PentaHiggs, thoughts?

Hmm, yeah, that makes sense to me -- a  should suffice.  Let's open an issue (including the note about adding cmoments).


Calculate the hashes on the gpu

It is a common knowledge that pipenv lock takes FOREVER to complete on large projects with hundreds of dependancies. I am not sure which step is the bottleneck, but if it is hash calculation of thousands of files, maybe it would be a good idea to utilise the gpu for that? It would be amazing if it was possible to reuse some bitcoin miners for pipfile locking.

This is definitely a good idea, but unfortunately is going to have to be out of scope until we implement some kind of plugin system which isn't even on the roadmap yet.  I don't fancy supporting the implementation for offloading processing to an optional GPU abstraction layer, packaging in python doesn't even support detection of this in environment markers yet :(
if you have implementation ideas I'd suggest submitting a PEEP for that (probably need one for a plugin system first with some sample use cases) but I'm super curious about how the actual GPU aspect would be managed so if there is a cogent proposal here it could expedite the plugin system aspect
closing this specific issue for now until we have something more concrete to act on, but we are doing too much directly in pipenv already so I am thinking this would need to be implemented somewhere else as an extension

Noong Biy, Mar 15, 2019, 2:16 AM, isinulat ni Juozas Masiulis <
<EMAIL_ADDRESS>ang:

It is a common knowledge that pipenv lock takes FOREVER to complete on
large projects with hundreds of dependancies. I am not sure which step is
the bottleneck, but if it is hash calculation of thousands of files, maybe
it would be a good idea to utilise the gpu for that? It would be amazing if
it was possible to reuse some bitcoin miners for pipfile locking.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/pypa/pipenv/issues/3618, or mute the thread
https://github.com/notifications/unsubscribe-auth/AqyQvt6NDQTa1sem4EL8dMkB36TIu-gbks5vWpIBgaJpZM4b06GT
.


RefMagic on Mono

Hello! I'm having mostly a good time integrating with your library, thank you for making it available!
But now I'm having problems with it :(
I have a game client that runs fine on Windows. Now I'm trying to run on Android and I'm having trouble with RefMagic. Do you have it documented anywhere, by the way?
First of all, Utils.IsMono doesn't seem to be enough for the conditional RefFieldValue call. Check if what I'm doing makes sense:
public static readonly bool IsMono = Type.GetType("Mono.Runtime") != null
                                  || RuntimeInformation.OSDescription.Contains("android");

After that, I'm getting the following error:

[libc] Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x40 in tid 23914 (.Client.Android), pid 23914 (.Client.Android)

Does this look like anything to you? I debugged it and it's not the first call that fails, but one of the first ten. I tried to move only this specific crashing call from RefFieldValueMono to RefFieldValueDotNet (probably insane) and got "arith type mismatch" (as expected). I also tried adding IntPtr.Size to the offset for this specific crashing call only, which passes the breaking point, but the returned reference is null, so it crashes somewhere else at call-site.
By the way, I'm running the last version of MonoGame on .NET 7 and x64 everywhere, but I've also tried every architecture I could and the repro seems the same. I'm using an Android emulator.
Thanks!

@Logerfo didn't checked Android+.NET 7. Unity(Mono)+Android works fine. Seems that offsets can be another on Android.

Do you have it documented anywhere, by the way?

It just adds two values, here's source:
https://github.com/RevenantX/LiteEntitySystem/blob/master/LiteEntitySystem/ILPart/RefMagic.il

Seems that offsets can be another on Android

@RevenantX what can I do to discover the offset? Thank you for your reply!

@Logerfo you can write simple app using RefMagic and try to scan memory (something like here https://github.com/RevenantX/LiteEntitySystem/blob/master/LiteEntitySystem/Internal/EntityClassData.cs#L74)
Set some value to 0xDEADBEEF and try scan memory in code or IDE.

@RevenantX if the offset is different on Android with .NET 7, isn't the check Utils.RefFieldValue<uint>(to, monoFieldOffset) == to.TestValue supposed to fail? Because it is not. Any chance the offset might not be always the same?

Can you explain the reason for the mask & 0xFFFFFF? I'm trying to understand the concept behind this code and hopefully fix the problem sending a PR.

@RevenantX if the offset is different on Android with .NET 7, isn't the check Utils.RefFieldValue<uint>(to, monoFieldOffset) == to.TestValue supposed to fail? Because it is not failing. Any chance the offset might not always be the same?

So .NET7 on android has same offset like mono?

Any chance the offset might not always be the same?

I don't think this can be. Maybe there is some issues with some memory aligning.
Maybe try with some minimal project to test.

@Logerfo

I debugged it and it's not the first call that fails, but one of the first ten.

Also if some of them works maybe there is some pattern or error in another place.

Also if some of them works maybe there is some pattern or error in another place.

I'm not sure if it works properly, all I can say for now it's that it doesn't crash, but I might dig deeper into it to see if I can give you a better answer.

I think I've found something! I changed your code a bit to have a step-by-step comprehension like that:

And it seems like a problem is happening with the first field (which is a SyncString) of the first entity to hit the yellow-highlighted line.
Here's the results running the desktop client:

And running the Android client:

Does this look like anything to you?

Hello! I'm having mostly a good time integrating with your library, thank you for making it available! But now I'm having problems with it :(
I have a game client that runs fine on Windows. Now I'm trying to run on Android and I'm having trouble with RefMagic. Do you have it documented anywhere, by the way? First of all, Utils.IsMono doesn't seem to be enough for the conditional RefFieldValue call. Check if what I'm doing makes sense:
public static readonly bool IsMono = Type.GetType("Mono.Runtime") != null
                                  || RuntimeInformation.OSDescription.Contains("android");

After that, I'm getting the following error:

[libc] Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x40 in tid 23914 (.Client.Android), pid 23914 (.Client.Android)

Does this look like anything to you? I debugged it and it's not the first call that fails, but one of the first ten. I tried to move only this specific crashing call from RefFieldValueMono to RefFieldValueDotNet (probably insane) and got "arith type mismatch" (as expected). I also tried adding IntPtr.Size to the offset for this specific crashing call only, which passes the breaking point, but the returned reference is null, so it crashes somewhere else at call-site.
By the way, I'm running the last version of MonoGame on .NET 7 and x64 everywhere, but I've also tried every architecture I could and the repro seems the same. I'm using an Android emulator. Thanks!

I've currently checked .Net + Monogame + Android and it works fine with your fix (RuntimeInformation.OSDescription.Contains("android");)
Maybe you can send to me project with problem?

I don't currently have a minimal repro. My game is being developed closed-source to avoid easing for cheaters, but I can give you access to the project if you're ok with the burden of the information overload that the non-minimal implementation can cause. I also changed a lot of the internal LiteEntitySystem code style to match the project's, so it can feel a little odd for you. What do you think?

@Logerfo well if you made changes, this can be anything. Including bugs of Monogame. Because simple monogame project with connection and players classes works fine with Mono check

I'll try to build a minimal repro using LiteEntitySystem as a git module.

I managed to reproduce the issue in a minimal repo. Here it is: https://github.com/Logerfo/liteentitysystem-android-bug-minimal-repro

Confirmed! Thank you very much!!


80 Add bundle path for ci test

Fixes #80 .
Description
This PR adds bundle path for ci test, thus the customized code can be recognized.
Status
Ready
Please ensure all the checkboxes:

[x] Codeformat tests passed locally by running ./runtests.sh --codeformat

Hi @wyli @yiheng-wang-nv ,
To standardize the bundles in the model-zoo, should we force developers to put the customized code in "scripts" folder and add the "scripts" to the PYTHONPATH?
What do you think?
Thanks.

From my experience, a standardized policy helps. Let developers to put code in "scripts" and add path is good.


Rename middleman-template?

Consistent with:

https://github.com/thoughtbot/react-native-template
https://github.com/thoughtbot/ios-template
https://github.com/thoughtbot/android-template
https://github.com/thoughtbot/django-template

Done! Thanks for that, Dan!


Syntax highlighting is done on plain text LICENSE files

For example, this file when migrated into Gogs is highlighted:
https://github.com/yurivict/node-line-tracer/blob/master/LICENSE

gogs mis-treat it as something like shell script?

yes

Fixed in develop branch. See in action: https://try.gogs.io/gogs/gogs/src/develop/LICENSE

Any special reason to fix incorrect highlighting only on LICENSE files? This can't be right.
All files using the standard .txt extension or files without an extension (usually in all caps, but not necessarily) like README, CHANGELOG, LICENSE, HISTORY, meta, todo etc., should not be highlighted because they're readable plain text.
Some examples below.
LICENSE - Not an actual LICENSE file, just renamed that way so no highlighting occurs
LICENSE2 - A number added to LICENSE automatically highlights the text
LICENSE.txt - the standard txt extension automatically highlights the text

@Unknwon Should I create a new issue?

@megadr01d

Any special reason to fix incorrect highlighting only on LICENSE files?

https://github.com/gogits/gogs/blob/master/modules/template/highlight/highlight.go#L16

@Unknwon I see. But what about any file with a .txt extension getting highlighted? highlightExts is not including .txt.
Does .txt have to be explicitly added in highlightMapping?

@megadr01d there is no perfect solution for highlighting, but I think in your case add following content to your app.ini should disable highlighting for .txt files:
[highlight.mapping]
.txt= nohighlight

@Unknwon Since I'm using a 3rd party hosted Gogs, let me get back to you, on this, before I close this and the other issue I created. I thought it was a bug within Gogs but I see it's normal behavior of highlightjs.


Use our own FFT implementation

Closes #40.

We ended up switching to the mel crate in #107 so this PR is no longer needed.


Admin page not working when setting a PUBLIC_URL

When setting a PUBLIC_URL via server.js, the admin page opens correctly at https://example.com/strapi/dashboard
However, the javascript files in the HTML page cannot be loaded (cf. screenshot) as the PUBLIC_URL (PUBLIC_ADMIN_URL) seems not to be used.
module.exports = ({ env }) => ({
  host: env('HOST', '<IP_ADDRESS>'),
  port: env.int('PORT', 1337),
  url: env('PUBLIC_URL', 'https://example.com/strapi'),
  proxy: env.bool('IS_PROXIED', true),
  admin: {
    auth: {
      secret: env('ADMIN_JWT_SECRET', 'myadminsecret'),
    },
	url: env('PUBLIC_ADMIN_URL', '/dashboard'),
  },
});

/admin/runtime~main.d1293aad.js
should be
/strapi/dashboard/runtime~main.d1293aad.js

Make sure to set those environment files during the build stage of your Dockerfile.

You might also want to check https://forum.strapi.io/t/deployed-strapi-admin-panel-trying-to-hit-localhost-1337/7598

did you run the strapi build command after changing the configuration? Afaik the strapi web ui needs to recompile after changing such fundamental configuration.

If the URL is defined at build time, then the output image is tied to that url and can not be reused to deploy to different environments like dev, qa, prod …
A suggestion is that the public url for the admin ui must be set at runtime instead of build time


Error Inference via command line for CPU

Describe the bug
I am trying to run an inference on colab using only CPU
Using Deepspeed CMD for inference
i am using the below command with my transcribe.py file and model_folder saved model folder
!deepspeed --include= localhost:0 "/content/vistaar/transcribe.py" "/content/manifest.json" "/content/model_folder/" "Hindi" 1 "/content/output_path.txt".
Required Output
what should i specify the parameter for device to use only cpu for inference "--include" or "--exclude" or install intel-extension-for-deepspeed
Existing output
[INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-08-22 11:30:51,120] [WARNING] [runner.py:201:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
Traceback (most recent call last):
  File "/usr/local/bin/deepspeed", line 6, in <module>
    main()
  File "/usr/local/lib/python3.10/dist-packages/deepspeed/launcher/runner.py", line 420, in main
    raise RuntimeError("Unable to proceed, no GPU resources available")
RuntimeError: Unable to proceed, no GPU resources available

Take a look at our CPU Inference workflow for an idea of how to setup your environment for CPU inference. For example, you will need to build and install the intel-extension-for-pytorch:
https://github.com/microsoft/DeepSpeed/blob/5e16eb2c939707d0d0062a458d77998fccb3afad/.github/workflows/cpu-inference.yml#L25

@Dayananda-Akaike-Tech what does ds_report output when you run it?
Also can you share the output of numactl --hardware? Thanks


Broken

example code is broken

@Hsanka6 just checked - it works fine for me in both Xcode 8 and 9. What exact issue do you have with the example project?


"Operation returned an invalid status: Not Acceptable" issue raised while using generatedetailedCost report from azure.mgmt.costmanagement SDK

Package Name:
Package Version:
Operating System:
Python Version:

Describe the bug
We are using azure.mgmt.costmanagement python SDK and when we tried to begin_create_operation() from generateDetailedCostReport class then ' Operation returned an invalid status 'Not Acceptable'' this error came up. And this error is not frequently comes up.
To Reproduce
Steps to reproduce the behavior:
1.
Expected behavior
This error should not happen.
Screenshots

Additional context
Add any other context about the problem here.

Hi @tsaindane-cloudwiz  thank you for the feedback, can you please provide the package-version, operating system and python-version and we will get back to you asap.

python package version: 4.0.1
environment: google cloud function V1
python version: 3.9


attempt fixing TestSCons.where_is() for win32

When using testing framework for external tests (-e), the method TestSCons.where_is() seems to always return None under win32. This PR fixes this. Tested with one of my projects on appveyor.

Can you point to URL for your appveyor results before/after this change?

Can you point to URL for your appveyor results before/after this change?

This job, for example, used the patched testing framework:
https://ci.appveyor.com/project/ptomulik/scons-tool-dvipdfm/builds/20088248/job/i2ke4pfn50uxi481
no skipped tests (means, where_is() detected correctly), while this
https://ci.appveyor.com/project/ptomulik/scons-tool-dvipdfm/builds/20089404/job/4tuf3jdfh6omkv95
uses unpatched testing framework -- three tests skipped (same programs not found by where_is()) .

This was the project under test: https://github.com/ptomulik/scons-tool-dvipdfm/tree/0.1.0/

From travis-ci (Build system information):
clang version
clang version 7.0.0 (tags/RELEASE_700/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/local/clang-7.0.0/bin

looks like they changed default version of clang on Xenial to '7.0.0' . The .travis/install.sh symlinks '5.0.0', but it seems to make clang tests to fail at later stage.

Good catch on clang. was just digging in.
It appears py3.7-dev image only no longer has clang version we're pointing to.. I'll update the travis file

rebase from master?

rebase from master?

yes, just did it.

Coverage decreased (-1.7%) to 79.917% when pulling 210b61f451bef2edf96bc2dafb5055d895f5af22 on ptomulik:whereis into 9685d23fc73e3266abd4a5a47669b9d321c53665 on SCons:master.

Coverage decreased (-1.7%) to 79.917% when pulling 210b61f451bef2edf96bc2dafb5055d895f5af22 on ptomulik:whereis into 9685d23fc73e3266abd4a5a47669b9d321c53665 on SCons:master.

Coverage decreased (-1.7%) to 79.917% when pulling 210b61f451bef2edf96bc2dafb5055d895f5af22 on ptomulik:whereis into 9685d23fc73e3266abd4a5a47669b9d321c53665 on SCons:master.

Clear Ci. Merging

Thanks!


assignment recipe appears to have incorrect definition.type

"type":"http://adlnet.gov/expapi/activities/module",
surely the activity is "doing an assignment"

For monday:
is it http://xapi.vocab.pub/datasets/adl/#assessment ?
https://github.com/jiscdev/xapi/blob/v0.5/recipes/assignment-submitted.md


Ability to persist files/directories within an enclave outside of Kurtosis in a location I know will be permanent

Background & motivation
I'm prototyping/deving a Python app within my package which has a Jupyter notebook, a database, and a Streamlit app inside of it. As I'm prototyping, I'm writing Jupyter notebook files to disk on the Jupyter container and Streamlit python files to disk on the Streamlit container, in addition to persisting database content to disk in the database container.
I have no way of getting my work out of my development environment except for going into the Docker UI and downloading each of the files I'm persisting in order to save them for later. This is most important for be to be able to use my work across multiple days, or share it with other people. In addition - the community I'm trying to build, which would use my prototyping tool, would have a hard time working with this workaround.
Desired behaviour
The ability to specify a "persistent write-out" directory (could be S3 or HDFS or remote filesystem, but also local is just fine for the local CLI use case) which Kurtosis would send directory/file-content to, for persistent directories, in my enclave
How important is this to you?
Painful; the lack of this feature makes using Kurtosis frictionful.
What area of the product does this pertain to?
Other: anything not covered by the above

Plus one for this. Would be nice to have option to use Directory as directory.
Like for postgresql data
ServiceConfig(
  image=POSTGRES_IMAGE,
  files={
    "/var/lib/postgresql/data": Directory(local_dir="/path/to/local/directory")
  }
)


Bug with TimeProvider (PING)

Describe the bug
I think there's a bug within the TimeProvider.updateWithPing().
I assume it's missing an "L" and casting wrong:
Corrent: long diff = ByteBuffer.wrap(pingPayload).getInt() * 1000L - System.currentTimeMillis();
instead of
Wrong: long diff = ByteBuffer.wrap(pingPayload).getInt() * 1000 - System.currentTimeMillis();

You are right, I fixed that.


Android MTP doesn’t work with NixOS 21.11

Describe the bug
Since I upgraded my machines, they doesn’t work with my Android anymore. What should I do? Thanks for any help!
Steps To Reproduce
Steps to reproduce the behavior:

Connect to an Android device
Nothing happens
...

Expected behavior
MTP storage is detected
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.
Notify maintainers

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
output here

Maintainer information:
# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

Duplicate of https://github.com/NixOS/nixpkgs/issues/145824

Oh, yes, thanks!


Missing content if it consist only from zero

Hello!
We used the library to get streamed responses from Openai.
And we found a problem that sometimes streamed chunks do not have content.
I ask model: "multiply 100 on 10" And responses is:
  {"id":"chatcmpl-7dyU298SlZywQtM5bhjXMlPJvd3Bx","object":"chat.completion.chunk","created":1689761646,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}
  {"id":"chatcmpl-7dyU298SlZywQtM5bhjXMlPJvd3Bx","object":"chat.completion.chunk","created":1689761646,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":"To"},"finish_reason":null}]}
  {"id":"chatcmpl-7dyU298SlZywQtM5bhjXMlPJvd3Bx","object":"chat.completion.chunk","created":1689761646,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":" multiply"},"finish_reason":null}]}
  {"id":"chatcmpl-7dyU298SlZywQtM5bhjXMlPJvd3Bx","object":"chat.completion.chunk","created":1689761646,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":" "},"finish_reason":null}]}
  {"id":"chatcmpl-7dyU298SlZywQtM5bhjXMlPJvd3Bx","object":"chat.completion.chunk","created":1689761646,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":"100"},"finish_reason":null}]}
  {"id":"chatcmpl-7dyU298SlZywQtM5bhjXMlPJvd3Bx","object":"chat.completion.chunk","created":1689761646,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":" by"},"finish_reason":null}]}
  {"id":"chatcmpl-7dyU298SlZywQtM5bhjXMlPJvd3Bx","object":"chat.completion.chunk","created":1689761646,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":" "},"finish_reason":null}]}
  {"id":"chatcmpl-7dyU298SlZywQtM5bhjXMlPJvd3Bx","object":"chat.completion.chunk","created":1689761646,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":"10"},"finish_reason":null}]}
  {"id":"chatcmpl-7dyU298SlZywQtM5bhjXMlPJvd3Bx","object":"chat.completion.chunk","created":1689761646,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":","},"finish_reason":null}]}
  {"id":"chatcmpl-7dyU298SlZywQtM5bhjXMlPJvd3Bx","object":"chat.completion.chunk","created":1689761646,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":" you"},"finish_reason":null}]}
  {"id":"chatcmpl-7dyU298SlZywQtM5bhjXMlPJvd3Bx","object":"chat.completion.chunk","created":1689761646,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":" simply"},"finish_reason":null}]}
  {"id":"chatcmpl-7dyU298SlZywQtM5bhjXMlPJvd3Bx","object":"chat.completion.chunk","created":1689761646,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":" multiply"},"finish_reason":null}]}
  {"id":"chatcmpl-7dyU298SlZywQtM5bhjXMlPJvd3Bx","object":"chat.completion.chunk","created":1689761646,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":" the"},"finish_reason":null}]}
  {"id":"chatcmpl-7dyU298SlZywQtM5bhjXMlPJvd3Bx","object":"chat.completion.chunk","created":1689761646,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":" two"},"finish_reason":null}]}
  {"id":"chatcmpl-7dyU298SlZywQtM5bhjXMlPJvd3Bx","object":"chat.completion.chunk","created":1689761646,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":" numbers"},"finish_reason":null}]}
  {"id":"chatcmpl-7dyU298SlZywQtM5bhjXMlPJvd3Bx","object":"chat.completion.chunk","created":1689761646,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":" together"},"finish_reason":null}]}
  {"id":"chatcmpl-7dyU298SlZywQtM5bhjXMlPJvd3Bx","object":"chat.completion.chunk","created":1689761646,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":":

"},"finish_reason":null}]}
  {"id":"chatcmpl-7dyU298SlZywQtM5bhjXMlPJvd3Bx","object":"chat.completion.chunk","created":1689761646,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":"100"},"finish_reason":null}]}
  {"id":"chatcmpl-7dyU298SlZywQtM5bhjXMlPJvd3Bx","object":"chat.completion.chunk","created":1689761646,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":" *"},"finish_reason":null}]}
  {"id":"chatcmpl-7dyU298SlZywQtM5bhjXMlPJvd3Bx","object":"chat.completion.chunk","created":1689761646,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":" "},"finish_reason":null}]}
  {"id":"chatcmpl-7dyU298SlZywQtM5bhjXMlPJvd3Bx","object":"chat.completion.chunk","created":1689761646,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":"10"},"finish_reason":null}]}
  {"id":"chatcmpl-7dyU298SlZywQtM5bhjXMlPJvd3Bx","object":"chat.completion.chunk","created":1689761646,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":" ="},"finish_reason":null}]}
  {"id":"chatcmpl-7dyU298SlZywQtM5bhjXMlPJvd3Bx","object":"chat.completion.chunk","created":1689761646,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":" "},"finish_reason":null}]}
  {"id":"chatcmpl-7dyU298SlZywQtM5bhjXMlPJvd3Bx","object":"chat.completion.chunk","created":1689761646,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":"100"},"finish_reason":null}]}
  {"id":"chatcmpl-7dyU298SlZywQtM5bhjXMlPJvd3Bx","object":"chat.completion.chunk","created":1689761646,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":[],"finish_reason":null}]}
[2023-07-19 10:14:08] local.INFO: streaming:  {"id":"chatcmpl-7dyU298SlZywQtM5bhjXMlPJvd3Bx","object":"chat.completion.chunk","created":1689761646,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":[],"finish_reason":"stop"}]}

As you can see the last chunk before finish should contain only on symbol 0.
I guess the problem is that PHP evaluates it as false that's why we have missing content.
Is last chunk consists of 00 or 000 everything is working fine
My code:
$client = OpenAI::client(config('openapi.key'));

            return response()->stream(function () use ($client, $request, $messages) {
                $stream = $client->chat()->createStreamed([
                    'model' => $request->input('model', 'gpt-3.5-turbo'),
                    'temperature' => 0.7,
                    'max_tokens' => $request->input('max_tokens', 512),
                    'top_p' => 1,
                    'messages' => $messages,
                    'stop' => $request->input('stop', null),
                    'n' => 1,
                    'user' => $request->input('user', ''),
                ]);

                foreach ($stream as $response) {
                    $text = json_encode($response->toArray());
                    \Log::info('streaming: ',  $response->toArray());
                    if (connection_aborted()) {
                        break;
                    }

                    echo "event: update\n";
                    echo 'data: ' . $text;
                    echo "\n\n";
                    ob_flush();
                    flush();
                }

                echo "event: update\n";
                echo 'data: <END_STREAMING_SSE>';
                echo "\n\n";
                ob_flush();
                flush();
            }, 200, [
                'Cache-Control' => 'no-cache',
                'X-Accel-Buffering' => 'no',
                'Content-Type' => 'text/event-stream',
            ]);

Are you on the latest version (0.6.3)? I believe this was fixed

I was on 0.4.1.
Thanks, update fix it.


Add yamllint configuration to extensions

Refer-to: Configuration — yamllint 1.11.1 documentation https://yamllint.readthedocs.io/en/stable/configuration.html
Signed-off-by: 林博仁(Buo-ren, Lin<EMAIL_ADDRESS>

Thanks!  This has been released as part of 1.0.14!


redshift service cannot access the display (with workaround)

Issue description
Redshift starts and immediately exits due to having insufficient access to X11. This is new as of:

https://github.com/NixOS/nixpkgs/issues/14623
https://github.com/NixOS/nixpkgs/commit/b6de72513f632bed4c16a15096a56868e711ffe4
https://github.com/NixOS/nixpkgs/commit/659c41a7d59d8354c58dccf5b97637b6a633aed9

Steps to reproduce

Enable Redshift

{
  services.redshift = {
    enable = true;
    latitude = "30.25";
    longitude = "-97.75";
    temperature.night = 1900;
  };
}

Start it ...

Apr 21 21:11:38 NdNdNx systemd[1]: Started Redshift colour temperature adjuster.
Apr 21 21:11:38 NdNdNx redshift[2584]: No protocol specified
Apr 21 21:11:38 NdNdNx redshift[2584]: `RANDR Query Version' returned error -1
Apr 21 21:11:38 NdNdNx redshift[2584]: Initialization of randr failed.
Apr 21 21:11:38 NdNdNx redshift[2584]: Trying next method...
Apr 21 21:11:38 NdNdNx redshift[2584]: No protocol specified
Apr 21 21:11:38 NdNdNx redshift[2584]: X request failed: XOpenDisplay
Apr 21 21:11:38 NdNdNx redshift[2584]: Initialization of vidmode failed.
Apr 21 21:11:38 NdNdNx redshift[2584]: Trying next method...
Apr 21 21:11:38 NdNdNx redshift[2584]: No more methods to try.
Apr 21 21:11:38 NdNdNx systemd[1]: redshift.service: Main process exited, code=exited, status=1/FAILURE
Apr 21 21:11:38 NdNdNx systemd[1]: redshift.service: Unit entered failed state.
Apr 21 21:11:38 NdNdNx systemd[1]: redshift.service: Failed with result 'exit-code'.

Workaround:
Override and specify an XAUTHORITY file it can use.
A more proper fix would be having redshift run as a user service, I believe. While we're at it, it probably shouldn't be running as root.
{
  systemd.services.redshift.environment = {
    XAUTHORITY = "/home/grahamc/.Xauthority"; # Use %h and a user-service somehow
  };
}

Technical details

System: 16.03.616.4ac4b35 (Emu)
Nix version: nix-env (Nix) 1.11.2
Nixpkgs version: "16.03.616.4ac4b35"

Related issue: https://github.com/NixOS/nixpkgs/pull/10018

Confirmed.
+1 for implementing Redshift.service as user.service.

Backported to 16.03 in 82ca7f75d6f239502fad7df553aea36339bae8da

Hi, thanks for the fix!
However when upgrading from system service to user service for Redshift, the system service seems to be still running and keeps spamming logs with the same error, while the user service also runs successfully.
I can't find a way to stop the system service since systemd don't "know" anymore about it (even though it's constantly trying to restart it).
I'd rather avoid rebooting the computer, but maybe it's the only solution?
Thanks!

I still get the redshift error messages repeating indefinitely in my journal, with only the redshift user service(s) running, on NixOS 16.09pre87431.3badad8 (Flounder).


Recover when connection cannot be established straight at startup

When the connection to the PostgreSQL instance cannot be established straight
at startup, a race condition can happen when autoDiscoverDatabases is true. If
discoverDatabaseDSNs fails, no dsn is set as the master database, and, if
scrapeDSN succeeds, checkMapVersions will have omitted the default metrics in
the server metric map. The metric map won't be updated unless the version
returned by the PostgreSQL instance changes. With this patch, scrapeDSN won't
be run unless discoverDatabaseDSNs succeeded and thus the race condition is
eliminated.

Coverage decreased (-0.1%) to 64.573% when pulling 91c51504554e5f36ca923cf9c8aa9a87a3501f02 on camptocamp:fix-race-condition into e2df41f43effdfbb4544b3efb70350380ad895fe on wrouesnel:master.


Data logging hangs on battery

Reported in Zoom chat by @microbit-giles and summarising here.
Using a program like this in MakeCode

Flash to micro:bit, detach USB cable and attach battery pack. The expectation is that the temperature is logged every minute. On battery this is not the case. It may be logged a few times and then program seems to hang.
The program works as expected when the micro:bit is plugged to a laptop via USB.
Using an Every block in MakeCode:
On usb:
Works for logging
Works for not-logging (ie, just displaying an image every minute)
On battery:
hangs when logging after few logs
Works for not-logging (ie, just displaying an image every minute)
Notes from Giles

Logging just temperature seems marginally more successful than logging temperature, sound and light (5 or so logs before hanging as opposed to 1 or 2) - unknown if that is caused by doing 3x the logging, or because it’s accessing other sensors / lighting the microphone LED.

It's possible that this is DAPLink putting the micro:bit to sleep when running on battery.
Next step is to reproduce in CODAL.

The batteries I used were box-fresh and reading 1.8v unloaded at the start of the experiment. I ran it for an hour logging temperature, light and sound once a minute using the every block and only 2 events were logged. The program had hung (did not respond to button presses - see code below) but microphone and power LED were still lit. Batteries subsequently depleted to just over 1.5v unloaded.
The same batteries seem to work ok for short burst logging of accelerometer data, eg every 100ms for less than 10 seconds.
Older batteries reading 1.3 - 1.5v unloaded also seem to cause hangs in various data logging projects I've tested.

Thanks @microbit-mark @microbit-giles
Sounds a lot like the KL27 is not waking up from the deep sleep mode it goes into when idle and running on battery power...
I'll try to create C++ repro to verify.

Also, for info: running this program just flashing a heart on the LED display every minute seems to work on the same batteries - only when data logging / sensor readings are involved does it seem to hang.

Hmm... Just ran a quick C++ test on battery, that logged accelerometer data every 10 seconds.
No hang. Logged 134 rows of data in about 21 minutes (which is when i stopped it.).
Source code below.
Three differences I'm aware of between this code and MakeCode example above:

Different logging delay (10 sec vs 60 sec). I'll try changing this next.
mirror to serial is disabled
we're using a loop, not a timed event

int main()
{
    // Initialise the micro:bit runtime.
    uBit.init();

    uBit.messageBus.listen(MICROBIT_ID_BUTTON_A, MICROBIT_BUTTON_EVT_CLICK, log_fs_write_test);
    uBit.messageBus.listen(MICROBIT_ID_BUTTON_B, MICROBIT_BUTTON_EVT_CLICK, log_fs_remount_test);
    uBit.messageBus.listen(MICROBIT_ID_BUTTON_AB, MICROBIT_BUTTON_EVT_CLICK, log_fs_format_test);
    uBit.messageBus.listen(MICROBIT_ID_LOG, MICROBIT_LOG_EVT_LOG_FULL, log_fs_full_test);

    DMESG("\nLOG_FS_TEST: READY\nBUTTON A:   APPEND\nBUTTON B:   REMOUNT\nBUTTON A+B: FORMAT\n");

    uBit.log.logData("X", NULL);
    uBit.log.logData("Y", NULL);
    uBit.log.logData("Z", NULL);

    uBit.log.setTimeStamp(TimeStampFormat::Seconds);
    uBit.log.setSerialMirroring(false);
    uBit.display.scroll("Log");

    int count = 0;
    while(1)
    {
        uBit.log.beginRow();
        uBit.log.logData("X", uBit.accelerometer.getX());
        uBit.log.logData("Y", uBit.accelerometer.getY());
        uBit.log.logData("Z", uBit.accelerometer.getZ());
        uBit.log.endRow();

        uBit.display.print(count);
        uBit.sleep(10000);

        if (uBit.log.isFull())
        {
            while (1)
            {
                uBit.display.scroll("LOG FULL");
                uBit.sleep(3000);
            }
        }

        count++;
        if (count == 10)
            count = 0;
    }
}

Thanks @finneyj - I'm also logging 3 different sensors, including one that lights an LED, so I wonder if that could be a factor either in the hanging or in power depletion leading to hanging? I'd also love to know if anyone can reproduce my findings in MakeCode just in case it's a one-off hardware issue - I'm not sure anyone else has tested it.

indeed - a 4th difference would be that @microbit-giles, thanks. I'll work through t'other ones first, as they seem more likely. The sensors are all nrf52 side, so not really any direct relationship to the logging.

Just for tracking, C++ also works fine at 60 second intervals on battery power. Tested for a little over 20 minutes.
int main()
{
    // Initialise the micro:bit runtime.
    uBit.init();

    uBit.messageBus.listen(MICROBIT_ID_BUTTON_A, MICROBIT_BUTTON_EVT_CLICK, log_fs_write_test);
    uBit.messageBus.listen(MICROBIT_ID_BUTTON_B, MICROBIT_BUTTON_EVT_CLICK, log_fs_remount_test);
    uBit.messageBus.listen(MICROBIT_ID_BUTTON_AB, MICROBIT_BUTTON_EVT_CLICK, log_fs_format_test);
    uBit.messageBus.listen(MICROBIT_ID_LOG, MICROBIT_LOG_EVT_LOG_FULL, log_fs_full_test);

    DMESG("\nLOG_FS_TEST: READY\nBUTTON A:   APPEND\nBUTTON B:   REMOUNT\nBUTTON A+B: FORMAT\n");

    uBit.log.logData("X", NULL);
    uBit.log.logData("Y", NULL);
    uBit.log.logData("Z", NULL);

    uBit.log.setTimeStamp(TimeStampFormat::Seconds);
    uBit.log.setSerialMirroring(false);
    uBit.display.scroll("Log");

    int count = 0;
    while(1)
    {
        uBit.log.beginRow();
        uBit.log.logData("X", uBit.accelerometer.getX());
        uBit.log.logData("Y", uBit.accelerometer.getY());
        uBit.log.logData("Z", uBit.accelerometer.getZ());
        uBit.log.endRow();

        uBit.display.print(count);
        uBit.sleep(60000);

        if (uBit.log.isFull())
        {
            while (1)
            {
                uBit.display.scroll("LOG FULL");
                uBit.sleep(3000);
            }
        }

        count++;
        if (count == 10)
            count = 0;
    }
}

Also worked fine when using a system_timer_event_every(). Pretty sure this is the same API behind the every block. Check out the consistency of those timestamp. yummy.
int count = 0;
void timed_log_event(Event)
{
    if (uBit.log.isFull())
    {
        while (1)
        {
            uBit.display.scroll("LOG FULL");
            uBit.sleep(3000);
        }
    }

    uBit.log.beginRow();
    uBit.log.logData("X", uBit.accelerometer.getX());
    uBit.log.logData("Y", uBit.accelerometer.getY());
    uBit.log.logData("Z", uBit.accelerometer.getZ());
    uBit.log.endRow();

    uBit.display.print(count);

    count++;

    if (count == 10)
        count = 0;
}

int main()
{
    // Initialise the micro:bit runtime.
    uBit.init();

    uBit.messageBus.listen(MICROBIT_ID_BUTTON_A, MICROBIT_BUTTON_EVT_CLICK, log_fs_write_test);
    uBit.messageBus.listen(MICROBIT_ID_BUTTON_B, MICROBIT_BUTTON_EVT_CLICK, log_fs_remount_test);
    uBit.messageBus.listen(MICROBIT_ID_BUTTON_AB, MICROBIT_BUTTON_EVT_CLICK, log_fs_format_test);
    uBit.messageBus.listen(MICROBIT_ID_LOG, MICROBIT_LOG_EVT_LOG_FULL, log_fs_full_test);

    DMESG("\nLOG_FS_TEST: READY\nBUTTON A:   APPEND\nBUTTON B:   REMOUNT\nBUTTON A+B: FORMAT\n");

    uBit.log.logData("X", NULL);
    uBit.log.logData("Y", NULL);
    uBit.log.logData("Z", NULL);

    uBit.log.setTimeStamp(TimeStampFormat::Seconds);
    uBit.log.setSerialMirroring(false);
    uBit.display.scroll("Log");

    uBit.messageBus.listen(0xCAFE, 0xCAFE, timed_log_event);
    system_timer_event_every(60000, 0xCAFE, 0xCAFE);

    while(1)
        uBit.sleep(100000);
}

...also fine with serial mirroring enabled.
@microbit-giles @microbit-mark were you testing this using WebUSB to program the micro:bit? If so, I'm wondering if this could be related to https://github.com/lancaster-university/codal-microbit-v2/issues/124...
int count = 0;
void timed_log_event(Event)
{
    if (uBit.log.isFull())
    {
        while (1)
        {
            uBit.display.scroll("LOG FULL");
            uBit.sleep(3000);
        }
    }

    uBit.log.beginRow();
    uBit.log.logData("X", uBit.accelerometer.getX());
    uBit.log.logData("Y", uBit.accelerometer.getY());
    uBit.log.logData("Z", uBit.accelerometer.getZ());
    uBit.log.endRow();

    uBit.display.print(count);

    count++;

    if (count == 10)
        count = 0;
}

int main()
{
    // Initialise the micro:bit runtime.
    uBit.init();

    uBit.messageBus.listen(MICROBIT_ID_BUTTON_A, MICROBIT_BUTTON_EVT_CLICK, log_fs_write_test);
    uBit.messageBus.listen(MICROBIT_ID_BUTTON_B, MICROBIT_BUTTON_EVT_CLICK, log_fs_remount_test);
    uBit.messageBus.listen(MICROBIT_ID_BUTTON_AB, MICROBIT_BUTTON_EVT_CLICK, log_fs_format_test);
    uBit.messageBus.listen(MICROBIT_ID_LOG, MICROBIT_LOG_EVT_LOG_FULL, log_fs_full_test);

    DMESG("\nLOG_FS_TEST: READY\nBUTTON A:   APPEND\nBUTTON B:   REMOUNT\nBUTTON A+B: FORMAT\n");

    uBit.log.logData("X", NULL);
    uBit.log.logData("Y", NULL);
    uBit.log.logData("Z", NULL);

    uBit.log.setTimeStamp(TimeStampFormat::Seconds);
    uBit.log.setSerialMirroring(true);
    uBit.display.scroll("Log");

    uBit.messageBus.listen(0xCAFE, 0xCAFE, timed_log_event);
    system_timer_event_every(60000, 0xCAFE, 0xCAFE);

    while(1)
        uBit.sleep(100000);
}

Also confirmed fine when logging temperature, as per @microbit-mark 's example at the top of this thread.
Sorry guys - I can't repro this using C++ and drag 'n' drop programming...
int count = 0;
void timed_log_event(Event)
{
    if (uBit.log.isFull())
    {
        while (1)
        {
            uBit.display.scroll("LOG FULL");
            uBit.sleep(3000);
        }
    }

    uBit.log.beginRow();
    uBit.log.logData("temp", uBit.thermometer.getTemperature());
    uBit.log.endRow();

    uBit.display.print(count);

    count++;

    if (count == 10)
        count = 0;
}

int main()
{
    // Initialise the micro:bit runtime.
    uBit.init();

    uBit.messageBus.listen(MICROBIT_ID_BUTTON_A, MICROBIT_BUTTON_EVT_CLICK, log_fs_write_test);
    uBit.messageBus.listen(MICROBIT_ID_BUTTON_B, MICROBIT_BUTTON_EVT_CLICK, log_fs_remount_test);
    uBit.messageBus.listen(MICROBIT_ID_BUTTON_AB, MICROBIT_BUTTON_EVT_CLICK, log_fs_format_test);
    uBit.messageBus.listen(MICROBIT_ID_LOG, MICROBIT_LOG_EVT_LOG_FULL, log_fs_full_test);

    DMESG("\nLOG_FS_TEST: READY\nBUTTON A:   APPEND\nBUTTON B:   REMOUNT\nBUTTON A+B: FORMAT\n");

    uBit.log.logData("temp", NULL);

    uBit.log.setTimeStamp(TimeStampFormat::Seconds);
    uBit.log.setSerialMirroring(true);
    uBit.display.scroll("Log");

    uBit.messageBus.listen(0xCAFE, 0xCAFE, timed_log_event);
    system_timer_event_every(60000, 0xCAFE, 0xCAFE);

    while(1)
        uBit.sleep(100000);
}

Yes, I used WebUSB to program the micro:bit. Is it worth me testing it after dragging and dropping the HEX file instead?
Also in case this is relevant, this is an 'RC' board, I had to update the firmware to get data logging to work.
# DAPLink Firmware - see https://mbed.com/daplink
Unique ID:<PHONE_NUMBER>624e450021601500000047000000009796990b
HIC ID: 9796990b
Auto Reset: 1
Automation allowed: 0
Overflow detection: 0
Incompatible image detection: 1
Page erasing: 0
Daplink Mode: Interface
Interface Version: 0255
Bootloader Version: 0254
Git SHA: 1436bdcc67029fdfc0ff03b73e12045bb6a9f272
Local Mods: 0
USB Interfaces: MSD, CDC, HID, WebUSB
Bootloader CRC: 0xdb256cca
Interface CRC: 0x5b5cc0f5
Remount count: 0
URL: https://microbit.org/device/?id=9904&v=0255

Here's mine. Looks like your bootloader is a little out of date, but the interface version is the same as mine (which is the bit that should matter). @microbit-carlos The bootloader version being a little old here should have no impact, right?
# DAPLink Firmware - see https://mbed.com/daplink
Unique ID:<PHONE_NUMBER>624e45002060190000000d000000009796990b
HIC ID: 9796990b
Auto Reset: 1
Automation allowed: 0
Overflow detection: 0
Incompatible image detection: 1
Page erasing: 0
Daplink Mode: Interface
Interface Version: 0255
Bootloader Version: 0255
Git SHA: 1436bdcc67029fdfc0ff03b73e12045bb6a9f272
Local Mods: 0
USB Interfaces: MSD, CDC, HID, WebUSB
Bootloader CRC: 0x828c6069
Interface CRC: 0x5b5cc0f5
Remount count: 0
URL: https://microbit.org/device/?id=9904&v=0255

@microbit-giles
Yes, it would be interesting to see what happens is you disable WebUSB and try drag 'n' drop programming from MakeCode. Ideally the simplest program you can.
Please report back what you find?

This program worked fine for me on https://makecode.microbit.org/beta, using drag 'n' drop programming. Tested for about 30 mins, logging every 30 seconds under battery power.

Also worked fine for me when re-flashing the same program using WebUSB.  (I just swapped the order of the icons around to ensure it was updated). Also ran for 20 mins.
Sorry guys... not much I can do without a repro.
Has anyone else reproduced this apart from on Giles' micro:bit?

p.s. exactly which release candidate version of the micro:bit are you running with @microbit-giles ?

@finneyj I have two sets of V2s, one set labelled 'RC' on a sticker on the speaker and another set without the sticker. (Confusingly I have removed some of the RC stickers for filming!). Both sets seem to be pre-main production as they all required a firmware update to get data-logging to work. Aside from providing the DETAILS.TXT file, how do I tell which version they are?

I transferred the same program I used before (screenshot above) by drag and drop (not WebUSB) and, on batteries, the program hung after recording 3 events.

@finneyj I'm currently testing a program identical to your last screenshot on the same micro:bit and battery pack, transferred by dragging and dropping the hex file. Will update here with what I find.

Update - @finneyj your program is still running fine for me after 1 hour and has logged data as expected. I'll try using the 'every' block instead of 'forever' next.
Can someone try my HEX file with batteries for 30 minutes or an hour?
21jun-long-term-logging.hex.zip

Can someone try my HEX file with batteries for 30 minutes or an hour?
21jun-long-term-logging.hex.zip. It doesn't flash the display when it logs but you'll know it's crashed if you can't stop and start it by pressing button A.

I'll load that in one of mine

After half an hour in an RC1 board on battery I've got 30 entries:

MY_DATA.HTM.zip
microbit.csv

@microbit-giles after a hang, could you copy the whole MY_DATA.HTM file and zip it up here? I wonder if there's corruption going on and if so whether we can see that in the flash contents.

@jaustin I'll put the original HEX file back, test again and do as you say.
@finneyj's program adapted to use 'every' instead of 'forever' and a pause is still working ok after one hour, so it's not the 'every' block that's to blame.

Okay, tried again and still unable to reproduce from my end. Steps I've followed:

Connect micro:bit to PC via USB cable, no battery pack attached
flash program hex file from https://github.com/lancaster-university/codal-microbit-v2/issues/130#issuecomment-866682115 via drag&drop
USB unplug
Plug battery pack
Press button A to start logging
Waited 2 hours, during this time I've press the button A twice (to disable and reenable logging, but mostly to print something on the display) at random times to confirm its still running
Connect USB cable to computer (battery pack still attached)
Extracted MY_DATA

microbit.csv
MY_DATA.HTM.zip
DETAILS.TXT# DAPLink Firmware - see https://mbed.com/daplink
Unique ID:<PHONE_NUMBER>624e45005160190000003b000000009796990b
HIC ID: 9796990b
Auto Reset: 1
Automation allowed: 0
Overflow detection: 0
Incompatible image detection: 1
Page erasing: 0
Daplink Mode: Interface
Interface Version: 0255
Bootloader Version: 0254
Git SHA: 1436bdcc67029fdfc0ff03b73e12045bb6a9f272
Local Mods: 0
USB Interfaces: MSD, CDC, HID, WebUSB
Bootloader CRC: 0xdb256cca
Interface CRC: 0x5b5cc0f5
Remount count: 0
URL: https://microbit.org/device/?id=9904&v=0255

I have had another freeze, this time using @finneyj's program modified to use 'every' and log temperature, light and sound every minute. The heart flashed many times but I noticed after ~2 hours that the heart was no longer flashing. Mic and power LEDs remain lit. Unplugged battery, reconnected by USB. I think only 2 events may have been logged on battery power after 2 hours:

So is it something to do with sound or light logging? That's the only thing I changed from the previous version that logged accelerometer data using the 'every' block and which works. Note that the manual logger that Anna and I also have freezing issues with also logs sound, light and temperature data.
microbit-test-for-joe-every-temp-light-sound.hex.zip
I can't compress the MY_DATA file for some reason but it's here: https://www.dropbox.com/s/q5nltz8kdd3ibk1/MY_DATA.HTM?dl=0

@microbit-giles just to confirm, in your last test from https://github.com/lancaster-university/codal-microbit-v2/issues/130#issuecomment-866813364, did you flash the hex via drag&drop?
And did you erase the previous logged data?

I've been tracking the thread and reproducing examples. The issue reproduces for me if I try to log temperature within a program Using and RC1 micro:bit that has been manually updated and a production release that had no data logged on it before testing.
DETAILS.TXT from production V2
# DAPLink Firmware - see https://mbed.com/daplink
Unique ID:<PHONE_NUMBER>434e45003e400e00000059000000009796990b
HIC ID: 9796990b
Auto Reset: 1
Automation allowed: 0
Overflow detection: 0
Incompatible image detection: 1
Page erasing: 0
Daplink Mode: Interface
Interface Version: 0255
Bootloader Version: 0255
Git SHA: 1436bdcc67029fdfc0ff03b73e12045bb6a9f272
Local Mods: 0
USB Interfaces: MSD, CDC, HID, WebUSB
Bootloader CRC: 0x828c6069
Interface CRC: 0x5b5cc0f5
Remount count: 0
URL: https://microbit.org/device/?id=9904&v=0255

Same results as Giles in https://github.com/lancaster-university/codal-microbit-v2/issues/130#issuecomment-866813364

Logging temp, light level, sound does not work (or anything with temp for me)
Logging x, y, z does work

Could this be to do with how the sensors are polled for updates when on battery? I have also tried initialising the temp as a variable in onStart but this does not resolve the issue.

@microbit-carlos My most recent test that failed (Joe's program modified to log temp, light, sound) was flashed by drag and drop. This program has no option to delete the previous log in software, but I have previously done so when testing the other more complex logging program.

Another test trying my original big HEX on a different V2 and different battery pack (reading 2.69v on load on pins). Flashed by drag and drop. This V2 has an 'RC1' sticker (the other did not have a sticker but I know it to be pre-production and may have had the RC1 sticker removed.)
This time logging temperature, light and sound over half an hour worked just fine.
# DAPLink Firmware - see https://mbed.com/daplink
Unique ID:<PHONE_NUMBER>624e45002a601500000034000000009796990b
HIC ID: 9796990b
Auto Reset: 1
Automation allowed: 0
Overflow detection: 0
Incompatible image detection: 1
Page erasing: 0
Daplink Mode: Interface
Interface Version: 0255
Bootloader Version: 0254
Git SHA: 1436bdcc67029fdfc0ff03b73e12045bb6a9f272
Local Mods: 0
USB Interfaces: MSD, CDC, HID, WebUSB
Bootloader CRC: 0xdb256cca
Interface CRC: 0x5b5cc0f5
Remount count: 0
URL: https://microbit.org/device/?id=9904&v=0255

Testing my 'faulty' micro:bit with Joe's program now just logging light and sound, it hung after around 5 minutes - seems it may have lasted a bit longer when only logging 2 items, maybe that suggests it's a power issue? And suggests it's not temperature (alone) that's the issue. This is like playing Cluedo...

Our testing today would also seem to confirm that not all micro:bits are affected, and not even all pre-production micro:bits.

I took sound out and only logged light on the micro:bit and battery pack I had problems with (code below) and it worked fine over 65 minutes, logged every minute as you'd expect and didn't hang. Will try logging sound only next.

Logging sound only, the troublesome micro:bit only logged 1 event on batteries. It had stopped flashing the heart after 5 minutes, possibly before. Program transferred, as before, by drag and drop.

Thanks for all the additional data folks. This is all very curious. We need to be careful not to jump to conclusions here - else we might end up chasing ghosts...
There's really very little difference in which sensor you choose to use. The microphone is much busier than the others, but the temperature sensor is very low overhead. I normally use accelerometer for these tests as it is off chip, and shares resources with the USB interface... hence is normally the most problematic.
@microbit-giles @microbit-mark : A couple of things.

what computer/operating system are you using?
can you please share some of the MY_DATA.HTM files that are fail cases? Maybe I can see something in there that will give us a hint as to what's going on here... Whatever it is, it feels like it's being caused by something we haven't considered yet.

Long programme, flashed via WebUSB
Tried again with the same steps and micro:bit (ID 428-2033) as listed here: https://github.com/lancaster-university/codal-microbit-v2/issues/130#issuecomment-866806597
The only difference was that I used WebUSB instead of drag&drop, and it worked fine for almost 2h.
Voltage on the edge connector at the end of the run was 2.79V.

MY_DATA.HTM.zip
DETAILS.TXT

✅ Works correctly
Long programme, slightly lower voltage batteries
Then I tried the same board with other batteries, they are just slightly more discharged, but were the oldest AAA batteries I could find.

Same micro:bit (ID 428-2033), untouched since the steps listed above (WebUSB flashed from the previous run)
Plugged the battery holder with different batteries
Measured 2.67V in the edge connector
Pressed A+B to erase the previous log
Pressed A to start logging
Waited over 1h
At this point the board is still responsive to pressing the button A to enable/disable logging
Unplugged battery pack
Connect USB cable to PC and extracted MY_DATA
Disconnected USB and connected battery to measure edge connector voltage without microphone LED: 2.62V

Files:

DETAILS.TXT
MY_DATA.HTM.zip

✅ Works correctly
Long programme, different micro:bit
Also tried with a different

Connected a different micro:bit (1.47.2 ID 170-2028) to PC via USB cable, no battery pack attached
Drag& drop flash the hex file from https://github.com/lancaster-university/codal-microbit-v2/issues/130#issuecomment-866682115
USB unplug
Plug battery pack
Read 2.77V from the edge connector
Pressed A+B to erase data from previous runs
Press button A to start logging
Waited for almost 2 hours ( 15.50 )
At this point the board is still responsive to pressing the button A to enable/disable logging
Unplugged battery pack
Connect USB cable to PC and  extracted MY_DATA
Disconnected USB and connected battery to measure edge connector voltage without microphone LED: 2.76V

Files:

DETAILS.TXT (flashed with factory image, so bootloader and interface at version 0255)
MY_DATA.HTM.zip

✅ Works correctly

Haven't been able to replicate yet

@finneyj Indeed I'm starting to think something totally random is going on and I'm ascribing causes that don't exist. Logging just temperature now for me, for example, is working fine.
I had to unlock the copied, failed MY_DATA.HTM file in order to ZIP it, here it is:
MY_DATA.HTM.zip
I'm using a MacBook Air, MacOS Big Sur 11.4
Hardware Overview:
Model Name:	MacBook Air
Model Identifier:	MacBookAir10,1
Chip:	Apple M1
Total Number of Cores:	8 (4 performance and 4 efficiency)
Memory:	16 GB
System Firmware Version:	6723.120.36
OS Loader Version:	6723.120.36
Serial Number (system):	C02FH013Q6LX
Hardware UUID:	BFA07E1E-5EAF-59EB-96F8-3FC495BCE3D2
Provisioning UDID:	00008103-001214A114F1001E
Activation Lock Status:	Enabled

Uh, hold on!
When I extracted the HTML files from my previous message i did it by copying the file into my PC and then opening my local version. This works fine for the first micro:bit (bootloader 0254, interface 0255), however the second one (1.47.2 ID 170-2028, factory DAPLink w/ bootloader 0255, interface 0255 ) reboots when I open the HTML in Chrome directly from the MICROBIT drive and I get this:

Saving the page into disk I get this:

micro_bit data log.zip

Opening the MICROBIT drive file from a text editor and it also reboots. This is what the text editor saves:

MY_DATA.txt

The crash also produces this assert.txt:
Assert
File: .\source\daplink\HardFault_Handler.c
Line: 74
Source: Application
Hexdumps
fffffffd
200024c0
20002bb8
00000000
200024e0
0000853b
000082da
21000000

@microbit-mark @microbit-giles I assume you (or Anna) have not seen this problem in your micro:bits?

@microbit-carlos No I have not seen empty data logs in MY_DATA.HTM, not have I seen ASSERT.TXT files on the micro:bit.
I re-tested the program simple program that logs temperature, light and sound every minute (microbit-test-for-joe-every-temp-light-sound.hex) over 2 hours and again it only logged two events at about 1 and 2 minutes and hung - so I do seem to be able consistently reproduce faults on this micro:bit, even if they seem to make little sense.

@microbit-carlos I've seen an empty log file when deleting the log as part of my program. eg
1. flash program
2. look at data in log file
3. press eg. A to fast delete
4. look at data log and see empty log that appears in offline mode
5. disconnect and reconnect resolves this

Tested data logging with the accelerometer on battery by adapting the short reproducer  in https://github.com/lancaster-university/codal-microbit-v2/issues/130#issue-927224156 and it does not reproduce
Works on battery

Fails on battery after a couple of readings

@microbit-mark are you using the 60000ms delay not the 5000 one in my "battery-reproduce-test-v1.hex" ? Also no screen blink?

For me, using the attached hex on battery, the heart stops 'beating' after some time (order 2 mins). If I 'reset' the micro:bit, it beats again for a while.
battery-reproduce-test-v1.hex.zip

@finneyj could you build us the closest possible analog to this in C++ please?.
Handily, we can look at my CSV to see how long it lasted each time before stopping logging and then me resetting it
microbit (6).csv

As discussed in meeting. Tested hex in https://github.com/lancaster-university/codal-microbit-v2/issues/130#issuecomment-867543141 and this reproduces issue.
Added in event handler for on long full and still reproduces

microbit-repro-logFullHandler.hex.zip
When disconnected USB, The program will log one entry on battery before stopping (no heart on screen).
MY_DATA.HTM.zip
Pressing reset will log two entries before stopping (heart appears twice on screen)

Okay, running the attached hex I get a fairly reliable crash on my micro:bit.

battery-reproduce-test-v1.hex 2.zip
Attached are the files on the device after plugging in.
datalog-post-crash.zip
What I did:

Flash the hex file over USB with drag-and-drop
Start logging and then unplug USB
Log on battery until the heart stops flashing
Replace the USB cable, reset the micro:bit (heart starts flashing)
ONCE only: leave for a long time on USB - no crash in 1900seconds.
Reset on USB, remove USB so logging on battery. Wait for crash
Return to 6

So, doing this, on battery I got crashes after 178, 325, 67, 230, 117 seconds
My battery voltage when measured by a V1 using the battery test hex 2.58V

This hex never stops beating for me... I've had it running for about 30 mins on battery power now.  :-/
one possible commonality... all fail cases reported thus far have been using a mac, right?

The assert.txt in your postcrash.zip is ominous. That's coming from daplink of course...
Could it somehow be possible that there's some residual state that's different between mac and PC, such that daplink behaves differently when on battery somehow? Just reaching at straws here though.

Have you got other micro:bit board to test this @finneyj I have one board that refuses to replicate, but a different one has the issue.

@microbit-mark could you try to use a usb power bank with one of the boards that easily reproduces this issue? Ideally with Jonny's hex to keep using the same one. I'll try this as well.

FYI, I'm now on my 5th micro:bit, and still can't replicate this. One was a 1.47.2 (but with up to date KL27 firmware), the other four were totally unused V2.00 boards straight out of the box.
The last one I set off last night around 21:00. it's still running now at 15:45 the next day...
bah!

p.s. the above tests were running the HEX file Jonny shared in battery-reproduce-test-v1.hex 2.zip above.

I've got two boards that can replicate this on battery and I can confirm that they also freeze as well from a USB power bank.
So that discards this being an issue at specific voltage levels.
To be fair Giles was also able to replicate with full batteries and with other batteries at lower voltages, but always good to get more data-points.

@microbit-carlos
That's mighty curious actually. In your test above, did you plug that USB power bank into the micro:bit USB port? I guess so?
In which case, we have multiple reports (I believe) that cite that the logging works OK under USB power, and only fails on battery.  So why does it also fail on a USB powerbank?

In your test above, did you plug that USB power bank into the micro:bit USB port? I guess so?

Yes

In which case, we have multiple reports (I believe) that cite that the logging works OK under USB power, and only fails on battery. So why does it also fail on a USB powerbank?

Right, on a USB power bank DAPLink will go into the same "mode" as in battery, as it hasn't been able to establish a USB connection.
The interesting part is that I have two boards with exactly the nRF hex (Jonny's hex) and the same DAPLink version (the final 0255 factory image), and one never triggers the issue but the other one does.

Thanks @microbit-carlos  - that explains that part at least!
As for the rest... I'm beginning to think that we've somehow managed to quantum entangle v2 micro:bits in the factory.

Well to make it even more weird, I've found another board that triggers this issue, and in this case this board is from the same batch as the one that never triggers it. Both are RC1 boards built 12 panels apart.

On the assumption that this is somehow related to the NRF52->KL27 I2C comms, I've made a few minor alterations to the I2C transaction code in the latest release: https://github.com/lancaster-university/codal-microbit-v2/commit/d558e51f9c9664c1a2d3f342721cbde468dfcc30
This should make the micro:bit a bit more tolerant of the KL27 being busy for a very long time, and also parameterize the retransmission counters, so that we can tune this further in the future if we need to via the MakeCode extension...
Once v0.2.30 goes live on MakeCode, it would be interesting to see if this makes any difference at all to this failure case... so if anyone has the chance to retest at that point, I'd appreciate it.

I recompiled my original problematic project this morning and it still hangs (on the same micro:bit and battery pack as before).

Looking at the interrupt signal in a oscilloscope:

An interrupt is triggered on every "log data" block, but when a board hangs I don't see any activity in the interrupt signal any more.

Thanks @microbit-carlos
That would be consistent. Does it stop in an active or passive state?

Ah, sorry. HIGH is passive... (inactive)

OK folks - I finally managed to reproduce this bug. Yay.
It seems to only affect some micro:bits. All five of my v2.00 RC boards do not reproduce, but a recent one I received off a test production line does (#17 from Pauline). So I could finally get a debugger on this thing and find out what was happening. Here's a stack trace when the device froze:

From stack frame 3 it's obvious that we're getting stuck inside the I2C transaction with the KL27 interface chip that's is trying to write the log data. On closer inspection of the I2C peripheral registers, it's also clear that the I2C bus has locked up. The transaction hasn't failed, nor has it completes, nor is it making any progress either...
This point squarely at the KL27 silicon errata that states that if the KL27 is in deep sleep (which it likely will be on battery power),  the first I2C transaction to it will be lost. However, "lost" seems a bit more invasive to the initiator of the transaction than we thought...
I've updated the CODAL code to always send a zero length write operation to the KL27 before every transaction, and that seems to clear up the issue on my board (rather than reply on the higher level NOP/retry functionality we had before). I've also added a ~1 second timeout to any I2C transaction, so that we at least recover if there are some cases where this isn't enough.
Here's C++ program using these changes that:

scrolls "log" on the screen
blinks a pixel every second
logs temperature every 5 seconds

If folks with a micro:bit that reproduces this bug can test with this HEX and report back, that would be awesome. @jaustin @microbit-carlos @microbit-mark @microbit-giles.
MICROBIT_I2C_FIX1.zip
The mystery still remains as to why some micro:bits exhibit this behaviour and others don't... I'm beginning to wonder if there's a feature on some micro:bit devices that is giving the KL27 insomnia...

@finneyj Your new HEX file has been running ok on my troublesome micro:bit for 45 minutes

Tested 3 boards that normally hang and after 1h30min they are still going 🎉

Thanks folks! Looking good!
Here are some equivalent build to test accelerometer, light, sound and temperature, as requested @microbit-giles
MICROBIT_I2C_FIX_TESTS.zip

...also if you want to go all out, here's a HEX that logs all of that at the same time.
MICROBIT_I2C_FIX_MEGATEST.zip

The sound one has been running ok for me for over an hour now.

The megatest has been running in my 3 boards for almost an hour now, looks good 👍

Thanks dudes. Well that was an exciting one!
Fix applied here:
https://github.com/lancaster-university/codal-microbit-v2/commit/8c269caf5455705610462ece2f50f77305711f00

Awesome detective work @finneyj and all! What a deeply strange bug.

Elementary my dear Giles.

FYI this patch is now live on MakeCode /beta


Remove deprecated ocio_config ImageIOConfigModel settings

Changelog Description
Remove deprecated ocio_config ImageIOConfigModel settings
Additional info
With the support of profiles to determine OCIO config in ayon-core this logic has been superseded and should be removed from each addon still having the deprecated settings.

@jakubjezek001 did you do a test run or just check the code? Do we need a test run?


Initial step to allow for connection parameters configuration in Linux

The L2capBle part to implement connection parameters configration, this is related to #83
Regards

Closing this based on the discussion from #83.


KOGITO-8740: data-index documentation

jira: https://issues.redhat.com/browse/KOGITO-8740

How to backport a pull request to a different branch?

In order to automatically create a backporting pull request please add one or more labels having the following format backport-<branch-name>, where <branch-name> is the name of the branch where the pull request must be backported to (e.g., backport-7.67.x to backport the original PR to the 7.67.x branch).

NOTE: backporting is an action aiming to move a change (usually a commit) from a branch (usually the main one) to another one, which is generally referring to a still maintained release branch. Keeping it simple: it is about to move a specific change or a set of them from one branch to another.

Once the original pull request is successfully merged, the automated action will create one backporting pull request per each label (with the previous format) that has been added.
If something goes wrong, the author will be notified and at this point a manual backporting is needed.

NOTE: this automated backporting is triggered whenever a pull request on main branch is labeled or closed, but both conditions must be satisfied to get the new PR created.

@cristianonicolai, @kaldesai  thanks for the review, suggestions applied!

@ricardozanini ready for review, Could you please take a look? thanks!

@radtriste Thanks for the review and the really useful comments. Could you take a look on the pending conversations and let me know what do you think? Thanks again!

looks like netlify did not redeploy with latest changes ...

@radtriste suggestions applied, Thanks for all!

Guys, I think this is ready to merge. Is this PR forgotten, or we need more approvals?


I can't run flutter app by flutter run command

Steps to Reproduce

cant run app by flutter run

Logs

Launching lib\main.dart on LG Q710AL in debug mode...
Running Gradle task 'assembleDebug'...

FAILURE: Build failed with an exception.

* What went wrong:
A problem occurred configuring root project 'android'.
> Could not resolve all artifacts for configuration ':classpath'.
   > Could not download builder.jar (com.android.tools.build:builder:3.5.0)
      > Could not get resource 'https://dl.google.com/dl/android/maven2/com/android/tools/build/builder/3.5.0/builder-3.5.0.jar'.
         > Could not GET 'https://dl.google.com/dl/android/maven2/com/android/tools/build/builder/3.5.0/builder-3.5.0.jar'.
            > dl.google.com
   > Could not download bundletool.jar (com.android.tools.build:bundletool:0.9.0)
      > Could not get resource 'https://dl.google.com/dl/android/maven2/com/android/tools/build/bundletool/0.9.0/bundletool-0.9.0.jar'.
         > Could not GET 'https://dl.google.com/dl/android/maven2/com/android/tools/build/bundletool/0.9.0/bundletool-0.9.0.jar'.
            > dl.google.com
   > Could not download protos.jar (com.android.tools.analytics-library:protos:26.5.0)
      > Could not get resource 'https://dl.google.com/dl/android/maven2/com/android/tools/analytics-library/protos/26.5.0/protos-26.5.0.jar'.
         > Could not GET 'https://dl.google.com/dl/android/maven2/com/android/tools/analytics-library/protos/26.5.0/protos-26.5.0.jar'.
            > dl.google.com
   > Could not download kotlin-compiler-embeddable.jar (org.jetbrains.kotlin:kotlin-compiler-embeddable:1.3.50)
      > Could not get resource 'https://jcenter.bintray.com/org/jetbrains/kotlin/kotlin-compiler-embeddable/1.3.50/kotlin-compiler-embeddable-1.3.50.jar'.
         > Could not GET 'https://jcenter.bintray.com/org/jetbrains/kotlin/kotlin-compiler-embeddable/1.3.50/kotlin-compiler-embeddable-1.3.50.jar'.
            > jcenter.bintray.com
   > Could not download fastutil.jar (it.unimi.dsi:fastutil:7.2.0)
      > Could not get resource 'https://jcenter.bintray.com/it/unimi/dsi/fastutil/7.2.0/fastutil-7.2.0.jar'.
         > Could not GET 'https://jcenter.bintray.com/it/unimi/dsi/fastutil/7.2.0/fastutil-7.2.0.jar'.
            > jcenter.bintray.com
   > Could not download guava.jar (com.google.guava:guava:27.0.1-jre)
      > Could not get resource 'https://jcenter.bintray.com/com/google/guava/guava/27.0.1-jre/guava-27.0.1-jre.jar'.
         > Could not GET 'https://jcenter.bintray.com/com/google/guava/guava/27.0.1-jre/guava-27.0.1-jre.jar'.
            > jcenter.bintray.com
   > Could not download httpclient.jar (org.apache.httpcomponents:httpclient:4.5.6)
      > Could not get resource 'https://jcenter.bintray.com/org/apache/httpcomponents/httpclient/4.5.6/httpclient-4.5.6.jar'.
         > Could not GET 'https://jcenter.bintray.com/org/apache/httpcomponents/httpclient/4.5.6/httpclient-4.5.6.jar'.
            > jcenter.bintray.com
   > Could not download httpcore.jar (org.apache.httpcomponents:httpcore:4.4.10)
      > Could not get resource 'https://jcenter.bintray.com/org/apache/httpcomponents/httpcore/4.4.10/httpcore-4.4.10.jar'.
         > Could not GET 'https://jcenter.bintray.com/org/apache/httpcomponents/httpcore/4.4.10/httpcore-4.4.10.jar'.
            > jcenter.bintray.com
   > Could not download antlr4.jar (org.antlr:antlr4:4.5.3)
      > Could not get resource 'https://jcenter.bintray.com/org/antlr/antlr4/4.5.3/antlr4-4.5.3.jar'.
         > Could not GET 'https://jcenter.bintray.com/org/antlr/antlr4/4.5.3/antlr4-4.5.3.jar'.
            > jcenter.bintray.com
   > Could not download jdom2.jar (org.jdom:jdom2:2.0.6)
      > Could not get resource 'https://jcenter.bintray.com/org/jdom/jdom2/2.0.6/jdom2-2.0.6.jar'.
         > Could not GET 'https://jcenter.bintray.com/org/jdom/jdom2/2.0.6/jdom2-2.0.6.jar'.
            > jcenter.bintray.com
   > Could not download FastInfoset.jar (com.sun.xml.fastinfoset:FastInfoset:1.2.13)
      > Could not get resource 'https://jcenter.bintray.com/com/sun/xml/fastinfoset/FastInfoset/1.2.13/FastInfoset-1.2.13.jar'.
         > Could not GET 'https://jcenter.bintray.com/com/sun/xml/fastinfoset/FastInfoset/1.2.13/FastInfoset-1.2.13.jar'.
            > jcenter.bintray.com

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

Do I can download them manually?

Hi @muath-ye
can you download any of them from browser?
could you provide your flutter doctor -v?

Are you sure you have internet connection ?

Hi @muath-ye
Without additional information, we are unfortunately not sure how to resolve this issue. We are
therefore reluctantly going to close this bug for now. Please don't hesitate to comment on the
bug if you have any more information for us; we will reopen it right away!
Thanks for your contribution

Hi @muath-ye
can you download any of them from browser?
could you provide your flutter doctor -v?

C:\WINDOWS\system32>flutter doctor -v
[√] Flutter (Channel stable, v1.17.3, on Microsoft Windows [Version 10.0.18363.535], locale ru-RU)
• Flutter version 1.17.3 at C:\dev\flutter
• Framework revision b041144f83 (12 days ago), 2020-06-04 09:26:11 -0700
• Engine revision ee76268252
• Dart version 2.8.4
[√] Android toolchain - develop for Android devices (Android SDK version 30.0.0)
• Android SDK at C:\dev\android\sdk
• Platform android-30, build-tools 30.0.0
• ANDROID_SDK_ROOT = C:\dev\android\sdk
• Java binary at: C:\Program Files\Android\Android Studio\jre\bin\java
• Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)
• All Android licenses accepted.
[√] Android Studio (version 4.0)
• Android Studio at C:\Program Files\Android\Android Studio
• Flutter plugin version 46.0.2
• Dart plugin version 193.7361
• Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)
[√] VS Code, 64-bit edition (version 1.45.1)
• VS Code at C:\Program Files\Microsoft VS Code
• Flutter extension version 3.11.0
[√] Connected device (1 available)
• sdk gphone x86 arm • emulator-5554 • android-x86 • Android 11 (API 30) (emulator)
• No issues found!

@smagul
does it persist and this is not simply network connection issue?
I'm not sure that being behind a firewall is your case, but you can try to use mirror site https://flutter.dev/community/china#configuring-flutter-to-use-a-mirror-site

I had this issue..It is most probably because of slow internet speed. And be patient, flutter downloads a lot - after all is done - your build will succeed!!

I solved my problem with re-running the command every time it fails because the command resume downloading necessary files.


cleanup

Annotate CloudFunction so it's clear it should only apply to Functions
Add first, basic test of generator

This could/should be landed as two separate commits – please!


[nextcloud] Can't download any files

After updating today i wasn't able to Download any files. Neither in the Webinterface, nor in the app.
Also the background image of the login-page doesn't load anymore (even after uploading a new one).
This is what I'm getting out of the nextcloud.log-file:
{"reqId":"SBQAynwuQo1RZb0caPDm","remoteAddr":"<IP_ADDRESS>","app":"PHP","message":"Redis::connect(): connect() failed: No such file or directory at \/nextcloud\/lib\/private\/RedisFactory.php#61","level":3,"time":"2017-02-25 01:39:25","method":"GET","url":"\/remote.php\/webdav\/Screenshot_1481220660.png","user":"test","version":"<IP_ADDRESS>"} {"reqId":"SBQAynwuQo1RZb0caPDm","remoteAddr":"<IP_ADDRESS>","app":"webdav","message":"Exception: {\"Message\":\"Redis server went away\",\"Exception\":\"RedisException\",\"Code\":0,\"Trace\":\"#0 \\\/nextcloud\\\/lib\\\/private\\\/Memcache\\\/Redis.php(114): Redis->incrBy('c803362cd3b66bf...', 1)\\n#1 \\\/nextcloud\\\/lib\\\/private\\\/Lock\\\/MemcacheLockingProvider.php(73): OC\\\\Memcache\\\\Redis->inc('files\\\/97d76eed5...')\\n#2 \\\/nextcloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/Common.php(680): OC\\\\Lock\\\\MemcacheLockingProvider->acquireLock('files\\\/97d76eed5...', 1)\\n#3 \\\/nextcloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/Wrapper\\\/Wrapper.php(588): OC\\\\Files\\\\Storage\\\\Common->acquireLock('files\\\/Screensho...', 1, Object(OC\\\\Lock\\\\MemcacheLockingProvider))\\n#4 \\\/nextcloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/Wrapper\\\/Wrapper.php(588): OC\\\\Files\\\\Storage\\\\Wrapper\\\\Wrapper->acquireLock('files\\\/Screensho...', 1, Object(OC\\\\Lock\\\\MemcacheLockingProvider))\\n#5 \\\/nextcloud\\\/lib\\\/private\\\/Files\\\/View.php(1894): OC\\\\Files\\\\Storage\\\\Wrapper\\\\Wrapper->acquireLock('files\\\/Screensho...', 1, Object(OC\\\\Lock\\\\MemcacheLockingProvider))\\n#6 \\\/nextcloud\\\/lib\\\/private\\\/Files\\\/View.php(1997): OC\\\\Files\\\\View->lockPath('\\\/Screenshot_148...', 1, false)\\n#7 \\\/nextcloud\\\/lib\\\/private\\\/Files\\\/View.php(1110): OC\\\\Files\\\\View->lockFile('\\\/Screenshot_148...', 1)\\n#8 \\\/nextcloud\\\/lib\\\/private\\\/Files\\\/View.php(967): OC\\\\Files\\\\View->basicOperation('fopen', '\\\/Screenshot_148...', Array, 'rb')\\n#9 \\\/nextcloud\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/File.php(305): OC\\\\Files\\\\View->fopen('Screenshot_1481...', 'rb')\\n#10 \\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/CorePlugin.php(85): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->get()\\n#11 [internal function]: Sabre\\\\DAV\\\\CorePlugin->httpGet(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#12 \\\/nextcloud\\\/3rdparty\\\/sabre\\\/event\\\/lib\\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\\n#13 \\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(479): Sabre\\\\Event\\\\EventEmitter->emit('method:GET', Array)\\n#14 \\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(254): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#15 \\\/nextcloud\\\/apps\\\/dav\\\/appinfo\\\/v1\\\/webdav.php(60): Sabre\\\\DAV\\\\Server->exec()\\n#16 \\\/nextcloud\\\/remote.php(165): require_once('\\\/nextcloud\\\/apps...')\\n#17 {main}\",\"File\":\"\\\/nextcloud\\\/lib\\\/private\\\/Memcache\\\/Redis.php\",\"Line\":114,\"User\":\"test\"}","level":4,"time":"2017-02-25 01:39:25","method":"GET","url":"\/remote.php\/webdav\/Screenshot_1481220660.png","user":"test","version":"<IP_ADDRESS>"}
Anyone else got the same problem?
Any solutions?

Redis server is not included anymore in the container (it shouldn't be), that's why you have this problem after the upgrade. The PHP redis extension is stil provided, you can either remove redis configuration from config.php or use redis server in a container : https://github.com/Wonderfall/dockerfiles/blob/master/nextcloud/README.md#how-to-configure-redis

Thx very much, have set up a redis server and it's working like a charm again :+1:


Fix/form data in response

What is the purpose of this pull request?

[ ] Documentation update
[x] Bug fix
[ ] New feature
[ ] Other, please explain:

What changes did you make? (provide an overview)

Added a test case illustrating an issue
Addressed a bug caused by missed function argument in Response implementation.
Addressed issue with form-data body where boundary fragments are emitted as strings while content fragments as binary data. Which caused exceptions.

Which issue (if any) does this pull request address?
See test case illustrating the issue. It throws exception without this fix.
Is there anything you'd like reviewers to know?

I had to workaround code coverage problem which reported issue for the line https://coveralls.io/builds/38068042/source?filename=src%2Fbody.js#L350 which is unreachable because FormData body is normalized into a stream and I don't believe there are no code paths in which request/response .body could be FormData.
https://github.com/node-fetch/node-fetch/blob/1780f5ae89107ded4f232f43219ab0e548b0647c/src/body.js#L53-L56
I do not believe changes here affected code coverage, however instead of removing those lines (which break no tests) I've added a assertion to exercise that code path.

Normalizing FormData to stream during body initialization has a problem of making content length undetectable:
https://github.com/node-fetch/node-fetch/blob/1780f5ae89107ded4f232f43219ab0e548b0647c/src/request.js#L172
And this code path will never be called
https://github.com/node-fetch/node-fetch/blob/1780f5ae89107ded4f232f43219ab0e548b0647c/src/body.js#L352-L355
But I'll defer this to another issue

(Last) commit dc5be05 adds bunch of type annotations which helped me to reason through by getting better IntelliSense as opposed to remembering all the ins and outs. Feel free to get rid of it if you don't want them.

superseded by #1314

#1314 should be landed soon which fixes this, thank you for taking the time with this ☺