Return created/updated objects

WHY are these changes introduced?
Currently, we can't fetch responses from  REST API. This has a lot of problems when a user wants to do something with the created object.
WHAT is this pull request doing?

Return response data array
Store updated object when save(true) or saveAndUpdate()

Type of change

[ ] Patch: Bug (non-breaking change which fixes an issue)
[x] Minor: New feature (non-breaking change which adds functionality)
[ ] Major: Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist

[x] I have added a changelog entry, prefixed by the type of change noted above
[x] I have added/updated tests for this change
[x] I have updated the documentation for public APIs from the library (if applicable)

I have signed CLA with my company


Added Shivana

metadata.json

Hi, this is not the appropriate place to submit a server - you must open a Pull Request.


Conversion of 40M poly model results in 0 poly due to absolute path bug

Are there any details on how to export a model and textures from Unity to be compatible with ARR?
I exported a model from Unity with embedded materials and textures because the instructions are unclear if the localAssetDirectoryPath can have subfolders (say for materials and textures).  The upload and convert succeed but the stdout.txt shows many errors for these embedded resources expecting to find the full path on the Azure servers:
[2020-04-10 - 23:12:55.446 (UTC)] Warning: Unable to resolve path to the texture "G:\source\Unity\Tests\azure-remote-rendering\Unity\Quickstart\Assets\Scenes\MountainTextures\Tile_1_0.jpg" in material "Tile_1_0"
[2020-04-10 - 23:12:55.446 (UTC)] Warning: Unable to find image file 'G:\source\Unity\Tests\azure-remote-rendering\Unity\Quickstart\Assets\Scenes\MountainTextures\Tile_1_0.jpg'
I would assume Azure would have to use the RelativeFilename path to each resource after it uploads everything so it can find the files in its blob container and ignore the absolute paths that are only relevant to the source machine.
I put the resulting .arrAsset and SAS URI into the Quickstart demo anyway and it runs but no model shows and it reports 0 polygons drawn on the status panel.  Anyone have advice?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: dc7693bc-840f-4744-bc28-884ed562b29f
Version Independent ID: 39dbf49c-cd15-2e63-a625-e5a40d0b6eb9
Content: Convert a model - Azure Remote Rendering
Content Source: articles/remote-rendering/quickstarts/convert-model.md
Service: azure-remote-rendering
GitHub Login: @FlorianBorn71
Microsoft Alias: flborn

Thanks for the ask  . At this time we are reviewing it and we will reply back as soon as we can .

ok I found that all textures need to be in the same folder as the fbx.  Now the stdout has no errors.  However the model still does not render in the Quickstart scene even after loading reaches 100%.  The Connection Status is Connected.
Wondering if the ARR system has some unpublished size limits, I also tried again with a small 50K poly model with the same result.  No errors I can see, but not rendering.

Hi @cevanno, do I understand your initial comment correctly that the render statistics indicate that no polygons are rendered? I assume the built-in engine model renders fine so the video stream etc. is working properly?
What can also be a problem is that the model is outside the view frustum, because it has been exported with far off-center pivot. To rule out that issue, can you try exporting with the "recenterToOrigin" option described here: https://docs.microsoft.com/en-us/azure/remote-rendering/how-tos/conversion/configure-model-conversion
If it is a problem with the model itself (e.g. for some reason it is exported as empty mesh) we should have a look at it here. Would it be possible to provide us with a small sample source model, assuming you may disclose it to us?

Here is what you can do without re-exporting: You can have a look at the bounding box extent of an exported model. This is described here:
https://docs.microsoft.com/en-us/azure/remote-rendering/how-tos/conversion/get-information
(The outputStatistics section -> boundingBox entry).
You may notice that bounding box values (min/max position) are not centered around the origin, but far off in one direction. In that case, you should re-export with "recenterToOrigin"

Thanks for the tip.  I am looking at this now.

From: FlorianBorn71<EMAIL_ADDRESS>Sent: Tuesday, April 14, 2020 12:52 PM
To: MicrosoftDocs/azure-docs<EMAIL_ADDRESS>Cc: cevanno<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/azure-docs] Conversion of 40M poly model results in 0 poly due to absolute path bug (#52193)
Here is what you can do without re-exporting: You can have a look at the bounding box extent of an exported model. This is described here:
https://docs.microsoft.com/en-us/azure/remote-rendering/how-tos/conversion/get-informationhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fremote-rendering%2Fhow-tos%2Fconversion%2Fget-information&data=02|01||b2d3f63d92c34efceb0908d7e09ca1a6|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637224835632051750&sdata=Kb%2FS3vfo%2B2YZ5T2ZH1tCwuZHONNhnitLM%2FrgzU6OJVk%3D&reserved=0
(The outputStatistics section -> boundingBox entry).
You may notice that bounding box values (min/max position) are not centered around the origin, but far off in one direction. In that case, you should re-export with "recenterToOrigin"
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fazure-docs%2Fissues%2F52193%23issuecomment-613587540&data=02|01||b2d3f63d92c34efceb0908d7e09ca1a6|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637224835632061744&sdata=qBUF18z8TpjlHMpgjuQRZZx1RbL2oev2r6FEKbErRiM%3D&reserved=0, or unsubscribehttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FADAIJ73VU4PQ4DYR2K6UWQTRMSPGVANCNFSM4MF2CPBA&data=02|01||b2d3f63d92c34efceb0908d7e09ca1a6|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637224835632061744&sdata=AMrjaPU6wNVECtbJ9FPIk6EaRVqmllCyrTtcuGb0y9A%3D&reserved=0.

There is no general problem with either model I tried.  They load in unity and works fine in their apps.  But there may need to be some requirement of the ARR that I am not accounting for.  Thanks to your info on the *info.json file.   I had uploaded a high and low poly model for this test (neither worked) but for now let me provide answers based on the low poly model to make sure polycount is not a factor. Based on this I have these findings (all referring to the Rampage50k model):
The output *info.json file is attached.  Is it correct to assume your boundingBox min/max define the box with opposite corner coordinates?   These numbers don't appear anywhere in Unity or my model that I can find so what are the steps needed to ensure ARR calculates the right values?  In this case the size of this model is {1465.6, 411.7, 1393.1} based on collider bounds which is much larger than the ARR bounding box.
Your link below is helpful to show how to recenter origin (but not provide a bounding box) so I think at this point even if I create a ConversionSettings.json to recenter the origin there will still be a problem because the ARR bounding box is so small compared to the model.
Looking forward to your comments.
Lance

From: FlorianBorn71<EMAIL_ADDRESS>Sent: Tuesday, April 14, 2020 12:34 PM
To: MicrosoftDocs/azure-docs<EMAIL_ADDRESS>Cc: cevanno<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/azure-docs] Conversion of 40M poly model results in 0 poly due to absolute path bug (#52193)
Hi @cevannohttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcevanno&data=02|01||8648ca21d017485f80da08d7e09a0894|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637224824479740413&sdata=S7nDWp6LOC99drMqQgN%2FUkk%2FkFNMwD%2Frv7ESeQBPkCQ%3D&reserved=0, do I understand your initial comment correctly that the render statistics indicate that no polygons are rendered? I assume the built-in engine model renders fine so the video stream etc. is working properly?
What can also be a problem is that the model is outside the view frustum, because it has been exported with far off-center pivot. To rule out that issue, can you try exporting with the "recenterToOrigin" option described here: https://docs.microsoft.com/en-us/azure/remote-rendering/how-tos/conversion/configure-model-conversionhttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fremote-rendering%2Fhow-tos%2Fconversion%2Fconfigure-model-conversion&data=02|01||8648ca21d017485f80da08d7e09a0894|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637224824479740413&sdata=vBOsHEPFTr2BKWKHktR39YulfnUaDN4IBtHwl71NoGg%3D&reserved=0
If it is a problem with the model itself (e.g. for some reason it is exported as empty mesh) we should have a look at it here. Would it be possible to provide us with a small sample source model, assuming you may disclose it to us?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fazure-docs%2Fissues%2F52193%23issuecomment-613578441&data=02|01||8648ca21d017485f80da08d7e09a0894|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637224824479750407&sdata=EaT4BzVtZ4G2U1dNKtfncdQ0UhgbKCyUHO3oEn9N2gQ%3D&reserved=0, or unsubscribehttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FADAIJ77X5RIPGEW6JYHFVULRMSNA5ANCNFSM4MF2CPBA&data=02|01||8648ca21d017485f80da08d7e09a0894|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637224824479750407&sdata=%2FF4dfkbV3%2BYlAOxO1rEZD%2BqJM5M5VUtIA0SeXLVvaXY%3D&reserved=0.

"The output *info.json file is attached"
I don't see an attached file?

Is it correct to assume your boundingBox min/max define the box with opposite corner coordinates?
Yes, a bounding box is defined by two positions (min/max) in 3d space, not position and size.
Please note that units are specified in meters. So a coordinate like 1465.6 means it is >1km off the camera.
There can be two main issues with large coordinates which I want to outline now. For simplicity, I leave y and z as placeholders:

Case 1: The box is far off-center. In this case, the bounding box looks like this:
(1500, y1,z1,  1501, y2,z2)
i.e. the extent of the box is OK (1m) but it is too far away from the camera so it is clipped by far clipping distance
Solution: re-center option for conversion, it will then become -0.5 … +0.5
Case 2 : The box is centered, but extremely large:
(-1500, y1,z1,  1500, y2,z2)
that means your camera starts in the center of the box but the scaling is so large that the model is clipped in all directions
Solution: Apply scaling factor during conversion, e.g. 0.001 if it is a km to meter issue.
Some model source formats contain hints what units they are using and our conversion tries to respect this and then converts to meter. Sometimes this hint isn't present or just wrong in which case such out-of-scale models might be produced.

How does one set the bounding box in Unity or Blender? At this point it seems I would need to upload a model just to generate the *.info.json file and then see what random size the boundingBox is.  then find a scale factor that reduces my model to fit and upload it again.  Seems there has to be a better way.
I agree with you that my model is larger than the boundingBox section suggests so it could be clipping.
I assume your mail system is stripping attachments so here is the file
{
"files": {
"input": "2019Rampage50K/Mountain-50K.fbx",
"output": "Mountain-50K.arrAsset"
},
"conversionSettings": {},
"inputInfo": {
"sourceAssetExtension": ".fbx",
"sourceAssetFormat": "Autodesk FBX Importer",
"sourceAssetFormatVersion": "7400",
"sourceAssetGenerator": "FBX SDK/FBX Plugins version 2020.0"
},
"inputStatistics": {
"numMeshes": 2,
"numFaces": 53998,
"numVertices": 161994,
"numMaterial": 2,
"numFacesSmallestMesh": 1280,
"numFacesBiggestMesh": 52718,
"numNodes": 3,
"numMeshUsagesInScene": 2,
"maxNodeDepth": 2
},
"outputInfo": {
"conversionToolVersion": "935eb028bf8550f1f923c5213abb11b628675491",
"conversionHash": "49509A611E28824D"
},
"outputStatistics": {
"numMeshPartsCreated": 2,
"numMeshPartsInstanced": 0,
"recenteringOffset": [
0,
0,
0
],
"boundingBox": {
"min": [
-622.329,
45.0938,
-458.72
],
"max": [
522.069,
456.79,
505.196
]
}
}
}

From: Florian Born<EMAIL_ADDRESS>Sent: Monday, April 20, 2020 3:43 AM
To: MicrosoftDocs/azure-docs<EMAIL_ADDRESS>Cc: cevanno<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/azure-docs] Conversion of 40M poly model results in 0 poly due to absolute path bug (#52193)
"The output *info.json file is attached"
I don't see an attached file?
Is it correct to assume your boundingBox min/max define the box with opposite corner coordinates?
Yes, a bounding box is defined by two positions (min/max) in 3d space, not position and size.
Please note that units are specified in meters. So a coordinate like 1465.6 means it is >1km off the camera.
There can be two main issues with large coordinates which I want to outline now. For simplicity, I leave y and z as placeholders:
Case 1: The box is far off-center. In this case, the bounding box looks like this:
(1500, y1,z1, 1501, y2,z2)
i.e. the extent of the box is OK (1m) but it is too far away from the camera so it is clipped by far clipping distance
Solution: re-center option for conversion, it will then become -0.5 … +0.5
Case 2 : The box is centered, but extremely large:
(-1500, y1,z1, 1500, y2,z2)
that means your camera starts in the center of the box but the scaling is so large that the model is clipped in all directions
Solution: Apply scaling factor during conversion, e.g. 0.001 if it is a km to meter issue.
Some model source formats contain hints what units they are using and our conversion tries to respect this and then converts to meter. Sometimes this hint isn't present or just wrong in which case such out-of-scale models might be produced.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fazure-docs%2Fissues%2F52193%23issuecomment-616401756&data=02|01||8f10a6a421a14a46249508d7e506e7eb|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637229690127380340&sdata=BBR9ay7i8dTBQVvaGWdqzHx%2BADtmpx6%2FAUkqa8HoHIs%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FADAIJ75HIO2R7VPMQQHXJVTRNQDLFANCNFSM4MF2CPBA&data=02|01||8f10a6a421a14a46249508d7e506e7eb|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637229690127390345&sdata=Mz48bjwnhV7MufW5GSdvRwsB8IDG%2F1dpa60kcjNXP%2BU%3D&reserved=0.

Since the units in the box are meters, it indeed looks like your model is too large and geometry is clipped in all directions.
There is no way to explicitly set a bounding box in content creation tools like Blender. Instead, the box is a result of the min/max extent converted geometry, specifically the vertex positions. You should see corresponding units in the source tool, and maybe there is a mismatch between the units used by Blender  and what the conversion thinks it is.
Example: In Blender the coordinates appear in nice -0.5 .. +0.5 range but units are set to kilometer (somewhere). Then our conversion assumes it needs to convert the units so 0.5 ends up as 500m.
Can you test and set a scaling value of say 0.001 to compensate? Just to verify the model itself works?
PS: The rest of the data in the output json looks valid, so I don't see there is a general problem with the model outside the scaling

#please-close

Sorry for the delay as I was pulled away. I uploaded a ConversionSettings.json to recenter and shrink.  Now the bounding box looks fine for both the high poly and low poly model (seen below).  The low poly model now loads and works in the demo which is great.  The high poly model still does not show even though the status says 100% loaded like before.  Both conversions are successful so this might be a limitation of the Unity RR script and not Azure?
Since the goal is to be able to show a user a model that their local computer alone might not be able to render (and use the power of Azure to render for them) I still hope to understand if my problem is a Azure limitation or something I am doing.
To quantify this here are some statistics:
"large" model:
"inputStatistics": {
"numMeshes": 86,
"numFaces": 45359144,
"numVertices": 136077432,
"Small" model:
"inputStatistics": {
"numMeshes": 2,
"numFaces": 53998,
"numVertices": 161994,
"boundingBox": {
"min": [
-5.56448,
11.3549,
-4.94013
],
"max": [
5.77943,
14.9224,
6.55381
]
}

Hi, the output statistics for the large model look good, however, the
"numFaces": 45359144,
entry suggests that you need a "Premium" size VM allocated, since 20M triangles is the limit for "Standard" size:
https://docs.microsoft.com/en-us/azure/remote-rendering/reference/limits#overall-number-of-polygons
Unfortunately the reporting on the runtime side isn't there yet (e.g. show a banner with information). But we are working on that.
Would you like to try with Premium size? (Note premium is more expensive than standard)
Florian

No thank you.  However I do have a quick question at this time to have the info I need to report about. With the 50K model working I was seeing a stream from Azure of 150KB/s.  I am wondering what the network rate would be when remote rendering the 45M poly model?   is there is some back end optimization going on, for instance automatic quad tree or octree pyramiding,  in the ARR conversion so you stream multiple LODs or would the network load to play this larger model be 1000 times more?

From: Florian Born<EMAIL_ADDRESS>Sent: Monday, May 4, 2020 5:45 AM
To: MicrosoftDocs/azure-docs<EMAIL_ADDRESS>Cc: cevanno<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/azure-docs] Conversion of 40M poly model results in 0 poly due to absolute path bug (#52193)
Hi, the output statistics for the large model look good, however, the
"numFaces": 45359144,
entry suggests that you need a "Premium" size VM allocated, since 20M triangles is the limit for "Standard" size:
https://docs.microsoft.com/en-us/azure/remote-rendering/reference/limits#overall-number-of-polygonshttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fremote-rendering%2Freference%2Flimits%23overall-number-of-polygons&data=02|01||878376fe069c482c288408d7f018482a|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637241859382792626&sdata=tmxw8Icq85Q1P5C1OnTlE%2Fv%2BsUZTh4jcqVlFseeJVaE%3D&reserved=0
Unfortunately the reporting on the runtime side isn't there yet (e.g. show a banner with information). But we are working on that.
Would you like to try with Premium size? (Note premium is more expensive than standard)
Florian
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fazure-docs%2Fissues%2F52193%23issuecomment-623391655&data=02|01||878376fe069c482c288408d7f018482a|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637241859382802621&sdata=DPrmV0EvnHPjpKJsLI0fP8BkNavfP3AJyzcIN%2ByEPZI%3D&reserved=0, or unsubscribehttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FADAIJ76OKLXGN5T6KYZKCULRP2MFDANCNFSM4MF2CPBA&data=02|01||878376fe069c482c288408d7f018482a|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637241859382802621&sdata=hVVyukmAyC1Rba%2BF1BgFSi5Qdbft%2FVIwHMDkCP3UEx4%3D&reserved=0.

After the model has been loaded on the server, it will transfer the scene hierarchy (all movable parts of the object) to the client. This data is tightly packed so it does not consume too much network bandwidth unless it consists of say a million parts. And even then you would only notice a one-time hit on the network. Note I'm referring to movable parts, not actual triangle geometry, because the latter is not transferred to the client - that is the whole idea of remote rendering ;-)
Now when the model is properly loaded, it does not at all matter how complex the model is, because we are just streaming a video image of the rendered content. If the connection is bad, you might see video compression artifacts as the compression quality will be throttled dynamically.
All the optimizations for managing the large scene graph and culling on the renderer side happens transparently on the server and never on the client.
Hope this answers your question?
Florian

Yes it does.  Thank you.  This is exactly as I had hoped.  I hope we can get our client to go for this solution as I am encouraged to go forward.  I did this test in Unity so do the Azure assets compile for ios and android?

From: Florian Born<EMAIL_ADDRESS>Sent: Thursday, May 7, 2020 3:10 AM
To: MicrosoftDocs/azure-docs<EMAIL_ADDRESS>Cc: cevanno<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/azure-docs] Conversion of 40M poly model results in 0 poly due to absolute path bug (#52193)
After the model has been loaded on the server, it will transfer the scene hierarchy (all movable parts of the object) to the client. This data is tightly packed so it does not consume too much network bandwidth unless it consists of say a million parts. And even then you would only notice a one-time hit on the network. Note I'm referring to movable parts, not actual triangle geometry, because the latter is not transferred to the client - that is the whole idea of remote rendering ;-)
Now when the model is properly loaded, it does not at all matter how complex the model is, because we are just streaming a video image of the rendered content. If the connection is bad, you might see video compression artifacts as the compression quality will be throttled dynamically.
All the optimizations for managing the large scene graph and culling on the renderer side happens transparently on the server and never on the client.
Hope this answers your question?
Florian
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fazure-docs%2Fissues%2F52193%23issuecomment-625100941&data=02|01||c7ced55bdfa04d3e36be08d7f25e1b90|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637244358302483181&sdata=kQB3kiszeM2xBi%2BlClnwvWa3sfIzNGr7JltQgEeZJHA%3D&reserved=0, or unsubscribehttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FADAIJ7YHC7VXQO3RICEKPJTRQJUHLANCNFSM4MF2CPBA&data=02|01||c7ced55bdfa04d3e36be08d7f25e1b90|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637244358302493176&sdata=OpS4Rpl%2FMIG0fI4S5SBMzyH4z9nszWIScNvcG6LrN0M%3D&reserved=0.

We currently have no version for iOS or Android, however you can run the client code as a desktop app on surface tablets. If you need more platforms, I suggest you create a feature request item here:
https://feedback.azure.com/forums/928696-azure-remote-rendering
There is already an item about platforms, you may want to upvote that as well.
Cheers,
Florian


Reimplement announcements

Closes #21496.
This PR improves the implementation of announcements by moving announcement strings into i18n.toml and simplifies the implementation by only calling partials/announcement.html and partials/deprecationwarning.html, instead of calling to shortcodes as well.
This PR does not remove the unused shortcodes. This will be done in a follow up task: see #21609.

@celestehorgan Thank you so much for this, it looks ✨ !

Let's see if I can remember the Prow command for merge methods...
/tide merge-method-squash

/label tide/merge-method-squash

/hold cancel
/lgtm
/approve


HELP opening vscode from powershell opens cli.js

Cross Posted

VS Code Version: 1.54.3
Powershell Version: 5.1.18362.752
OS Version: 10.18362.0

Steps to Reproduce:

Go to powershell cd to the working directory and type code .

Does this issue occur when all extensions are disabled?: Yes/No
Yes

Is code pointing to code.cmd or something else?

well since it opens something in vscode I'm assuming it's pointing to code.exe

The issue may be you have VS Code's root install dir on your $PATH then, not the bin dir

yea that's it. I just deleted that but how do I fix that? VSCode isn't installed to the C drive
Installed drive is: D:\Program Files\Microsoft VS Code\

You can edit your system's environment variables through the start menu, make sure D:\Program Files\Microsoft VS Code\bin is there instead of D:\Program Files\Microsoft VS Code\

D:\Program Files\Microsoft VS Code\bin

wait that's what I had before. I know that vscode shouldn't open cli.js so how does this get fixed??

Sure,
❯ Get-Command code
Get-Command: The term 'code' is not recognized as a name of a cmdlet, function, script file, or executable program.
Check the spelling of the name, or if a path was included, verify that the path is correct and try again.


errors in polishing after clustering

Operating system
Which operating system and version are you using?
Linux
Package name
Which package / tool is causing the problem? Which version are you using, use tool --version. Have you updated to the latest version conda update package? Have you updated the complete env by running conda update --all? Have you ensured that your channel priorities are set up according to the bioconda recommendations at https://bioconda.github.io/#set-up-channels?
isoseq3
Conda environment
What is the result of conda list? (Try to paste that between triple backticks.)
it went wrong when i did polishing after clustering demo data from isoseq3.
Describe the bug
A clear and concise description of what the bug is.
Error message
Paste the error message / stack.

|> 20201115 13:21:42.367 -|- ERROR -|- operator() -|- 0x2b2763668700|| -|- Could not find ZMWs for cluster transcript/6
|> 20201115 13:21:42.368 -|- ERROR -|- operator() -|- 0x2b276406d700|| -|- Could not find ZMWs for cluster transcript/11
|> 20201115 13:21:42.368 -|- ERROR -|- operator() -|- 0x2b276426e700|| -|- Could not find ZMWs for cluster transcript/12
|> 20201115 13:21:42.384 -|- ERROR -|- operator() -|- 0x2b2763065700|| -|- Could not find ZMWs for cluster transcript/4
|> 20201115 13:21:42.384 -|- ERROR -|- operator() -|- 0x2b2762e64700|| -|- Could not find ZMWs for cluster transcript/2
|> 20201115 13:21:42.386 -|- ERROR -|- operator() -|- 0x2b276446f700|| -|- Could not find ZMWs for cluster transcript/13
|> 20201115 13:21:42.387 -|- ERROR -|- operator() -|- 0x2b2764c73700|| -|- Could not find ZMWs for cluster transcript/17
|> 20201115 13:21:42.388 -|- ERROR -|- operator() -|- 0x2b2762c63700|| -|- Could not find ZMWs for cluster transcript/1
|> 20201115 13:21:42.388 -|- ERROR -|- operator() -|- 0x2b2764a72700|| -|- Could not find ZMWs for cluster transcript/16
|> 20201115 13:21:42.388 -|- ERROR -|- operator() -|- 0x2b2763467700|| -|- Could not find ZMWs for cluster transcript/5
|> 20201115 13:21:42.389 -|- ERROR -|- operator() -|- 0x2b2763266700|| -|- Could not find ZMWs for cluster transcript/3
To Reproduce
Steps to reproduce the behavior. Providing a minimal test dataset on which we can reproduce the behavior will generally lead to quicker turnaround time!
ccs m64065_201020_094110.subreads.2--2.bam ccs.isoseq.bam -j 40 --skip-polish --min-rq 0.9
lima ccs.isoseq.bam primers.fasta demux.ccs.bam -j 20 --isoseq --peek-guess --no-pbi
isoseq3 refine demux.ccs.primer_5p--primer_3p.bam primers.fasta Isoseq.flnc.bam -j 20 --require-polya
isoseq3 cluster Isoseq.flnc.bam Isoseq.clustered.unpolished.bam --verbose
isoseq3 polish Isoseq.clustered.unpolished.bam m64065_201020_094110.subreads.2--2.bam Isoseq.polished.bam -j 20

There is an error in step 5, and I have tried many existing solutions on the web without solving the problem!
Expected behavior
A clear and concise description of what you expected to happen.
finishing the polishing successfully.

I'll have a look
On Sun, 15 Nov 2020, 15:25 Luffy<EMAIL_ADDRESS>wrote:

Operating system
Which operating system and version are you using?
Linux
Package name
Which package / tool is causing the problem? Which version are you using,
use tool --version. Have you updated to the latest version conda update
package? Have you updated the complete env by running conda update --all?
Have you ensured that your channel priorities are set up according to the
bioconda recommendations at https://bioconda.github.io/#set-up-channels?
isoseq3
Conda environment
What is the result of conda list? (Try to paste that between triple
backticks
https://help.github.com/articles/creating-and-highlighting-code-blocks/
.)
it went wrong when i did polishing after clustering demo data from isoseq3.
Describe the bug
A clear and concise description of what the bug is.
Error message
Paste the error message / stack.
|> 20201115 13:21:42.367 -|- ERROR -|- operator() -|- 0x2b2763668700|| -|-
Could not find ZMWs for cluster transcript/6
|> 20201115 13:21:42.368 -|- ERROR -|- operator() -|- 0x2b276406d700|| -|-
Could not find ZMWs for cluster transcript/11
|> 20201115 13:21:42.368 -|- ERROR -|- operator() -|- 0x2b276426e700|| -|-
Could not find ZMWs for cluster transcript/12
|> 20201115 13:21:42.384 -|- ERROR -|- operator() -|- 0x2b2763065700|| -|-
Could not find ZMWs for cluster transcript/4
|> 20201115 13:21:42.384 -|- ERROR -|- operator() -|- 0x2b2762e64700|| -|-
Could not find ZMWs for cluster transcript/2
|> 20201115 13:21:42.386 -|- ERROR -|- operator() -|- 0x2b276446f700|| -|-
Could not find ZMWs for cluster transcript/13
|> 20201115 13:21:42.387 -|- ERROR -|- operator() -|- 0x2b2764c73700|| -|-
Could not find ZMWs for cluster transcript/17
|> 20201115 13:21:42.388 -|- ERROR -|- operator() -|- 0x2b2762c63700|| -|-
Could not find ZMWs for cluster transcript/1
|> 20201115 13:21:42.388 -|- ERROR -|- operator() -|- 0x2b2764a72700|| -|-
Could not find ZMWs for cluster transcript/16
|> 20201115 13:21:42.388 -|- ERROR -|- operator() -|- 0x2b2763467700|| -|-
Could not find ZMWs for cluster transcript/5
|> 20201115 13:21:42.389 -|- ERROR -|- operator() -|- 0x2b2763266700|| -|-
Could not find ZMWs for cluster transcript/3
To Reproduce
Steps to reproduce the behavior. Providing a minimal test dataset on which
we can reproduce the behavior will generally lead to quicker turnaround
time!
ccs m64065_201020_094110.subreads.2--2.bam ccs.isoseq.bam -j 40
--skip-polish --min-rq 0.9
lima ccs.isoseq.bam primers.fasta demux.ccs.bam -j 20 --isoseq
--peek-guess --no-pbi
isoseq3 refine demux.ccs.primer_5p--primer_3p.bam primers.fasta
Isoseq.flnc.bam -j 20 --require-polya
isoseq3 cluster Isoseq.flnc.bam Isoseq.clustered.unpolished.bam --verbose
isoseq3 polish Isoseq.clustered.unpolished.bam
m64065_201020_094110.subreads.2--2.bam Isoseq.polished.bam -j 20
There is an error in step 5, and I have tried many existing solutions on
the web without solving the problem!
Expected behavior
A clear and concise description of what you expected to happen.
finishing the polishing successfully.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/PacificBiosciences/pbbioconda/issues/352, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAK55RSLM3YJ7FOK5M3SLKDSP7QEBANCNFSM4TWHDM3Q
.

Can you please verify that the input to ccs is already demuxed and thus you are trying an unsupported workflow?

yes，I sure that the input to ccs is already demuxed. The workflow is as follows:
ccs m64065_201020_094110.subreads.2--2.bam ccs.isoseq.bam -j 40 --skip-polish --min-rq 0.9
lima ccs.isoseq.bam primers.fasta demux.ccs.bam -j 20 --isoseq --peek-guess --no-pbi
isoseq3 refine demux.ccs.primer_5p--primer_3p.bam primers.fasta Isoseq.flnc.bam -j 20 --require-polya
isoseq3 cluster Isoseq.flnc.bam Isoseq.clustered.unpolished.bam --verbose
isoseq3 polish Isoseq.clustered.unpolished.bam m64065_201020_094110.subreads.2--2.bam Isoseq.polished.bam -j 20

Hi @DongnaMa
Has your problem been solved?
I have encountered the same problem on multiple datasets. Unfortunately, I have not yet found a solution.

Hi @litao1503
Because I used HIFI sequencing, the accuracy is very high, so I did not polish


Add custom event data to event metrics

(WIP, will fill this out when I have a little more work done on this)

@tairosonloa oh, sure, if you like! I'm busy last and this week, I was planning on getting some more done next week though.
There's going to need to be some refactoring of the front-end table code to support this use case in the way I'm thinking about it, but I know @mikecao has plans for event data too, and it'd be great to hear from him on how he's thinking about all of this too.
Gonna write out some thoughts and assumptions here, just to have it written down somewhere:

Given the current database structure, it seems like event data is intended to basically be any kind of JSON data. In the current state, there are fields in the database for different data types (string, number, date, boolean, array), which ends up being awkward to both query and write to (see this mess). Would it make sense to instead store event data directly as json/jsonb?
I could see it being useful to allow some transformations on custom event data (i.e. being able to choose fields to group metrics by) when viewing event data. Certainly this could be taken quite far, but I know we're not trying to build Splunk here — but it'd be worth thinking about what users might want from that kind of functionality.
Looking further into the future, what if umami deployers could write plugins for custom event data views? I'm thinking about how Payload allows you to plop in your own components in certain places.

We are working on an event data feature for the next release, so I wouldn't recommend putting too much effort here. Event data is not meant to be returned as part of regular statistics. We are building out a Reports feature where you will be able to query your event data specifically.
The reason we moved away from json-b is due to performance. There is no way to properly index the data when the data can be anything.

Ah gotcha, makes sense. Thanks for the heads up, I'll hold off for now.

We are working on an event data feature for the next release, so I wouldn't recommend putting too much effort here. Event data is not meant to be returned as part of regular statistics. We are building out a Reports feature where you will be able to query your event data specifically.
The reason we moved away from json-b is due to performance. There is no way to properly index the data when the data can be anything.

@mikecao did this land? Is there a public facing roadmap of features? Thanks!

This feature is now out of scope with the current version.


Add the ability to skip TLS verify on servers

Removes the need to have a valid SSL certificate installed on the server in order to execute the k6 tests (such as many non-production or example servers).  This is especially useful when altering or developing load tests where you may be working with local servers in order to validate the correctness of the load test scripts themselves.

thank you, one question. What is the difference to the default option we have https://github.com/owncloud/cdperf/blob/c758a4dbead9b59e5e19d65e33a43479939f7552/src/k6/src/lib/k6/options.ts#L8

Yes, you are right - if you are happy with the default of this being true then let's close this PR.

In some cases I use the tests with a plain k6, to have the value inside ts helps running it. I would love to see a way to keep the default in the ts file but make it overwriteable by using --k6-insecure

Hmm I see - they use different mechanisms but it appears --k6-insecure=false will win (ie. will prevent running load tests if the SSL certificate is not valid) if you run it with that option.
I'm indifferent to having this PR accepted, up to you =)

doesn't look to me that it gets overwritten, i tested it with the following command
./scripts/cdperf --cloud-vendor=ocis --k6-test-host=https://localhost:9200 --k6-docker=false --cloud-docker=false --k6-scripts=./src/k6/dist/test-issue-github-ocis-1018-upload-download-delete-with-new-user.js --k6-insecure=false
i like the idea to be able to disable it. Can you re-check if you see an option to keep it?

Yep let's close this one.


$set for modifying objects in array for reactivity

Vue.js version
2.1.0
Problem:
I can't update an object in my array of objects editedPlayers[] when I use this.$set(). When I try to do this.editedPlayers.$set(id, this.editedPlayers[id].stats[type] + 1). I know this is wrong.
editedPlayers[{stats:{goals: 12, assists:3, played: 10}}, {stats:{goals: 23, assists:7, played: 30}}]
I want to update the stats.goals in editedPlayers[someId]. Since $set or splice() can be used to substitute the current index value with another, I want just to modify it (ie: stats.goals). What do I do?

Please make sure to read the Issue Reporting Guidelines before opening new issues. The issue list only accepts bug reports and feature requests. Questions should be posted to the Gitter chat room,  forum or StackOverflow. Thanks.

You just do editedPlayers[id].stats[type]++. If the property is already observed there's no need to use $set at all.
Also, in 2.0 Arrays are no longer augmented with $set, and this.$set on component instances are also different from the 1.x version: https://vuejs.org/v2/api/#vm-set

When I do this.editedPlayers[id].stats[type]++, Vue doesn't detect the change because I am "directly setting an item with the index". See this, https://vuejs.org/v2/guide/list.html#Caveats. Now, this.$set only allows me to 'substitute' a value in the array given the key, with another value. I want to 'modify' editedPlayers[id].stats[type] and not entirely replace editedPlayers[id].

No, the actual mutation happens on the object inside the array, not the array itself, so it is not subject to the index detection caveat.

I've made this repo to show you the problem. Please take a look at it. https://jsfiddle.net/donnyjeremiah/tfvouu61/.
You can see here that once editedPlayers['someID'] is loaded for the first time in the interpolation with the {{ }}, it isn't picking up the updated state which is being changed in the editPlayerStat() method later. However you do see in the console that editedPlayers['someID'] is actually being updated. This is my problem. I need it to be updated in within the curly brackets, {{ }}.
P.S: I've tried using this.$set() to rewrite the whole value, but since this.$set() takes the 'index' of the indexed array as second argument, I run into another problem and that is because my 'index' is a string and basically there is no such thing as associative arrays in javascript. And hence, I get thrown the error, array length too big....

You seem to have a misunderstanding of how JavaScript array works... the way you are using it is like an Object hash, not an Array. Mutations to Arrays in JavaScript happen through methods like push() and pop().
Doing this is adding a new property to the Array as an Object, not pushing a new element to the Array:
arr['someId'] = { ... }

But this is not what Arrays are for. If you want to store things with string keys you should use an Object.
And since you are adding a new property, you should use Vue.set().

Yeah, I understood that an array with strings for indexes is basically an object with a property, when I was creating this repo. I didn't know that before. But there is one thing. Despite, the way or purpose I've used the array, why isn't Vue updating it? It should, right? So since arr['someId'] is just arr.someId and can be accessed by arr['someId'] AND arr.someId, why isn't it updating?

Oh boy.
Here is your fixed repo: https://jsfiddle.net/tfvouu61/2/
Please note Vue.set and read Change Detection Caveats once again.

@simplesmiler Thank you :) So just to clarify, because I'm very restless when I don't know what it was that I did wrong, was it because that I was using Vue.set() to update the value instead of using it to set the object to begin with? And did I hit the change detection caveat?

Yes. Vue was unable to detect the addition of a new property someID to the object this.editedPlayers due to the limitations of JavaScript.

Evan Almighty! So I wasn't insane after all! It's just that I wasn't sure what problem I was running into. Thank you @simplesmiler and @yyx990803! You guys have good day :)

"due to the limitations of JavaScript." -- hogwash, it's due to the way Vue is architected, which is OK by the way, just don't try to make it sound like it's the underlying language's fault


Update link to Circuit Breaker pattern

@MikeWasson : Thanks for your contribution! The author, @tamram, has been notified to review your proposed change.

@tamram Sorry, I should have opened this PR in azure-docs-pr.

@MikeWasson - No problem, thanks for making the change. (-;
#sign-off

@MikeWasson - can you make sure to check the link after this change is published? The public repo doesn't have link validation and I can't do it manually.  thanks!


Lag in scrolling

When you scroll your RecyclerView (fast/quickly) the imageview part goes white and then it gets loaded, which gives a bad user experience.

I advice you to use FeaturedLinearLayoutManager which is packaged in the library.


PSA: Indicate in the error returned when we know that an AEAD nonce length is invalid, not just unsupported

Fixes #5144.
The motivation for this PR is that the PSA Crypto Compliance test suite expects us to return PSA_ERROR_INVALID_ARGUMENT when psa_aead_encrypt() or psa_aead_decrypt() is passed the arguments nonce = NULL, nonce_length = 0.
This PR consists of 2 commits (based on top of #5094, due to the changes we need to make to EXPECTED_FAILURES):
The first commit (02ee7dc) restores the behaviour that we return PSA_ERROR_INVALID_ARGUMENT if we are passed an invalid nonce length when using CCM or GCM. This matches the behaviour found in released versions of Mbed TLS, and development_2.x.
This existing behaviour is however slightly inconsistent, since when using ChaCha20-Poly1305 we always return PSA_ERROR_NOT_SUPPORTED if nonce_length != 12 - even though nonce_length == 8 is also valid, just unsupported by Mbed TLS.
The second commit (a6c255e) fixes this inconsistency by checking for nonce_length == 8 and returning PSA_ERROR_INVALID_ARGUMENT in every other case.
I'm open to dropping the second commit if we feel like it's unnecessary / out of scope.
Status
READY
Requires Backporting
development_2.x: #5194

The development_2.x backport is up: #5194

I've rebased this PR on top of development to resolve the merge conflicts.


BREAKING CHANGE: Updates to schema

Seems this is required immediately in the main branch, as Auth seems working as expected

Probably, need to do multiple PRs for a quality schema. Currently we will work on User, Profile and QuickAdd APIs
later we PRs deal with polishing of remaining stuff


Song name and Artist name didn't displayed on validation message when uploading song

When you upload a song, this happen

#12 fixe it in theory


Dead link in README.md

The link to blog post returns 404
http://blog.opendatalab.de/hack/2013/11/07/android-host-card-emulation-with-acr122/
remove the trailing slash will do the trick

Thanks. It's fixed now.


Passing wrong default chainId in makeWeb3Provider puts provider in corrupt state

It's possible I've misunderstood documentation, but I've scoured as many examples as I can and I can't seem to figure out how to resolve this issue:
My dapp connects to walletlink without any special config:
        const walletLink = new WalletLink({ … });
        this._provider = walletLink.makeWeb3Provider(INFURA_API_KEY);

The d.ts file states that the default second param to makeWeb3Provider (the chainId) is 1. I'm confused by this param because I have no idea what chain the user would like to connect to, and if it's not mainnet, it can cause issues. Here's an example flow:

User opens Coinbase Wallet on his phone and chooses Kovan as his "Default Network".
On opens the dapp, clicks to connect Coinbase Wallet. QR code is shown.
User scans QR code to connect.

At this point, the dapp connects the user and correctly identifies that the user wants Kovan (42). walletlink automatically sets "DefaultChainId = 42" in local storage. The dapp is able to check balances and execute transactions on the Kovan network. Until…

User refreshes the page on the dapp. The dapp reconnects as per the code above, passing no chainId.

Now the dapp appears connected and still reports Kovan as its network, because walletlink internally checks local storage to report network, but it's a lie. When attempting to make any requests or execute any contracts, the provider uses whatever network was passed to makeWeb3Provider, or in my case, the default which is mainnet. The provider is corrupted.
To temporarily resolve the issue, I've used the following hack to detect the user's last chainId and pass it as the default chainId:
      let defaultChainId = 1;
      try {
        const storage = window.localStorage.getItem(
          '-walletlink:https://www.walletlink.org:DefaultChainId'
        );
        if (storage) {
          defaultChainId = Number(storage);
        }
      } catch (error) {
        console.warn(
          `Couldn't connect to local storage to find walletconnect's default chain id.`
        );
      }

It seems there's a few possible solutions y'all could use internally:

Have the WalletLinkProvider constructor check for the network storage's default if no chainId is passed. This would essentially take my code above and move it into the WalletLinkProvider.
Likely better because it avoids stale state: don't ask developers to provide a default chainId and don't assume any defaults; always check the wallet's current network.

Thanks!

Hmm, perhaps I've resolved this on my own. I failed to consider that I was passing the mainnet infura URL to the WalletLink constructor and then passing a Kovan chainId to makeWeb3Provider.
Needless to say, I've found this API very confusing.

It's possible I've misunderstood documentation, but I've scoured as many examples as I can and I can't seem to figure out how to resolve this issue:
My dapp connects to walletlink without any special config:

        const walletLink = new WalletLink({ … });

        this._provider = walletLink.makeWeb3Provider(INFURA_API_KEY);

The d.ts file states that the default second param to makeWeb3Provider (the chainId) is 1. I'm confused by this param because I have no idea what chain the user would like to connect to, and if it's not mainnet, it can cause issues. Here's an example flow:

User opens Coinbase Wallet on his phone and chooses Kovan as his "Default Network".

On opens the dapp, clicks to connect Coinbase Wallet. QR code is shown.

User scans QR code to connect.

At this point, the dapp connects the user and correctly identifies that the user wants Kovan (42). walletlink automatically sets "DefaultChainId = 42" in local storage. The dapp is able to check balances and execute transactions on the Kovan network. Until…

User refreshes the page on the dapp. The dapp reconnects as per the code above, passing no chainId.

Now the dapp appears connected and still reports Kovan as its network, because walletlink internally checks local storage to report network, but it's a lie. When attempting to make any requests or execute any contracts, the provider uses whatever network was passed to makeWeb3Provider, or in my case, the default which is mainnet. The provider is corrupted.
To temporarily resolve the issue, I've used the following hack to detect the user's last chainId and pass it as the default chainId:

      let defaultChainId = 1;

      try {

        const storage = window.localStorage.getItem(

          '-walletlink:https://www.walletlink.org:DefaultChainId'

        );

        if (storage) {

          defaultChainId = Number(storage);

        }

      } catch (error) {

        console.warn(

          `Couldn't connect to local storage to find walletconnect's default chain id.`

        );

      }

It seems there's a few possible solutions y'all could use internally:

Have the WalletLinkProvider constructor check for the network storage's default if no chainId is passed. This would essentially take my code above and move it into the WalletLinkProvider.

Likely better because it avoids stale state: don't ask developers to provide a default chainId and don't assume any defaults; always check the wallet's current network.

Thanks!

hi. sorry for the belated response. we are marking stale issues closed for now. please try our latest version 3.6.0. feel free to open a new one if needed. thanks.


Adds new command 'pp dataverse webresource list'. Closes #5785

Closes #5785

@greczimarton when we merge your PR, we will add you to our team page. May we know your name and surname so we can add you?

Seems like your code doesn't have 100% coverage, could have a look at it?

Seems like your code doesn't have 100% coverage, could have a look at it?

Oops My bad, I missed a few lines. Should be 100% now

@greczimarton when we merge your PR, we will add you to our team page. May we know your name and surname so we can add you?

Sure, my first name is Márton and my surname/lastname is Gréczi

Hi @greczimarton did you lose track of this? Don't want to put any pressure just double-checking.

Hi Milan, thanks for the review.
Next week I will be able to go through the comments.

@greczimarton are you still willing to work on this PR?

Hi @milanholemans, Sorry for the delay.
I'd be happy to work on this, however I started working at a new job and unfortunately I have less capacity now.
I also lost my accesses to D365 environments so it'd be difficult to test.
Maybe at the end of this month I'll have more time and I can finish this PR.
Let me know how you would like to proceed.

Hi @greczimarton, things in life can change. Personal/professional life always takes priority. If you find the time to finish it, that would be cool. If you don't, that's also fine, we can always close the PR and leave it for someone else.

Let me know how you would like to proceed.

Just review the comments I made in my last review and change the code accordingly.

Hi @greczimarton, are you still willing/able to work on this PR? If not, feel free to say so.

Hi @milanholemans, sorry I can no longer work on this PR.

Not a problem at all! Thank you the time spent.


add: support for pushing to the Hub and loading from the Hub

As discussed in https://github.com/NUS-HPC-AI-Lab/OpenDiT/issues/61.
Stuff done so far:

Added support for pushing the final experimental files from train.py to the Hub if push_to_hub flag is enabled.
Loading from the Hub (only for the image models).
Incorporate support for using accelerate to massively speed up model loading time.

You can use the script below to verify the third point:
from opendit.models.dit import DiT_models
from accelerate import init_empty_weights
from accelerate.utils import set_module_tensor_to_device
from huggingface_hub import hf_hub_download
from contextlib import nullcontext
import torch
import argparse

def main(args):
    init_ctx = init_empty_weights if args.speedup_loading else nullcontext
    image_size = 256
    input_size = image_size // 8
    dtype = torch.float32

    ckpt_path = hf_hub_download(repo_id="sayakpaul/OpenDiT", filename="OpenDiT-DiT-XL-2-256x256.pt")
    state_dict = torch.load(ckpt_path, map_location="cpu")

    with init_ctx():
        model = DiT_models["DiT-XL/2"](
            use_video=False,
            input_size=input_size,
            num_classes=1000,
            enable_flashattn=False,
            enable_layernorm_kernel=False,
            dtype=dtype,
            text_encoder=None,
        )
    if args.speedup_loading:
        for param_name, param in state_dict.items():
            set_module_tensor_to_device(model, param_name, "cpu", value=param)
    else:
        model.load_state_dict(state_dict)

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--speedup_loading", action="store_true")
    args = parser.parse_args()

    main(args)

Here are the results:
Without accelerate:
CPU times: user 236 ms, sys: 34.3 ms, total: 270 ms
Wall time: 34.3 s

With accelerate:
CPU times: user 61.7 ms, sys: 9.31 ms, total: 71 ms
Wall time: 8.14 s

While comparing the results, I ensured that the model was cached and wasn't re-downloaded.
I have also taken the liberty to host the OpenDiT checkpoint here: https://huggingface.co/sayakpaul/OpenDiT/. Would be nice if you could create an org on the Hub so that we can transfer this checkpoint to your org (or you can do so yourself if it's okay).
I want to get a review first from the library maintainers to check if they're okay with the changes. I can then propagate them to the video scripts too. Will add a section in the README about the usage as well.
Cc: @oahzxl
Cc: @LysandreJik for awareness.

@oahzxl a gentle ping :-)

May I know why this was closed without any reasoning and despite having agreement on having the feature? I find that to be outright disrespectful.

we are very thankful for your contribution. the reason why we close this pr is that it is already outdated because we dont use that model any more. and currently all latest checkpoints are on huggingface.

Thanks.
Can the checkpoints be directly loaded into the model without manual downloads?

yeah currently we use model.from_pretrained to get models from huggingface


Pdoexceptions

Hi, when trying to use php artisan koel:init
it returns me this error
 [Illuminate\Database\QueryException]
  SQLSTATE[HY000]: General error: 1025 Error on rename of './koel_data/#sql-a
  29_11' to './koel_data/songs' (errno: 150) (SQL: ALTER TABLE songs CHANGE c
  ontributing_artist_id artist_id INT UNSIGNED DEFAULT NULL)

  [Doctrine\DBAL\Driver\PDOException]
  SQLSTATE[HY000]: General error: 1025 Error on rename of './koel_data/#sql-a
  29_11' to './koel_data/songs' (errno: 150)

  [PDOException]
  SQLSTATE[HY000]: General error: 1025 Error on rename of './koel_data/#sql-a
  29_11' to './koel_data/songs' (errno: 150)

`
Is there a workaround?
```

I think someone posted the same error (which was strictly related to their system/setup), do you mind a search? Cheers.


impute CpG values for gene by cell matrix

Hi there,
I have a CpG methylation level matrix (gene by cell), and I'm curious is there any way to use DeepCpG tool to impute the methylation level on the genes?

See the documentation and the examples in examples/.


Use systemUrlStreamHandlerFactory id it was already initialized from Main

Main calls FrameworkContext.setupURLStreamHandleFactory();

Thanks for the fix! I also missed that one. I'll we check this PR next week.

Sorry, missed this for the 6.1.3 release, will include it in the next.


fixed bug in VirtualCompliance class

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 81.42%.  Consider uploading reports for the commit dcce779 to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4472      +/-   ##
==========================================
- Coverage   81.43%   81.42%   -0.01%
==========================================
  Files         110      110
  Lines       53480    53481       +1
==========================================
- Hits        43550    43547       -3
- Misses       9930     9934       +4


Update radial-gauge.ts

Move gauge initialisation into ngAfterViewInit due to Angular 9 changes

Sorry needs further testing


Clarify that implementations can perform a flush operation for CBO.INVAL (and CBO.CLEAN)

Request for a non-normative statement from https://github.com/riscv/riscv-profiles/issues/87

Hi @dkruckemyer-ventana did this go anywhere? It would be good to have the non-normative statement added

@dkruckemyer-ventana You'll need to create a PR that the IC chairs can review and merge.  It is desirable to get this done soon as part of the ISA manuals' cleanup cycle that we are currently in.

Addressed in https://github.com/riscv/riscv-isa-manual/pull/1428


adds getModelByProperty Function

as discussed in the issue https://github.com/AnalyticalGraphicsInc/cesium/issues/2937

@jbo023 sorry for the delay here; we did a lot of refactoring and unit testing in the 3d-tiles branch, and the branch is now in good shape for contributions.
I think this is a useful function (thanks again!), but I want to wait on merging it since @mramato and crew are going to implement declarative styling for 3D Tiles, which will need more generic queries than this, e.g., instead of just ===, we'll want more general expressions and a really fast implementation for modifying models based on them instead of filling up the array and relying on the caller.


is it compatible to use your first edition in ubunut 22.04

hi sir, I am getting started with kernel, is it ok to use 22.04, or else should I have to use 18.04, and when is your latest book on kernel development is releasing sir(sorry to raise this as issue sir)

Np. You can use Ubuntu 22.04.. There might be small differences in package versions, but overall it should work for kernel dev.
Am working on the 2nd ed of the LKP book, should hopefully be done by September.

OK, closing


Add linux-htb QoS and Ingress/Egress tests

Changes

Added tests for linux-htb performance and QoS change in queues.
Added test for pod ingress/egress limits.

Thanks for the feedback @stonepreston, last commit implemented all the suggested changes.


Improve overall plugins load performance

-antigen-parse-args is very expensive. Avoiding using it in internal calls improve overall performance.
Instead of using -antigen-parse-args just invoke functions with arguments in expected order.
I got 50% performance improvement.

Just a note, another way to get 50% improvement is to avoid using -antigen-parse-args at all. For example in my own .zshrc I specified function right after source ./antigen/antigen.zsh
    antigen-fbundle() {
        local loc=$1
        local url=${2:-$ANTIGEN_DEFAULT_REPO_URL}
        local make_local_clone=${3:-true}
        if $make_local_clone; then
            -antigen-ensure-repo $url
        fi
        -antigen-load "$url" "$loc" "$make_local_clone"
    }

After that I use simplified syntax of

    # antigen use oh-my-zsh
    antigen-fbundle lib

    antigen-fbundle plugins/bower
    #... other plugins

    # load from url
    antigen-fbundle / https://github.com/Tarrasch/zsh-autoenv.git

    # load from local locations (my own plugins)
    antigen-fbundle / ./plugins/aliases false

    # Load plugin
    antigen-fbundle powerlevel9k.zsh-theme https://github.com/bhilburn/powerlevel9k.git

    antigen apply

I think they'll like to keep the old syntax. Regardless of that it's an alternative to the whole cache thing at https://github.com/zsh-users/antigen/pull/129. I'll try this later today.

@desyncr sure, it make sense, I don't want to change the syntax, this is why I just mentioned that it can be another way to improve performance.
This pull request is only about avoiding unnecessary using -antigen-parse-args in internal calls.
We definitely need to fix -antigen-parse-args, as it makes antigen load in 4x times slower. As a proposal we can make it less generic and work only for antigen bundle (including all other alternatives, like use or theme).

Just remember, desyncr isn't a contributor to antigen at all. I have a fixed version of antigen parse args #132, but abandoned it because the maintainers seem to be absent, so I just switched to zgen.

@forivall looking on the history of commits it seems like that original maintainer @sharat87 is looking on patches once in a month, so we just need to have a little bit of patience. Could you reopen your pull request as it can be very useful for all antigen users.

Thank you @outcoldman for these changes. It has been merged.


Add without_no_data function

See comment here: maybe something like nan_to_num perhaps called without_no_data, analog of extant with_no_data
For int types: just remove nodata cell type designation.
For floating types: Change ND to zero.  Inf to MaxDouble? -Inf to MinDouble? And leave cell type the same unless user defined 0 nodata value, in which case use default/NaN.

While doing this, consider how we can add rf_with_no_data to SQL ...


terraform module to deploy ranger admin and usersync

As sysops,
I wan't to create terraform module intial version to deploy ranger admin and usersync services on ECS,

fixed as part of https://github.com/ExpediaInc/apiary-authorization/pull/2


fixes an issue where the detail from uglify error messages are lost, …

…making it impossible to track down the underlying cause of a problem.

Merged, thank you, @johnspackman


feat: Expose config for mTLS certificates.

Summary of Changes
Provide Env Vars to configure a client cert for mTLS.
A client certificate can be specified to present with requests to
metadata and search services for mTLS. The configuration is given
as a pair of paths to the PEM encoded cert and key files. This is
described in more detail in the requests documentation:
https://requests.readthedocs.io/en/master/user/advanced/#client-side-certificates
This assumes that the frontend should present the same client
cert to both the search and metadata services, which is consistent
with the idea that a client cert should uniquely identify the
service that is presenting it.
If the environment variables are not set, then the clients are
created as normal without any expectation of mTLS.
This addresses https://github.com/amundsen-io/amundsen/issues/754
Tests
I did not see any existing tests around the API utils, and I did not know what approach to take for them.
Documentation
There are Python docstrings on the fields added to the config.
CheckList
Make sure you have checked all steps below to ensure a timely review.

[x] PR title addresses the issue accurately and concisely. Example: "Updates the version of Flask to v1.0.2"

In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

[x] PR includes a summary of changes, including screenshots of any UI changes.
[x] PR adds unit tests, updates existing unit tests, OR documents why no test additions or modifications are needed.
[x] In case of new functionality, my PR adds documentation that describes how to use it.

All the public python functions and the classes in the PR contain docstrings that explain what it does

[x] PR passes all tests documented in the developer guide

will take a look

lgtm mostly with one question

thanks @davcamer


fix: increase scrolling threshold to fix weird scrolling behaviour

Description
Widget scrolling was introduced in the context switching project to ensure that a selected widget is never out of view. This went through some changes and it seems like when widgets on edges increase height on interaction (select widget) or if the widget is longer than the current view, the scrolling behaviour did not make sense.
To fix this problem, we have increased the threshold of scrolling, and will only scroll when less than 5% of the widget is visible on screen. Anything more and it will not scroll the page
PR fixes following issue(s)
Fixes #25146
Media
https://github.com/appsmithorg/appsmith/assets/12022471/2cc251a3-01c9-44d8-9b6e-536e727250ec
Type of change

Bug fix (non-breaking change which fixes an issue)

Testing

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Also list any relevant details for your test configuration.
Delete anything that is not relevant

[ ] Manual
[ ] Jest
[ ] Cypress

Test Plan

Add Testsmith test cases links that relate to this PR

Issues raised during DP testing

Link issues raised during DP testing for better visiblity and tracking (copy link from comments dropped on this PR)

Checklist:
Dev activity

[ ] My code follows the style guidelines of this project
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] PR is being merged under a feature flag

QA activity:

[ ] Speedbreak features have been covered
[ ] Test plan covers all impacted features and areas of interest
[ ] Test plan has been peer reviewed by project stakeholders and other QA members
[ ] Manually tested functionality on DP
[ ] We had an implementation alignment call with stakeholders post QA Round 2
[ ] Cypress test cases have been added and approved by SDET/manual QA
[ ] Added Test Plan Approved label after Cypress tests were reviewed
[ ] Added Test Plan Approved label after JUnit tests were reviewed

/build-deploy-preview env=release

/ok-to-test

Looks Good @hetunandu

/ok-to-test

/ok-to-test

/ok-to-test

/ok-to-test


Switch pygeofilter dependency to PyPI package

Description
Switch pygeofilter dependency to PyPI package

Remove pygeofilter from requirements_dev.txt.
Delete requirements_build.txt, remove pip install from dockerfile and add dependencies to setup.py instead.
Remove getter build stage which downloaded pygeofilter source.
Remove pygeofilter pip install to stop install from source in build.
Add version to pygeofilter in setup.py.
Update README.md

Fixes #4
Type of change
Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Instructions
Running the test suite without errors will confirm this change is effective.
How Has This Been Tested?

[x] Run scripts/test

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[x] Any dependent changes have been merged and published in downstream modules

@JBurkinshaw The testing Dockerfile seems to inherit from the main image, then install a bunch of dependencies. That'll be pretty slow. All of the test requirements will need to be installed every time a single file changes in the base image.
A multistage build would be a lot faster, but I'm not sure how much time you have to put into this.

@JBurkinshaw The testing Dockerfile seems to inherit from the main image, then install a bunch of dependencies. That'll be pretty slow. All of the test requirements will need to be installed every time a single file changes in the base image.
A multistage build would be a lot faster, but I'm not sure how much time you have to put into this.

Putting this in another issue to address later: #19


Support selectors

I would like to
- name: Create conda environment
  uses: mamba-org/setup-micromamba@main
  with:
    environment-file: environment.yaml
    environment-name: myenv
    init-shell: bash
    cache-downloads: true
    post-cleanup: all
    create-args: |
      sel(win): clang_win-64

but that does not seem to be supported any longer

How about something like
- name: Create conda environment
  uses: mamba-org/setup-micromamba@main
  with:
    environment-file: environment.yaml
    environment-name: myenv
    init-shell: bash
    cache-downloads: true
    post-cleanup: all
    create-args: |
      ${{ runner.os == 'Windows' && 'clang_win-64' || '' }}

(untested)
Alternatively, you should still be able to add selectors to environment.yaml.

Thanks, that works!


First Release

First Release of function and relative terraform script to get DB Audit Logs from DataSafe and put in OCI Logging

I have read the CLA Document and I hereby sign the CLA

I have read the CLA Document and I hereby sign the CLA


Generic error when publishing if there are assets that don't belong on the site

Describe the bug
If an asset folder has permissions only on Site A, and Site B has a page that uses an asset from that folder, then publishing that page will throw a generic "error" in the front end.
To Reproduce
Steps to reproduce the behavior:

Create two sites (Navigation button at the top left)

Create an asset folder and add a few Rich Text assets to it.

Click the asset folder, then click the Folder properties button

Select only one site in the permissions for that folder.  Save the settings

In the site that you did not select for step 5, create a new page

Add a new Rich Text widget to the page in the Layout view.  Then switch to the Content view.

Drag a Rich Text asset from the asset folder you created directly onto the page where the Rich Text widget is located

Save and publish the page.  Note that an "error" comes up on the page with no additional info on what went wrong.

Expected behavior
An error should come up explaining that there is an asset on the page that does not belong on the site (site doesn't have permissions on that asset folder)

@matomario8 What version / build is affected by this?

I updated the original post but the version where I saw the issue was Version 8.0.3 Build 202204R01 (118)

Thanks

It looks like this would need to get updated in multiple places:

drag drop of asset to widget in finder needs a check
jcr query in image / file list widgets
asset browse dfialogs on image and file widgets
Image slider widget


[SYSTEMML-1160] [WIP] Enable prefetching of batches via for loop iterator

This PR extends current for construct to support iterating over batch. The current version only supports iterating over batches of rows.
The syntax
for(X_batch in X, nrow=batch_size) {
  ...
}

is equivalent to
i = 0
for(i in 1:iters) {
	i = i + 1
	# Get next batch
	beg = ((i-1) * batch_size) %% N + 1
	end = min(N, beg + batch_size - 1)
	X_batch = X[beg:end,]
}

Note: This PR does not introduce rectangular blocks in SystemML, instead handles it in the ForProgramBlock. To enable prefetching, the user just has to set a config flag:
ml.setConfigProperty("prefetch.budget.mb", "300"); // 300 mb prefetch memory

@dusenberrymw @mboehm7 @bertholdreinwald @frreiss @asurve

Note: This PR does not introduce rectangular blocks in SystemML and certain robustness checks are remaining (such as extremely large and/or wide batches).

I will run few more experiments and update this PR soon :)

Approach 1: With Prefetching
The below statistics show that for convnet on 200K column dataset with batchsize of 50, we only wait for fetching of the first batch (~41 seconds). For remaining batches, there is almost zero-wait time.
ml.setConfigProperty("prefetch.budget.mb", "300"); // 300 mb prefetch memory

for (e in 1:epochs) {
        i = 0
        for(X_batch in X, nrow=batch_size) {
			i = i + 1
			if(i == 100) {
			   stop ("Early stopping for benchmarks")
			}

		  # Get next batch
		  beg = ((i-1) * batch_size) %% N + 1
		  end = min(N, beg + batch_size - 1)
		  y_batch = Y[beg:end,]
	  ....
	}
}

time: 1384233.345149ms
SystemML Statistics:
Total elapsed time:             0.000 sec.
Total compilation time:         0.000 sec.
Total execution time:           0.000 sec.
Number of compiled Spark inst:  76.
Number of executed Spark inst:  3.
Batch-Fetch (next,indx,wait,first):     41.586/355.839/41.585/41.583 sec.
Cache hits (Mem, WB, FS, HDFS): 19796/0/0/1.
Cache writes (WB, FS, HDFS):    4767/139/0.
Cache times (ACQr/m, RLS, EXP): 0.090/0.018/30.515/0.000 sec.
HOP DAGs recompiled (PRED, SB): 0/595.
HOP DAGs recompile time:        1.699 sec.
Functions recompiled:           1.
Functions recompile time:       0.164 sec.
Spark ctx create time (lazy):   0.017 sec.
Spark trans counts (par,bc,col):0/0/102.
Spark trans times (par,bc,col): 0.000/0.000/355.286 secs.
Total JIT compile time:         172.599 sec.
Total JVM GC count:             773.
Total JVM GC time:              123.818 sec.
Heavy hitter instructions (name, time, count):
-- 1)   conv2d_bias_add         817.763 sec     297
-- 2)   sel+    196.118 sec     396
-- 3)   -       45.240 sec      2178
-- 4)   conv2d_backward_filter  44.979 sec      297
-- 5)   relu_maxpooling         44.417 sec      297
-- 6)   +*      41.579 sec      1485
-- 7)   r'      28.868 sec      495
-- 8)   conv2d_backward_data    28.579 sec      198
-- 9)   maxpooling_backward     21.912 sec      297
-- 10)  *       19.874 sec      2880

Approach 2: Maxi-batch
for (e in 1:epochs) {
      maxi_batch_Size = 20*batch_size
      maxi_iters = ceil(N / maxi_batch_Size)
      stopping_i = 0
      for(maxi in 1:maxi_iters) {
          # Get next batch
          maxi_beg = ((maxi-1) * maxi_batch_Size) %% N + 1
          maxi_end = min(N, maxi_beg + maxi_batch_Size - 1)
          X_maxibatch = X[maxi_beg:maxi_end,]
          y_maxibatch = Y[maxi_beg:maxi_end,]
          maxi_N = nrow(y_maxibatch)
          iters = ceil(maxi_N / batch_size)
          for(i in 1:iters) {
            stopping_i = stopping_i + 1
            if(stopping_i == 100) {
              stop ("Early stopping for benchmarks")
            }
            # Get next batch
            beg = ((i-1) * batch_size) %% maxi_N + 1
            end = min(maxi_N, beg + batch_size - 1)
            X_batch = X_maxibatch[beg:end,]
            y_batch = y_maxibatch[beg:end,]
			....
		}
	}
}

time: 2334624.686333ms
SystemML Statistics:
Total elapsed time:             0.000 sec.
Total compilation time:         0.000 sec.
Total execution time:           0.000 sec.
Number of compiled Spark inst:  81.
Number of executed Spark inst:  8.
Cache hits (Mem, WB, FS, HDFS): 19895/0/0/6.
Cache writes (WB, FS, HDFS):    7403/64/0.
Cache times (ACQr/m, RLS, EXP): 80.049/0.025/33.767/0.000 sec.
HOP DAGs recompiled (PRED, SB): 0/605.
HOP DAGs recompile time:        1.907 sec.
Functions recompiled:           1.
Functions recompile time:       0.303 sec.
Spark ctx create time (lazy):   0.026 sec.
Spark trans counts (par,bc,col):0/0/5.
Spark trans times (par,bc,col): 0.000/0.000/79.953 secs.
Total JIT compile time:         83.437 sec.
Total JVM GC count:             1441.
Total JVM GC time:              254.973 sec.
Heavy hitter instructions (name, time, count):
-- 1)   conv2d_bias_add         780.249 sec     297
-- 2)   conv2d_backward_filter  442.185 sec     297
-- 3)   conv2d_backward_data    304.401 sec     198
-- 4)   sel+    200.346 sec     396
-- 5)   uack+   115.980 sec     495
-- 6)   maxpooling_backward     107.120 sec     297
-- 7)   rangeReIndex    85.561 sec      203
-- 8)   relu_backward   61.133 sec      495
-- 9)   ba+*    43.830 sec      594
-- 10)  relu_maxpooling         42.892 sec      297

I used the same cluster and same configuration as our VLDB experiments.
@bertholdreinwald @mboehm7 @dusenberrymw

@niketanpansare Thanks for running this experiment.  A 1.7x speedup certainly shows that our current methods are leaving a large amount of performance on the table.

These numbers are a bit puzzling. If there is a 1000s difference, how come that rangeReIndex (which also covers the lazy spark indexing) only requires 85s (from which the 80s for ACQ give us an upper bound on the time for collecting the indexed data)?

I noticed backwards instructions are also taking more time.

On Feb 17, 2017, at 7:09 PM, Matthias Boehm<EMAIL_ADDRESS>wrote:
These numbers are a bit puzzling. If there is a 1000s difference, how come that rangeReIndex (which also covers the lazy spark indexing) only requires 85s (from which the 80s for ACQ give us an upper bound on the time for collecting the indexed data)?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Here is the statistics on a different cluster with 70g driver memory:
Approach 1: With Prefetching
time:<PHONE_NUMBER>08ms
SystemML Statistics:
Total elapsed time:             0.000 sec.
Total compilation time:         0.000 sec.
Total execution time:           0.000 sec.
Number of compiled Spark inst:  75.
Number of executed Spark inst:  3.
Batch-Fetch (next,indx,wait,first):     61.151/168.933/61.151/61.149 sec.
Cache hits (Mem, WB, FS, HDFS): 19796/0/0/1.
Cache writes (WB, FS, HDFS):    7506/3/0.
Cache times (ACQr/m, RLS, EXP): 0.110/0.010/3.239/0.000 sec.
HOP DAGs recompiled (PRED, SB): 0/595.
HOP DAGs recompile time:        0.985 sec.
Functions recompiled:           1.
Functions recompile time:       0.086 sec.
Spark ctx create time (lazy):   0.013 sec.
Spark trans counts (par,bc,col):0/0/102.
Spark trans times (par,bc,col): 0.000/0.000/168.039 secs.
Total JIT compile time:         54.711 sec.
Total JVM GC count:             508.
Total JVM GC time:              96.755 sec.
Heavy hitter instructions (name, time, count):
-- 1)   conv2d_bias_add         206.028 sec     297
-- 2)   conv2d_backward_filter  157.850 sec     297
-- 3)   sel+    110.459 sec     396
-- 4)   conv2d_backward_data    70.346 sec      198
-- 5)   uack+   50.411 sec      495
-- 6)   maxpooling_backward     50.044 sec      297
-- 7)   relu_backward   28.905 sec      495
-- 8)   relu_maxpooling         21.635 sec      297
-- 9)   ba+*    20.543 sec      594
-- 10)  +*      15.317 sec      1485

Approach 2: Maxi-batch
time:<PHONE_NUMBER>38ms
SystemML Statistics:
Total elapsed time:             0.000 sec.
Total compilation time:         0.000 sec.
Total execution time:           0.000 sec.
Number of compiled Spark inst:  80.
Number of executed Spark inst:  8.
Cache hits (Mem, WB, FS, HDFS): 19796/0/0/6.
Cache writes (WB, FS, HDFS):    7438/1/0.
Cache times (ACQr/m, RLS, EXP): 138.341/0.009/3.571/0.000 sec.
HOP DAGs recompiled (PRED, SB): 0/605.
HOP DAGs recompile time:        0.900 sec.
Functions recompiled:           1.
Functions recompile time:       0.146 sec.
Spark ctx create time (lazy):   0.013 sec.
Spark trans counts (par,bc,col):0/0/5.
Spark trans times (par,bc,col): 0.000/0.000/138.253 secs.
Total JIT compile time:         41.224 sec.
Total JVM GC count:             574.
Total JVM GC time:              102.319 sec.
Heavy hitter instructions (name, time, count):
-- 1)   conv2d_bias_add         196.187 sec     297
-- 2)   conv2d_backward_filter  160.273 sec     297
-- 3)   rangeReIndex    142.017 sec     203
-- 4)   sel+    108.674 sec     396
-- 5)   conv2d_backward_data    70.360 sec      198
-- 6)   maxpooling_backward     51.672 sec      297
-- 7)   uack+   49.394 sec      396
-- 8)   relu_backward   36.753 sec      396
-- 9)   relu_maxpooling         20.152 sec      297
-- 10)  +*      18.896 sec      1485

Does this second cluster use 10Gb or 1Gb Ethernet?

10 Gb Ethernet:
$ lspci | grep -iE --color 'network|ethernet'
01:00.0 Ethernet controller: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
01:00.1 Ethernet controller: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
16:00.0 Ethernet controller: Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
16:00.1 Ethernet controller: Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
16:00.2 Ethernet controller: Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
16:00.3 Ethernet controller: Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)

@bertholdreinwald @mboehm7 @dusenberrymw Any review updates for this PR ?
Since its being a while, here is the summary of the discussion related to this PR:

This PR adds a new syntax to DML language, which is useful for implementing SGD-style algorithms:

for(X_batch in X, nrow=batch_size) {
  ...
}

Additionally, it allows for prefetching of X_batch with the help of the configuration property prefetch.budget.mb. The memory specified by this property is subtracted from the local memory budget. This could be a drawback in low-memory settings.

Alternative is to compute the aggregate memory budget of all the child blocks and subtract it from local memory budget to get the prefetch budget. In many cases, the aggregate memory budget will likely be a gross over-estimate of actual memory required as it doesnot take into account the effect of rmvar and our caching. We can extend ProgramBlock to compute this:

/**
 * Computes the memory estimate if known, else -1.
 * If OptimizerUtils.isMemoryBasedOptLevel(), it returns OptimizerUtils.INVALID_SIZE.
 *
 * @return the memory estimate for this program block
 * @throws HopsException if error occurs
 */
public double getMemEstimate() throws HopsException {
	if(_sb == null)
		return 0;
	if(!OptimizerUtils.isMemoryBasedOptLevel())
		return OptimizerUtils.INVALID_SIZE;

	double memBudget = 0;
	for(Hop h : _sb.get_hops()) {
		double memEstimateForH = h.getMemEstimate();
		if(memEstimateForH < 0)
			return -1;
		else
			memBudget += memEstimateForH;
	}
	return memBudget;
}

Alternatively, we can replace memBudget += h.getMemEstimate(); with memBudget = max(memBudget, memEstimateForH) as an optimistic estimate assuming immediate rmvar and/or caching.

I'm still not convinced by this PR because the experimental results were non-conclusive (especially in comparison to simple rewrites), it does not nicely fit into our the architecture of memory management (without unnecessary over-provisioning), it unnecessarily differs from R syntax, and the CP batch iterator might actually create more evictions in the presence of memory pressure. So I would recommend to not rush this in.

Refer to this link for build results (access rights to CI server needed):
https://sparktc.ibmcloud.com/jenkins/job/SystemML-PullRequestBuilder/1439/

additional things to think about: (1) handling of updates in terms of assignments or left indexing (we would need to handle this during validate), and (2) the impact on codegen, which is already able to fuse right indexing of column vectors into the subsequent fused operator.

Refer to this link for build results (access rights to CI server needed):
https://sparktc.ibmcloud.com/jenkins/job/SystemML-PullRequestBuilder/1725/


Remove the custom profile from cargo test in logging example

What
Remove the custom profile from cargo test in logging example. and explain when the custom profile is required.
Why
When running tests the default test profile already has logs enabled, so it is unnecessary to tell people to use the custom profile when running tests.

Something went wrong with PR preview build please check


Viki Login

Checklist

[ ] I'm reporting a broken site support issue
[x] I've verified that I'm running youtube-dl version 2021.01.08
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar bug reports including closed ones
[x] I've read bugs section in FAQ

Verbose log

PASTE VERBOSE LOG HERE

Description

WRITE DESCRIPTION HERE
I've getting the error while using the -u parameter for viki premium and debug code was below,
Fix the bug as soon.
root@wanlab:~/demo# youtube-dl -F  https://www.viki.com/videos/1026409v-my-love-from-the-star-episode-1 -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-F', u'https://www.viki.com/videos/1026409v-my-love-from-the-star-episode-1', u'-v']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.01.08
[debug] Python version 2.7.16 (CPython) - Linux-4.19.0-13-amd64-x86_64-with-debian-10.7
[debug] exe versions: ffmpeg 4.1.6-1, ffprobe 4.1.6-1
[debug] Proxy map: {}
[viki] 1026409v: Downloading video JSON
[viki] 1026409v: Downloading MPD manifest
WARNING: [viki] Unknown MIME type image/jpeg in DASH manifest
[info] Available formats for 1026409v:
format code  extension  resolution note
mpd-http-6   m4a        audio only DASH audio   49k , m4a_dash container, mp4a.40.2 (48000Hz)
mpd-http-7   m4a        audio only DASH audio   65k , m4a_dash container, mp4a.40.2 (48000Hz)
mpd-http-5   m4a        audio only DASH audio   81k , m4a_dash container, mp4a.40.2 (48000Hz)
mpd-http-8   m4a        audio only DASH audio  193k , m4a_dash container, mp4a.40.2 (48000Hz)
mpd-http-9   m4a        audio only DASH audio  193k , m4a_dash container, mp4a.40.2 (48000Hz)
mpd-http-1   mp4        426x238    DASH video  203k , mp4_dash container, avc1.640015, 24fps, video only
mpd-http-2   mp4        640x360    DASH video  405k , mp4_dash container, avc1.64001E, 24fps, video only
mpd-http-3   mp4        852x480    DASH video  628k , mp4_dash container, avc1.64001E, 24fps, video only (best)
root@wanlab:~/demo# youtube-dl -F  https://www.viki.com/videos/1026409v-my-love-from-the-star-episode-1 -u<EMAIL_ADDRESS>-v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-F', u'https://www.viki.com/videos/1026409v-my-love-from-the-star-episode-1', u'-u', u'PRIVATE', u'-v']
Type account password and press [Return]:
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.01.08
[debug] Python version 2.7.16 (CPython) - Linux-4.19.0-13-amd64-x86_64-with-debian-10.7
[debug] exe versions: ffmpeg 4.1.6-1, ffprobe 4.1.6-1
[debug] Proxy map: {}
[viki] Logging in
ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 632, in _request_webpage
return self._downloader.urlopen(url_or_request)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2248, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
File "/usr/lib/python2.7/urllib2.py", line 435, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 473, in error
return self._call_chain(*args)
File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

x-viki-app-ver : 6.0.0
x-client-user-agent : Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.183 Safari/537.36


Added Container Registry storage

This PR change the Deployer in hacbs-sidecar to load the deployer from a factory and then have the ability to add more deployer.
The current default is still S3 for now, but this PR also adds a Container Registry Storage option.
Signed-off-by: Phillip Kruger<EMAIL_ADDRESS>

/approve

/ok-to-test

with https://github.com/redhat-appstudio/jvm-build-service/pull/129 in repo e2e's are green
/retest

though the compile is failing now at least in https://github.com/redhat-appstudio/jvm-build-service/runs/7493797597?check_suite_focus=true

@stuartwdouglas seems like the error has something to do with testcontainers that needs ryuk to performs fail-safe cleanup of containers. see https://www.testcontainers.org/features/configuration/
How does other builds that needs testcontainers handle this ?

What makes you say that? It looked like docker.io failed to service an image to me.

/retest

/retest

/retest

/retest

/retest

/retest


[BUG - Release] The Outer Worlds: Spacers Choice Edition FName::FName issue

Branch or Release
The Tag for the Release you are using, or the commit you used if you built from source yourself:
Release
UE4SS - v2.5.1 Beta #0 - Git SHA #1481302
Game and Engine Version
Game where the bug occurred, and the UE engine version of the game:
The Outer Worlds: Spacers Choice Edition
Engine version: 4.27.2
Describe the bug
A clear and concise description of what the bug is:
The current release version is unable to locate the function FName::FName using its default aob.
To Reproduce
Steps to reproduce the behavior:

Install release XInput version
start up the game via the steam client.
See error in log

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots and UE4SS Log
If applicable, add screenshots to help explain your problem, and upload the log:
applicable log slice.
[2024-01-21 05:36:41] Scan attempt 1/60
[2024-01-21 05:36:41] Engine Version: 4.27 <- Built-in
[2024-01-21 05:36:41] StaticConstructObject_Internal address: 0x7ff7b9521b60 <- Built-in
[2024-01-21 05:36:41] FName::ToString address: 0x7ff7b9327630 <- Built-in
[2024-01-21 05:36:41] Info: Found FName::ToString in both Engine & CoreUObject modules, using the first match
[2024-01-21 05:36:41] AOB scans could not be completed because of the following reasons:
Was unable to find AOB for 'FName::FName'
You can supply your own in 'UE4SS_Signatures/FName_Constructor.lua'

Desktop (please complete the following information):

OS: Win11

Additional context
Add any other context about the problem here.
Im not sure if it can help but this tool seems to work with the game atleast partially https://github.com/RussellJerome/UnrealModLoader/blob/main/Profiles/Indiana-Win64-Shipping.profile

Just tested the config from the wiki and it fails as well.

Try the latest experimental build.
Here's a link to it: https://github.com/UE4SS-RE/RE-UE4SS/releases/download/experimental/UE4SS_v2.5.2-570-g37e727b.zip
With this build you'll need both dwmapi.dll and UE4SS.dll from the zip file, and you should remove xinput1_3.dll that came from any other version of UE4SS as well as the UE4SS_Signatures directory.

sorry for taking a bit but i jumped to version UE4SS - v3.0.1 Beta #0 - Git SHA #d935b5b since thats the latest version as of today.
and it does seem to provide a much clearer response
[2024-04-17 10:29:28] PS Scan attempt 1
[2024-04-17 10:29:28] [PS] Reading image
[2024-04-17 10:29:28] [PS] Starting scan
[2024-04-17 10:29:28] [PS] Scan finished in 459.7368ms
[2024-04-17 10:29:28] [PS] Found EngineVersion: 4.27
[2024-04-17 10:29:28] [PS] Found GUObjectArray: 0x7ff64a99a150
[2024-04-17 10:29:28] [PS] Found GMalloc: 0x7ff64a915990
[2024-04-17 10:29:28] [PS] Found FName::ToString: 0x7ff6461172c0
[2024-04-17 10:29:28] [PS] Failed to find FName::FName(wchar_t*): iter returned multiple unique values
[2024-04-17 10:29:28] [PS] You can supply your own AOB in 'UE4SS_Signatures/FName_Constructor.lua'
[2024-04-17 10:29:28] [PS] Found StaticConstructObject_Internal: 0x7ff646311b60
[2024-04-17 10:29:28] [PS] Found FText::FText(FString&&): 0x7ff64602c7c0
[2024-04-17 10:29:28] [PS] Scan failed

if i remember correctly I think they possibly optimized way the fname constructor and you more or less need to "reconstruct it".


Added mode indicator

Will show different colors for sleep, dictation, mixed, command/other modes

Probably want a default=False for the activation setting so that if the .talon file gets deleted (a thing users are supposed to be able to do) the Python file doesn't start erroring.

Yeah good point. I think the standard thing here would be to use defaults for everything and then have the settings commented out in the settings file, but I think just doing that for activation setting is reasonable. Then can have a line that's commented out in the settings file that will activate it when I commented

Probably want a default=False for the activation setting so that if the .talon file gets deleted (a thing users are supposed to be able to do) the Python file doesn't start erroring.

Yeah good point. I think the standard thing here would be to use defaults for everything and then have the settings commented out in the settings file, but I think just doing that for activation setting is reasonable. Then can have a line that's commented out in the settings file that will activate it when uncommented

Initially I had all the defaults in the python file but after a lengthy discussion with @phillco we decided that the Talon file and the python file is part of the plugin. If you don't want this you can just delete the entire folder.

@pokey Gradient added as per your request :D

@lunixbochs Looks like screen scaling is applied automatically on mac, but not on windows. Is that a bug or something that we should work with?

Another Mac versus Windows difference that results in inconsistent behavior in the mode indicator appears to be the corner of the window that stays fixed when resizing a canvas:
Mac:
>>> from talon.canvas import Canvas; canvas = Canvas.from_rect(Rect(0,0,0,0)); canvas.resize(20,20); canvas.rect
Rect(0.0, -20.0, 20.0, 20.0)

Windows:
>>> from talon.canvas import Canvas; canvas = Canvas.from_rect(Rect(0,0,0,0)); canvas.resize(20,20); canvas.rect
Rect(0.0, 0.0, 20.0, 20.0)

@lunixbochs — should this behavior be consistent cross-platform in core Talon or should we be accommodating it in our Python code?

@nriley I have now updated it so it resizes before move. Please try it out again.


[Tracking Issue] Module-based Model Runtime Interface for AOT

This issue tracks the implementation of apache/tvm-rfcs#46 (Module-based Model Runtime Interface for AOT).
This RFC proposes a fairly wide-sweeping refactoring of the metadata exported from compiler to runtime. We plan to implement this in phases to ease code-review and unblock some downstream work dependent on the C-source-based AOTExecutor. Here are the steps:

[x] New Metadata classes (hand-written, without checking in the code-generator) to contain minimal metadata needed by AOTExecutor (#10282)
[x] AOTExecutor implementation and c target code-generator (#10283)
[x] llvm target code-generator (#10753)
[ ] Code-generator to auto-generate Metadata classes from a small Python description (PR not yet uploaded)
[ ] Unify metadata passed from compiler to runtime using new Metadata classes

@areusch is this still planned?

ah yeah these two unfulfilled tasks haven't been done yet


WIP: Make alloc optional

This contains an easy way of making alloc optional.  #479

Can we finally get that in? Every year I'm starting a new project that would benefit from using bytes in  no_std & no_alloc environment.
From https://github.com/tokio-rs/bytes/issues/479#issuecomment-778177512

The main problem is that we recently released v1 and don't want to release v2 right away.

I would assume that more than 3 years after v1 release we can allow a major update that would allow more people to finally use this lib.

I would assume that more than 3 years after v1 release we can allow a major update that would allow more people to finally use this lib.

The issue I see with it is that bytes is part of the public API of tokio, so that's the situation unfortunately.

(Sorry, I moved my comment to another issue that seemed to have more people following since I saw that there was no reviewer on this PR)

The issue I see with it is that bytes is part of the public API of tokio, so that's the situation unfortunately.

Since the current major version is 1, how having a breaking change to a new major version would be a problem?

Since the current major version is 1, how having a breaking change to a new major version would be a problem?

The issue is that it creates a lot of friction if bytes v2 gets released but tokio based crates stay at v1

So what would be the recommended way for libs/app code base who have to support async/tokio down to no_std/no_alloc? Just use another lib?
It sounds like changing a single line in Cargo.toml (i.e. adding features = ["alloc"] when already using default-features = false, which might not even break if they are not using alloc functionalities in the first place) in the few projects (that aren't using tokio because they are no_std) would be far less work than making whole libs/apps switch to another similar lib to properly support those functionalities.
Note that manipulating bytes safely makes particularly more sense on constrained devices (those no_std/no_alloc ones), and using a single, reliable library everywhere would be very nice for the ecosystem.
I would even consider putting that into a minor update, since this simply fixes a feature definition mistake when releasing v1, and shouldn't deeply impact a lot of projects.

I'll close this in favor of #606.


msbuild.exe / full dotnet SDK

Hi,
I'd like to write about CI with Windows Containers and msbuild.
Can you help me put together an image which could run msbuild / nuget and output DLLs etc?
Here are a couple of posts I'm thinking about bringing under CI in a tutorial:
Docker voting app converted to Windows containers + ASP.NET:
http://blog.alexellis.io/p/2bca0a86-6c72-4b36-a5d0-73dc58cf847a/
Getting started guide for IIS / Windows Containers + .NET
http://blog.alexellis.io/run-iis-asp-net-on-windows-10-with-docker/

Potential Dockerfile here? Thoughts?
https://github.com/alexellis/aspnet-voteservice/blob/master/src/Dockerfile.build

@alexellis We are looking at publishing official "-onbuild" images on DockerHub similar to what we've done for .NET Core already
Your Dockerfile looks good for just building your application, however it might be worthwhile to say perhaps have IIS/IIS Express to run integration tests for a web application. We've haven't reached consensus on what we're putting in that image, but we definitely plan to address the CI scenario.
As a data point, here's another Dockerfile I've seen out in the wild, so we realize you are not the only person with this question.
cc @staceyhaffner @glennc @richlander

This is where I got - http://blog.alexellis.io/3-steps-to-msbuild-with-docker/

We are looking at publishing official "-onbuild" images on DockerHub similar to what we've done for .NET Core already

Is this still in the roadmap?

Hi Alexellis, I am working on a project where I have to set up CI workflow for .net applications using Docker containers. I refered your blog to understand how to use Docker containers to build and deploy .net applications. Thanks for your work. Now I am struggling for a container to perform unit tests after code build. I am new to .net using containers and I hope you can provide some help with this.

Hi I'm glad you found the blog post useful. I'm not sure if this issue is the best forum to provide support, perhaps you could take it up with the .NET team at Microsoft or the .NET community forums?
Alex

Producing onbuild images is no longer something that we are considering.  The usage of onbuild images doesn't seem to have taken off.  Please refer to the samples for our recommended usage patterns.