Add OIDN Denoiser.

This PR Adds the OIDN Denoiser (Denoiser) into the core three-gpu-pathtracer.
It works, and looks decent but there are clear issues to resolve.
API Added

maxSamples Pathtracer now stops when reaching this number. Also is the number the system denoises at
enableDenoiser If we are using the denoiser (NOTE, the denosier actually initializes regardless of this for webGL compat)
weightsUrl Lets you bypass the jsDelivr URL for your own location of the TZAs. Also how you could pass your own weights in general
denoiserQuality Fast by default, Balanced is better. "high" is really heavy and not supported in the Denoiser yet
useAux Whether or not you want to use just the pathtraced input (fast, but bad) or with the Aux
renderAux If you want to view the aux inputs set this to albedo or normal null gets you the denoised output
cleanAux If you are 100% the aux inputs are clean, ANY noise will be projected into the output (NOTE, upstream fix may be needed)
externalAux If the user is supplying their own Aux Textures. disables automatic generation
denoiserDebugging Sets the denoiser to log more things including an output report.

State API Properties (read-only)

isDenoising If the denoiser is running
isDenoised if the current output is the denoised output

Methods

generateAux() runs the albedo/normals pass on the current scene. returns { albedo: THREE.Texture, normal: THREE.Texture }
setAuxTextures(albedoTexture, normalTexture) if the user wants to set the aux themselves, like in a pipeline or has a deferred setup. Deactiviates internal automatic creation (this.externalAux = true)
denoiseSample(inputTexture, albedoTexture, normalTexture) loads automatically or can be overridden letting you send to the denoiser directly to the DenoiserManager Not sure about exposing this, called internally by renderSample

Changes to Core

Not much was done to the core itself. The main thing is the addition of another fullScreenQuad pass that blends the pathtracer output with the denoised (or aux if you're debugging) outputs.
The second significant change is the addition of maxSamples and it's stopping the pathtracer when reached.
There were other changes to better support the denoiser, but nothing breaking.

One change was added to the BlendingMaterial to allow a conversion to happen to texture2. While we resolve the colorspace issues it may be useful.
Additions

AlbedoNormalPass: Generates Albedo & Normal textures based on the current scene
DenoiserManager: Holds all things related to the denoiser and interfaces with the Denoiser class directly

Flow With Denoising
Assuming Denoiser is enabled and all options default/set

Pathtracer runs until maxSamples is hit
Denoiser process started. Block future denoiseSample calls until finished
Albedo and Normal Textures Generated (only when needed by the denoiser, not every frame)
RawPathtracerTexture (pathtracer.target.texture), AlbedoTexture, and NormalTexture sent to the DenoiserManager.
If the renderer size has changed, regenerate renderTargets and set Denoiser sizes.
Render the RawPathtracer texture to a Quad with same tonemapping as default PT so resulting texture matches and within [0, 1]
Extract raw WebGLTextures from internals of input textures (THREE.Textures)
Set those Inputs on the denoiser
Execute the denoiser, return denoisedWebGLTexture
If not already created, create and load a ThreeJS texture that is properly initialized and setup within threejs internals outputTexture ( one of the many steps to get WebGL/Three to play nice)
Merge the denoisedWebGLTexture with the outputTexture
denoisedTexture is held in the class ready to be rendered
denoiser marked finished, results say it is now denoised
In the next renderSample call now that isDenoised is true, render using DenoiserManager.renderOutput()
Quad material now set to BlendingMaterial
Blend between previous PT Output (results of step 6) and the denoisedTexture (Note: if RenderAux is passed, will render the aux texture provided so you can see either the albedo or normal textures visually. Not sure the normals render how you'd expect, as they should be outputting in [-1, 1])

If reset is called it all drops back to normal, hides the outputs etc and starts over.
Known Issues
1. ColorSpaces/tonemapping/conversion: Something is clearly not setup right through the pipeline regarding colors. The denoiser DOES NOT CARE what colorspace you input. Whatever you input, it will output. Color and Albedo inputs should be in the same colorspace, and normals are expected to be in linear. The output looks correct but dark. converting makes it look flat.
This will take tweaking and someone smarter than me with regards to color to follow the renderTargets (all setup with THREE.SRGBColorSpace and what we should convert/adjust
2. Normal Generation: I have a script setup to use worldspace or viewspace normals and output to [-1, 1]. I don't know if colorSpace/RT's might be effecting this (I don't think it matters). Also, I have added code to accept normalMaps on meshes so they can be used instead of the raw surface. I tried converting the NormalMaps to worldspace with tragic results. For the moment If an object has a map I use it.
Something very important to point out about normals and the denoiser. The denoiser does not actually use them for normal values or any kind of light calculations whatsoever. They can be in world/local space. The normal maps just help define edges and breaks in materials.
3. Albedo Generation: In The OIDN Docs they go into detail about the albedos. In general, most matte surfaces can use just the basic color output or textures. This isn't true for highly reflective surfaces (like the floor). Here albedo wants something different, you can read the docs and it says something to the effect that the reflected surface should have the albedo of whatever they are reflecting, or a full 1 value. It gets even weirder for reflections.
4.  Edges with floor and background. Looking at original OIDN examples they do not expect black backgrounds or hard edges where soft edges blend with the original background. The normals/albedo read those as hard crisp edges or flat planes. So the gradient background gets flattened into a weird flat gradient and the floor has sharp edges. So with threejs backgrounds or envMaps we should generate something for both of these passes and include any floor/horizons.
The only one of these issues I see as a blocker is the colorspaces. Generating albedos and normals etc we can work on for a while and it would be fine to release improvements as updates. But gotta get closer on the color outputs IMO.
Other Changes
I updated the plain example (index) to include denoiser props and some other slight adjustments. I added a rawOutput canvas to the html which is very easy to setup when debugging to see exactly what the denoiser is outputting and confirm your inputs/outputs.
Notes
I tried commenting a lot to explain what is happening, and I realize my commits are terribly labeled. I was focusing on getting this added as the main thing and not making a lot of gradual changes.

I've done most of the things on your list,
Most importantly made the denoiser external OIDNDenoiser and I seem to have resolved the coloring issue for the denoised output at least.
It turns out we were passing Linear to the denoiser. Even though the render target and texture were set to SRGB it still output Linear. Adjusting the output manually in the shader corrected this.
Albedo & Normals were rewritten, still needs some work and the background always renders linear (weird)
I modified the BlendMaterial again as it's the best place to debug outputs/colors. I also used a more correct LinearToSRGB function.
Something helpful to check colors and interesting to compare is the "Split" property in the denoiser panel. This will divide the page between the pathtracer output and either the denoised result or the AuxTexture for comparison purposes.
Here you can see the sky is still wrong

A little messy still. I need a cleanup pass tomorrow.

Here you can see the sky is still wrong

This looks like a tone mapping issue - as in tone mapping is not applied to the final denoised image. If you disable "tone mapping" in the example UI the colors look the same:

I actually already tonemap the texture prior to it going to the denoiser.
...
At the moment I'm super happy with the color result. The pathtraced and denoised look great.

The quality does look really nice - but there is some color shift once it goes through the denoiser. It's just not clear why to me, yet. I'll have to take a look this weekend.

After battling render flow, making a bunch of charts  I think I have things in a stable setup. There are still a few problems to resolve.
Notable changes:

Simplified Index example, added Denoiser example with simpler controls but more denoiser controls
Got the color pipeline to match.

What the problem was: Threejs does not apply toneMapping to renderTargets. So even though I was copying the exact same textures/materials I was getting different outputs with not only different tonemapping but different colorSpaces.
I have added (and removed) a few key conversions that take place so we are now sending a SRGB non-tonemapped Pathtraced Image, a SRGB converted Albedo, and a standard Normal to the Denoiser then doing the tonemapping in the final step.
If I recall correctly though this is the wrong order and the tonemapping should happen BEFORE we send to the denoiser for the best outcome (standard for every pipeline with ANY denoiser, not just OIDN) but I could be wrong.
Problems:

Albedos just use basicMesh. I think this is the cause for some issues, most notably edges around the floor. If you set the floor to a 0 opacity and it has to render the background there is very little noise around the edges. But with the reflective floor (and a black albedo) it has issues. Transparency we'll have to test too. (this is a longer term issue to resolve)
Tiling seams. All the color conversions/pipelines/flow before & after denoising and not being tonemapped prior to the denoiser running, are causing seams in the tiling to become slightly visible. I dont know if this is the tiling from the pathtracer but I'm treating is as an issue with the denoiser.
I am working on better blending upstream in the main denoiser class but it's a little difficult to test. I'll setup something with this PR and a special version of the denoiser with the new tiling to test both systems out.
Speed. In general the speed is not a problem (especially on fast machines) but it can always be better. Initializing and tiling both lock up the thread. Upstream I am working on this so the denoiser will report progress of it's batches. This is a non blocking problem and one I'll continue to address.

Please take a look and let me know what you think.

Got the color pipeline to match.

I can take a look at the colors

If I recall correctly though this is the wrong order and the tonemapping should happen BEFORE we send to the denoiser for the best outcome

Do you have a reference for this? I would think ideally tone mapping is always applied before colors have been quantized into 8 bits to avoid banding but it shouldn't be a huge deal. We can take a look at this later I think.

... Transparency we'll have to test too.

We should be able to just set the mesh opacity and alpha maps to help with this. Transmission is more complicated because we can't correctly model the refraction that happens in that case.

Tiling seams. ...

Interesting - I haven't seen any tiling seems before so my assumption is that this is related to the denoiser. I'll try to keep an eye on this why looking at the color processing.

Speed. ... Initializing and tiling both lock up the thread

Avoiding the lock up I think would be great. The two things that come to mind are performing tiles over multiple frames and performing async material compile so other tasks can run while compiling is happening.
An event or flag that specifies how complete the denoising process is (just 0 to 1) would let us display it. And likewise a way to "cancel" the denoiser for cases where the camera moves, scene changes, etc would be nice.

I've discovered what is adding all the weird stuff to the package.json
Parcel polyfills when it finds node methods:

The code that starts all this is using node-fetch which I definitely don't. But searching I found in RenderVideos build file it mentioned. It was in a package for ffmpeg which is loaded by the package canvas-capture
So when parcel hit that package it started doing it's own thing...
It's a dev-dependency so doesn't add to the bundle size but it's annoying I get that.
Does it not do this when y'all build the package? It runs every time I use npm start

Made cleanup/minor changes
Denoiser has a more soft tiler but I now see the issue is only really visible in places the denoiser gets confused. For example the floor with 0 albedo but a reflective surface. Hiding the floor or setting its color to white (albedo white too) And the issue isn't visible.
Added a onProgress upstream and attached handlers to it as well as UI for the denoiser.
Abort will come tomorrow.
In general I'm happy with the colors. It's fast and matches the original.
Transparency will need to be resolved but we can spend time on that after merge.
Parcel is still weird...
Last little blocking issue IMO is there's a weird few pixel border being added. It's in the upstream one too so I'll spend time tomorrow chasing that down.

Weekend update:

Merged in @gkjohnson 's two PR's with much better Albedo & Normals handling.
Still some work to be done there with reflectance but I think that's something we'll be tweaking down the road.
These PR's removed the Split/Preview feature of the Denoise Demo. Not sure the best way to add them back. Easiest would be to put the split function in the clampedInterpolate, or add a dedicated split material. But I figured you might want to keep the library simple so I figured I make that material in the demo itself as a way to show how to get both outputs. Added to todo list.
Upstream: Added Abort to main class and Tiler.
Abort runs Async, but we can't stop a model in process but once it comes back from the GPU it does the "aborted" check. For now this works, although I think I'll add more abort checks through the process to potentially exit faster. But if the first build starts for example I can't stop it. It does stop the output so hopefully no more garbled outputs from an aborted execution.
Abort wired to reset() and fires the progress to 0. Not sure this is the right place to put it, but it seems to work.
Upstream: Added position specific masks and a mask cache. This resolves white borders on edges.
Upstream: Added even more complex smoother
Upstream: Increased standard overlap of tiles to 32 so more blending can happen

These upstream changes to the tiler are to resolve the borders & Seams seen in gradients of colors (particularly over bad albedos). Borders are gone, I might see a phantom of the grid  over areas with 0 albedos when in fast quality, but in balanced I no longer see any of the grid anywhere.

Removed old materials, added DenoiserSplitMaterial and loaded it in the Denoiser example to show how to split outputs from the denoiser in realtime.
I honestly think the ability to flip through different models to see what works best with the denoiser and what it struggles with is very helpful. Especially now until we can work out those finer details ( Like the ring and gelatinous cube)
If the goal of the denoiser example is to show off how to interface with and how the denoiser works I think it's probably the best it can be right now.
Let me know what you think left and I can work on it.
It would be great if we can get it merged this week, I'd like to work on the upstream examples/docs and the library will likely be moving to Poimandres but I don't want to do that until it's solid here.

I suppose we can leave it for now but all of these models can be checked in the main example with denoising and urls copied into the denoiser example with more options for dev. These example files have a significant maintenance burden whenever APIs change or need to be updated so keeping them simple is a priority for me.

For sure not trying to add burden, I'm just copying the examples that are already in use with the PBR example so if they change or get removed it's just as easy to remove them here. Yes you can run the denoiser in the PBR example, but you can't run the split comparison or look at how the transparent, reflective, metallic, etc objects generate albedos/normals etc. Instead of finding a super-model that tests everything in one go, it might be easier to use the same models we already load and see how they compare to each other.
Whatever you want do do though is fine with me. I can always make my own example on the main denoiser site with more models.

Normals are currently stored as sRGB which is atypical. Are we sure the normal image is supposed to be sRGB? The data stored in a normals texture is not considered color data and is usually loaded and read without color space transformations, even when reading from image files.

This is a mistake. This comes from battling color conversions. Regardless of the colorspace of inputs Normals should be in Linear. I will adjust this, The denosier doesn't actually use normals in the common sense, but this will better match training data.

I've mentioned it a few times but is it possible to precompile the tensor flow shaders with the KHR_parallel_shader_compile](https://developer.mozilla.org/en-US/docs/Web/API/KHR_parallel_shader_compile) extension to prevent locking the page?

It's actually implemented from what I can tell but it's burried in github discussions and not doccumented. I will experiment with this upstream.

I've noticed the denoiser can stall for quite awhile at 99% but it's not clear to me from profiling what's causing this. Are you seeing it, as well?

I am going to work on a flowchart of the denoiser so it's easier to explain, but right now the progress is only measuring the inference/tiling stage which executes with frame blocking to allow the callbacks. The next steps execute in a single step and theres no way to call back progress until it's totally done. The biggest delay is probably syncing from gpu to cpu.
Over time the upstream denoiser will get better at "progress" but for now I block at 99% waiting for the renderable WebGLTexture to return before finalizing with 100%. 100% is also a "done" marker at this stage too.

It looks like instantiating the Denoiser has a side effect enabling the scissor test for some reason:

const gl = renderer.getContext();
console.log( gl.getParameter( gl.SCISSOR_TEST ) ); // false
this.denoiser = new Denoiser( 'webgl', renderer.domElement );
console.log( gl.getParameter( gl.SCISSOR_TEST ) ); // true

Weird, I've added it to my list to put in the WebGLStateManager class (although I thought it was there)

I'm not sure if you saw this issue here - but it looks like once the desnoiser has started you can't perform any other draw calls without corrupting the denoised image.

Calling a draw call means Threejs takes over the state. I'm sure this messes things up. With the new abort flag the denoiser shouldn't output anything after the abort is called so if you need to draw, abort the denoiser and it should stop anything returning. It's a new feature so if it doesn't work right lmk and I'll try and test for it

White colors are still getting significantly blown out by the denoise process and it doesn't seem right? We wind up losing a lot of detail in brighter parts of the image for some reason. I'm not sure how difficult it is to test just passing the canvas directly into the denoiser again to see if it's causing the same problem? It looks like a color conversion may be happening somewhere causing us to clamp or lose precision or something.

Weird, I thought this got resolved with the correct tonemapping etc.
We can actually test pretty easily. using usepassThrough Honestly, I think this is probably an issue with all the color conversion s and working in SRGB not linear.
I've read a lot of people saying to send input in as SRGB but I think thats not actually the right way. OIDN itself defaults to Linear input, and I"ve actually dug in and found that when you use SRGB it converts the inputs to linear before processing and then reapplies SRGB to output.
Along with the normals I'll do some tests with a Linear version and I'll add a trigger for passthrough on the denoiser example (we can remove it later)
Hopefully it's just a colorspace thing. The next thing I could think would be an albedos/normal issue? One thing I did also try is using more samples. Using more samples seemed to stop the denoiser guessing so much on things..

Something in the recent commits has broken things...
When you rotate after denoise the denoised image turns to the blank canvas.

I'll try to go through the commits to find it

Ok the marking dirty is from removing the check if the size has changed. I will work on this upstream to protect against it.
However the blend not blending is still a problem. Both introduced in "some cleanup"

Something in the recent commits has broken things...
When you rotate after denoise the denoised image turns to the blank canvas.

Just push a line I forgot to commit - sorry about that. You're talking specifically about the denoiser.html example, right? I'm not seeing issues in the index.html page.

However the blend not blending is still a problem.

What does "blend not blending" mean?

Latest Version changes:

Upstream: Protect against same values marking dirty (stops denoiser rebuilding every run making it SIGNIFICANTLY faster)

Upstrream: Warmstart using Parallel shader compile

Upstream: WebGLStateManager improvements

Upstream: Cleanup

Upstream: Slight model change looking for speed improvement

Linear colorspace: All render targets (pathtracer, albedo, normals) now run in Linear colorspace no conversion to SRGB used.

Names changed to reflect operation not colorspace

Minor adjustments and fixes

Debug controls added for now (probably removed later)

I spent time with the models to check color/brightness. In general with the linear flow I like the output better but I need to check the lego model and see if it's doing the same thing.
My main check was colors, the floor, the sky

Color areas usually checked out.
I did notice if I zoomed out the NASA label got slightly bright, and when I checked the albedos at this level, it matched the higher brightness. So this is still one thing to keep messing with.
Regarding the state. I added restoring external state to the state manager. so it not only restores it's own state but the running state. This works fine with my original pathtracer setup, but this version it has conflicts every other execution. Luckily I added a "ignoreRestore" flag to the state manager and this seems to correct the every other WebGL error (instance draw without buffer) My guess is something in three changes and when the state restore is run it's out of date. Mixing webGL state is super fragile so for now I think this the best we can get.
Shader compiling/warm starting added, however, there is a minor speed improvement, but for the most part it seems it still waits to send data to the GPU util execution. There might be some exploration here but I don't suspect much gain.
I still get 20-30 second load times to compile the GPU Pathtracer shaders when first starting btw.
Minor model changes. My Mac version now executes 500-1000ms and my PC I've dipped below 1000 but that was probably a smaller size. most of the time my PC (3060) gets ~3500ms
I'm curious to see how WebGPU compares but I'll save that for another day.

Ok spent some time on the X-Wing..
This has a split slightly left of center. You can see the colors themselves are all matching correct so doubt it's a color space issue

Tested the normals and albedo and got this result:

My theory: The 100% white Albedo is being treated as a transparent surface. Combined with the fairly uniform normals it thinks it's a super reflective/transparent surface and is taking the albedo map as the color source.
I'm not sure what else it could be and it's something we can mess with. The colors in the scene are all correct looking including the backgrounds.
One thing to note, my normals are significantly different than the ones in the new pass.

A key to remember for the denoiser is not so much that normals are 100% light correct, but that I think they act as a segmentation mask helping find the edges of shapes.
This is all just a theory, but I tested the regular X-Wing which uses grey not white, and it renders fine.

Upstrream: Warmstart using Parallel shader compile
...
Shader compiling/warm starting added, however, there is a minor speed improvement, but for the most part it seems it still waits to send data to the GPU util execution.

Nice! Can you point to where this parallel compile is being done? If you attempt to use the shader after starting the parallel compile before it's done then the driver will still stall the page. The goal here is less of a speed improvement and more preventing the page from stalling.

I still get 20-30 second load times to compile the GPU Pathtracer shaders when first starting btw.

This is an issue with DirectX that's unavoidable. For some reason it takes forever to compile shaders. The Chrome / ANGLE team have said it's a known issue. The use of async compile should mean the page isn't completely locked while the shader compiles, though. This isCompiling flag is used in renderSample to prevent the path tracing material from being used until it's ready.

Ok spent some time on the X-Wing..

I took a look today and it seems to be due to clamping colors when converting from the Float texture to Uint8 target. This clamps and loses data above 1.0 used in the final tone mapping step which is why we're losing some of the bright highlights in the final image.
The solution is to either denoise the final tone mapped image or deoise the hdr path traced image and get an hdr result. Denoising the hdr image would be the ideal solution since it can be nice to adjust tone mapping, exposure, and post effects like bloom (which requires hdr anyway) without another denoising step. But I'm not sure how much work getting hdr working is.

One thing to note, in my other renderer, the normals are significantly different than the ones in the new pass.

The normals on the left are packed into the range [ 0, 1 ] - looking at the OIDN docs again it looks like they're supposed to be in the range [ -1, 1 ], which I must have misread previously:

Just like any other input image, the normal image should be anti-aliased (i.e. by accumulating the normalized normals per pixel). The final accumulated normals do not have to be normalized but must be in the [-1, 1] range (i.e. normals mapped to [0, 1] are not acceptable and must be remapped to [−1, 1]).

What format are we supposed to feed into the denoiser, then? The current format used for the normals texture will clamp the values of the output normals and negative values will just be black.
This is looking good, though. Looks like just a few final things.

The solution is to either denoise the final tone mapped image or deoise the hdr path traced image and get an hdr result. Denoising the hdr image would be the ideal solution since it can be nice to adjust tone mapping, exposure, and post effects like bloom (which requires hdr anyway) without another denoising step. But I'm not sure how much work getting hdr working is.

So, we can try working in HDR, but is that the output you want from the pathtracer? The denoiser will output HDR if we want it to. But I didn't know if that's what the pathtracer should send to the texture. We'll need to test a bit as I recall it had some issues. Otherwise I wonder if we can move the clamping to after the tonemap step.

What format are we supposed to feed into the denoiser, then? The current format used for the normals texture will clamp the values of the output normals and negative values will just be black.

I'm not sure what you mean exactly, right now I take a standard normal texture as input and re-map it. In my base renderer this seems to work ok.

Regarding HDR. I've had hit or miss results with it and found the standard def had better results. We can start working in HDR but wouldn't that make everything HDR or would we clamp it down at the end?

The goal would be to denoise in HDR then tone map the image to SDR. We could tone map before denoising to fix the color clamping but this way is more flexible and let's the user change the tone mapping without denoising and also let's you save a denoised HDR image. Right now the final result from the path tracer in a rendertarget is an HDR float buffer that is tone mapped when when written to the canvas.

Regarding Normals: I'll have to double check but I'm pretty sure I'm already remapping to [-1, 1] during the texture setting phase (I assume most users and shaders will use [0, 1])

What I'm wondering is what happens when a component is negative. If using an rgb8 texture (as the normal textures are here) any negative components will be clamped to 0 and there will be no distinction in normals. What's right thing to do here? Should we be using float textures? Does OIDN support that?

Have been following the denoising thread for quite some time, then the project went silent. Would make sense to merge this PR even if denoising is happening in SDR?

Have been following the denoising thread for quite some time, then the project went silent. Would make sense to merge this PR even if denoising is happening in SDR?

That's on me, I got busy with some other projects and lost the track on this one. I have a big post it note saying to get this done.
I'll make it a priority in a few weeks at the start of the year

Have been following the denoising thread for quite some time, then the project went silent. Would make sense to merge this PR even if denoising is happening in SDR?

That's on me, I got busy with some other projects and lost the track on this one. I have a big post it note saying to get this done.
I'll make it a priority in a few weeks at the start of the year

That would be great, and make the path tracer very useful indeed


Move "mongo.shell.path" to workspace settings

Guys you do a really good work) I'd like to suggest you making possible moving an option with "mongo.shell.path" to workspace settings (for today it works only from user settings).
E.g. for my case i prefer working with a local MongoDB inside docker container. Every project has docker-compose.yml files to build environment with mongo, redis etc. Working this way makes me to change user settings every time when switching between projects. Option with "mongo.shell.path" seems like:
{
    "mongo.shell.path": "docker exec -it mongodb mongo"
}

If the option is available from workspace settings, it could be more convenient, cause you can store different settings for different projects.

I ran into this in the functions extension. I'm guessing the fix is to change the scope to 'resource' instead of 'workspace.

FYI, we released 0.6.0 with this fix a couple of days ago.  Please let us know if you run into troubles.


[Request] Chillout VR Support?

If I try to use this tool in my CVR project, it seems to cause a compiler error. Can you make a version that is compatible, or have it check for presence of CVR CCK or VRC SDK?

It used to be compatible but I haven't really been following cvr updates. What's the error?

PR 93 wasn't ever included with a release package.

I get an error for LegacyDestroyer.cs(98,28): error CS0117: 'Helpers' does not contain a definition for 'DestroyMissingScriptsInGameObject'
That seems to be whats hanging things up

Will fix this tomorrow

Fixed in the next update


code for recommendation

Hi,
Thanks for sharing the code!
could you share the code for the recommendation too??
thank you

I'm also looking for the realse of your recommendation codes of GraphGAN


clip-video-encode: implement resuming failed runs

this is VERY hacky for now. It's going to require more work to make this be clean and work well. We need to implement input sharding because currently all videos are fed as a constant stream into the SharedQueue based on which finish reading first so it's hard to determine which videos are in which shard. I temporarily solved this by iterating over 'fake shards' by just indexing the full arrays


Using Emoji causes twice the memory usage on iOS

Scaffold(
backgroundColor: Colors.white,
appBar: AppBar(
elevation: 0.0,
title: Text('test'),
),
body: Container(
child: Text('test'),
));

Scaffold(
backgroundColor: Colors.white,
appBar: AppBar(
elevation: 0.0,
title: Text('点评讨论'),
),
body: Container(
child: Text('🎬'),
));

Hi @ldhlfzysys
Can you please provide your flutter doctor -v your flutter run --verbose and a complete reproducible minimal code sample
Thank you

[✓] Flutter (Channel stable, 1.22.3, on Mac OS X 10.15.5 19F96, locale zh-Hans-CN)
[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.1)
[✓] Xcode - develop for iOS and macOS (Xcode 12.0.1)
[✓] Android Studio (version 4.0)
[✓] VS Code (version 1.50.1)
[✓] Connected device (2 available)
• No issues found!

Text('🎬')

Can you provide more info about what kind of devices you have tested?

macos, iPhone 11 device and iPhone 11 simulator

All emoji have this problem

Hi @ldhlfzysys
I can reproduce the memory issue using other emojis as well

code sample
import 'package:flutter/material.dart';

void main() => runApp(MyApp());

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Material App',
      home: Home(),
    );
  }
}

class Home extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return Scaffold(
        backgroundColor: Colors.white,
        appBar: AppBar(
          elevation: 0.0,
          title: Text('点评讨论'),
        ),
        body: Container(
          child: Text('🔥'),
        ));
  }
}

flutter doctor -v
[✓] Flutter (Channel stable, 1.22.3, on Mac OS X 10.15.7 19H2, locale en-GB)
    • Flutter version 1.22.3 at /Users/tahatesser/Code/flutter_stable
    • Framework revision 8874f21e79 (5 days ago), 2020-10-29 14:14:35 -0700
    • Engine revision a1440ca392
    • Dart version 2.10.3

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.2)
    • Android SDK at /Users/tahatesser/Code/sdk
    • Platform android-30, build-tools 30.0.2
    • ANDROID_HOME = /Users/tahatesser/Code/sdk
    • Java binary at: /Applications/Android
      Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build
      1.8.0_242-release-1644-b3-6222593)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 12.1)
    • Xcode at /Volumes/Extreme/Xcode.app/Contents/Developer
    • Xcode 12.1, Build version 12A7403
    • CocoaPods version 1.10.0

[!] Android Studio (version 4.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    ✗ Flutter plugin not installed; this adds Flutter specific functionality.
    ✗ Dart plugin not installed; this adds Dart specific functionality.
    • Java version OpenJDK Runtime Environment (build
      1.8.0_242-release-1644-b3-6222593)

[✓] VS Code (version 1.50.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.16.0

[!] Connected device
    ! No devices available

! Doctor found issues in 2 categories.

[✓] Flutter (Channel master, 1.24.0-7.0.pre.73, on Mac OS X 10.15.7 19H2
    darwin-x64, locale en-GB)
    • Flutter version 1.24.0-7.0.pre.73 at /Users/tahatesser/Code/flutter_master
    • Framework revision fb90544cab (2 hours ago), 2020-11-02 22:48:03 -0800
    • Engine revision<PHONE_NUMBER>
    • Dart version 2.12.0 (build 2.12.0-3.0.dev)

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.2)
    • Android SDK at /Users/tahatesser/Code/sdk
    • Platform android-30, build-tools 30.0.2
    • ANDROID_HOME = /Users/tahatesser/Code/sdk
    • Java binary at: /Applications/Android
      Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build
      1.8.0_242-release-1644-b3-6222593)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 12.1)
    • Xcode at /Volumes/Extreme/Xcode.app/Contents/Developer
    • Xcode 12.1, Build version 12A7403
    • CocoaPods version 1.10.0

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build
      1.8.0_242-release-1644-b3-6222593)

[✓] VS Code (version 1.50.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.16.0

[✓] Connected device (4 available)
    • Taha’s iPad (mobile) • 00008020-000255113EE8402E • ios            • iOS
      14.1
    • macOS (desktop)      • macos                     • darwin-x64     • Mac OS
      X 10.15.7 19H2 darwin-x64
    • Web Server (web)     • web-server                • web-javascript •
      Flutter Tools
    • Chrome (web)         • chrome                    • web-javascript • Google
      Chrome 86.0.4240.183
    ! Error: Taha’s iPad is busy: Copying cache files from device. Xcode will
      continue when Taha’s iPad is finished. (code -10)

• No issues found!

Related https://github.com/flutter/flutter/issues/42586

@TahaTesser  Hi~ Is this problem improved in version 1.25?

How's fixing going, we have several new features blocked by this issue. REALLY NEEDS FIX!

urgently need to solve this problem，The Flutter engine takes up enough memory~

I found the reason for this problem in font_skia.cc,  in the gteTableSize(tag) method, the Apple Color Emoji's sbix table is too large，This sbix table contains all emoji png data, so the sbix table_szie have 120mb,
const size_t table_size = typeface->getTableSize(tag);
  if (table_size == 0)
    return nullptr;
  void* buffer = malloc(table_size);
  if (buffer == nullptr)
    return nullptr;

i also found the sbix table，just use to layout？only the methods is_color_bitmap_font and hb_font_get_glyph_extents(hbFont, glyph_ix, &extents) use the sbix table blob. is there any way to avoid malloc sbix table? @TahaTesser @AlexV525 @chinmaygarde

I found the reason for this problem in font_skia.cc,  in the gteTableSize(tag) method, the Apple Color Emoji's sbix table is too large，This sbix table contains all emoji png data, so the sbix table_szie have 120mb,
const size_t table_size = typeface->getTableSize(tag);
  if (table_size == 0)
    return nullptr;
  void* buffer = malloc(table_size);
  if (buffer == nullptr)
    return nullptr;

i also found the sbix table，just use to layout？only the methods is_color_bitmap_font and hb_font_get_glyph_extents(hbFont, glyph_ix, &extents) use the sbix table blob. is there any way to avoid malloc sbix table? @TahaTesser @AlexV525 @chinmaygarde

@jason-simmons for thoughts on this, and whether this issue will be affected by https://github.com/flutter/flutter/issues/39420.

@jason-simmons for thoughts on this, and whether this issue will be affected by https://github.com/flutter/flutter/issues/39420.

I looked into emoji font memory usage in Libtxt and SkParagraph.
Libtxt
Libtxt provides font tables to HarfBuzz through the GetTable callback in font_skia.cc.  This function allocates a buffer for the table and calls SkTypeface::getTableData to copy the data.
HarfBuzz APIs (such as the ones mentioned in https://github.com/flutter/flutter/issues/69566#issuecomment-755936926) may call the callback to request the SBIX table.  On my iPhone this table had a size of 57MB.
Most of that data is bitmaps for the emoji glyphs.  However, the buffer allocated for use by HarfBuzz is just used to obtain font properties during layout and is not used during painting.  Skia's Mac/iOS font implementation will use Apple font APIs to get the emoji bitmaps used for painting.
If I patch GetTable to return null for the SBIX table, then the emoji glyphs basically seem to render as expected in a simple test app.
I'm not sure what might go wrong if Libtxt did not provide this table to HarfBuzz/Minikin.  If there is data in the table that is required to do some layouts correctly, then Libtxt would need some other way to provide that data to HarfBuzz.
SkParagraph
By default SkShaper's create_hb_face will call SkTypeface::openStream and provide a copy of the font asset to HarfBuzz (including the SBIX table).  So SkParagraph's memory usage for the emoji font will be similar to Libtxt.
create_hb_face does have another code path where it will provide a callback to HarfBuzz that will copy tables out of the SkTypeface on demand (similar to how Libtxt works).  I tried forcing usage of this mode and found that the SBIX table was not loaded by HarfBuzz.  So this saved a significant amount of memory.
@bungeman What would happen if SkShaper always used hb_face_create_for_tables/skhb_get_table when building HarfBuzz faces?
Or would it make sense to have SkShaper do this selectively based on the size of potentially unneeded tables like SBIX in the SkTypeface?

I looked into emoji font memory usage in Libtxt and SkParagraph.
Libtxt
Libtxt provides font tables to HarfBuzz through the GetTable callback in font_skia.cc.  This function allocates a buffer for the table and calls SkTypeface::getTableData to copy the data.
HarfBuzz APIs (such as the ones mentioned in https://github.com/flutter/flutter/issues/69566#issuecomment-755936926) may call the callback to request the SBIX table.  On my iPhone this table had a size of 57MB.
Most of that data is bitmaps for the emoji glyphs.  However, the buffer allocated for use by HarfBuzz is just used to obtain font properties during layout and is not used during painting.  Skia's Mac/iOS font implementation will use Apple font APIs to get the emoji bitmaps used for painting.
If I patch GetTable to return null for the SBIX table, then the emoji glyphs basically seem to render as expected in a simple test app.
I'm not sure what might go wrong if Libtxt did not provide this table to HarfBuzz/Minikin.  If there is data in the table that is required to do some layouts correctly, then Libtxt would need some other way to provide that data to HarfBuzz.
SkParagraph
By default SkShaper's create_hb_face will call SkTypeface::openStream and provide a copy of the font asset to HarfBuzz (including the SBIX table).  So SkParagraph's memory usage for the emoji font will be similar to Libtxt.
create_hb_face does have another code path where it will provide a callback to HarfBuzz that will copy tables out of the SkTypeface on demand (similar to how Libtxt works).  I tried forcing usage of this mode and found that the SBIX table was not loaded by HarfBuzz.  So this saved a significant amount of memory.
@bungeman What would happen if SkShaper always used hb_face_create_for_tables/skhb_get_table when building HarfBuzz faces?
Or would it make sense to have SkShaper do this selectively based on the size of potentially unneeded tables like SBIX in the SkTypeface?

I'm confused. Using hb_face_create_for_tables will force a copy for each table loaded (and each time SkTypeface::copyTableData is called). hb_face_create itself should not create copies. When using an SkTypeface backed by a system CTFont the first time SkTypeface::openStream is called it will need to load that font into accessible memory and will keep hold of that data for future use. Is this the copy of sbix you're seeing? If so, this is something of a design tradeoff due to CoreText not providing an API to get at all of the font data (before SkTypeface::makeClone was added holding on to this data was really important, I'd have to check to see if that is no longer the case). If instead you're seeing HarfBuzz making copies of the data I'd be interested to know where.
Using hb_face_create_for_tables would be quite bad for the other systems like FreeType, DirectWrite, and even CTFonts created from data. These currently should all be zero copy.
There is information in the sbix table which should be used for correct layout. If you don't provide an sbix table HarfBuzz will just use whatever is in the glyf table which may or may not work in expected ways, depending on how the font is structured.
I'm not sure if it would help, but in theory Paragraph could be modified to use SkShaper::MakeCoreText. I'm not sure if that has been tried, but the CoreText shaper has system level access directly into the internal system font table cache and can avoid making any copies at all. Of course it is the system shaper, so will give slightly different results.

I'm confused. Using hb_face_create_for_tables will force a copy for each table loaded (and each time SkTypeface::copyTableData is called). hb_face_create itself should not create copies. When using an SkTypeface backed by a system CTFont the first time SkTypeface::openStream is called it will need to load that font into accessible memory and will keep hold of that data for future use. Is this the copy of sbix you're seeing? If so, this is something of a design tradeoff due to CoreText not providing an API to get at all of the font data (before SkTypeface::makeClone was added holding on to this data was really important, I'd have to check to see if that is no longer the case). If instead you're seeing HarfBuzz making copies of the data I'd be interested to know where.
Using hb_face_create_for_tables would be quite bad for the other systems like FreeType, DirectWrite, and even CTFonts created from data. These currently should all be zero copy.
There is information in the sbix table which should be used for correct layout. If you don't provide an sbix table HarfBuzz will just use whatever is in the glyf table which may or may not work in expected ways, depending on how the font is structured.
I'm not sure if it would help, but in theory Paragraph could be modified to use SkShaper::MakeCoreText. I'm not sure if that has been tried, but the CoreText shaper has system level access directly into the internal system font table cache and can avoid making any copies at all. Of course it is the system shaper, so will give slightly different results.

Yes - calling SkTypeface::openStream loads the entire font asset into memory (around 58MB for the Apple emoji font in my test setup).  Most of the 58MB is the contents of the sbix table.
The hb_face_create_for_tables approach avoids making a copy of sbix because HarfBuzz did not request that particular table in the SkShaper test scenarios that I tried.  I don't know if there are other cases in which HarfBuzz would need sbix.

Yes - calling SkTypeface::openStream loads the entire font asset into memory (around 58MB for the Apple emoji font in my test setup).  Most of the 58MB is the contents of the sbix table.
The hb_face_create_for_tables approach avoids making a copy of sbix because HarfBuzz did not request that particular table in the SkShaper test scenarios that I tried.  I don't know if there are other cases in which HarfBuzz would need sbix.

One can think of SkTypeface::openStream as being greedy but shared and SkTypeface::copyTableData as being lazy but unshared. For all fonts other than CoreText system fonts we already have access to the bytes so in those cases greedy works fine and also provides sharing. In theory SkTypeface::copyTableData could be shared (the typeface could hold on to the table data) or SkTypeface::openStream could be made less greedy.
Technically it would be possible to do something like have a stream backed by a userfaultfd like mechanism. This would give the best user experience since it would be both a flat memory backed view but only commit memory as needed. I don't know that macOS has a mechanism like that.
Short of that we might be able to internally signal if SkTypeface::openStream is inexpensive or not. We could swap over to hb_face_create_for_tables if it looks like it will be expensive (there isn't already a stream available and the created stream would need more than a megabyte for backing, or something like that).
Longer term we're working on a new OpenType color font format (COLRv1) which is vector and gradient based instead of bitmap based. While it isn't a short term solution, eventually we hope this font will no longer need to store gradient filled images at 2048x2048 in png format.

One can think of SkTypeface::openStream as being greedy but shared and SkTypeface::copyTableData as being lazy but unshared. For all fonts other than CoreText system fonts we already have access to the bytes so in those cases greedy works fine and also provides sharing. In theory SkTypeface::copyTableData could be shared (the typeface could hold on to the table data) or SkTypeface::openStream could be made less greedy.
Technically it would be possible to do something like have a stream backed by a userfaultfd like mechanism. This would give the best user experience since it would be both a flat memory backed view but only commit memory as needed. I don't know that macOS has a mechanism like that.
Short of that we might be able to internally signal if SkTypeface::openStream is inexpensive or not. We could swap over to hb_face_create_for_tables if it looks like it will be expensive (there isn't already a stream available and the created stream would need more than a megabyte for backing, or something like that).
Longer term we're working on a new OpenType color font format (COLRv1) which is vector and gradient based instead of bitmap based. While it isn't a short term solution, eventually we hope this font will no longer need to store gradient filled images at 2048x2048 in png format.

Longer term we're working on a new OpenType color font format (COLRv1) which is vector and gradient based instead of bitmap based. While it isn't a short term solution, eventually we hope this font will no longer need to store gradient filled images at 2048x2048 in png format.

@bungeman CLORv1 is very attractive, FreeType already supports it, can flutter use the latest FreeType to render?
In addition, does flutter now support CLORv0? I submitted a separate issue https://github.com/flutter/flutter/issues/79365

How's it going? I hope it can be solved ASAP~

Hope to be resolved too

Any update on this?

Also I'm not quite sure how CLORv0 is meant to resolve the issue with Apple Color Emoji, which is entirely bitmap based?

I fix it
this is my code
font_skia.cc

I fix it this is my code
font_skia.cc

could you create a pull request, i think it will help more people?

I fix it this is my code
font_skia.cc

could you create a pull request, i think it will help more people? @tinyc-z 💪

@jason-simmons @bungeman @knopp any thoughts about the patch in https://github.com/flutter/flutter/issues/69566#issuecomment-1005470784?

IIRC I tried similar approach where I just disregarded any SBIX table and it seemed to work. It didn't look like any of the font layout got broken. But I didn't test it thoroughly.

https://github.com/flutter/flutter/issues/69566#issuecomment-769575070 discusses the potential effects of ignoring the SBIX table.
The https://github.com/flutter/flutter/issues/69566#issuecomment-1005470784 patch affects font table loading in Libtxt, and we are planning to migrate to SkParagraph soon which uses a different code path.
For SkShaper/SkParagraph I'd be interested in exploring ways to avoid the SkTypeface::openStream call for expensive fonts and instead load tables on demand (see https://github.com/flutter/flutter/issues/69566#issuecomment-770036791)

I had a fix for this in mind where instead of ignoring the SBIX table we would process it, preserving layout information but stripping bitmap data (which is not used during rasterization). But this is not really my domain so perhaps there are better ways of solving this.

I think this one is fixed. Could we confirm that?

This has been fixed in SkParagraph.


nerian_sp1: 1.3.3-0 in 'jade/distribution.yaml' [bloom]

Increasing version of package(s) in repository nerian_sp1 to 1.3.3-0:

upstream repository: https://github.com/nerian-vision/nerian_sp1.git
release repository: https://github.com/nerian-vision/nerian_sp1-release.git
distro file: jade/distribution.yaml
bloom version: 0.5.21
previous version for package: 1.3.1-0

nerian_sp1
* Updated SP1 software release to version 2.1.6
* Contributors: Konstantin Schauwecker

trying to retrigger travis


Subdirectory Code Search

Issue Type: Feature Request
I use the seach feature all the time. It seems that it will only search in 1) the base directory of the project and 2) any other files opened in the editor.  It will not search subdirectories. for example, if i have a project called Test and it is saved in ~/Test and have most of the source code in ~/Test/src and open Code in ~/Test it will not search any files in ~/Test/src unless I already have all of them open in the editor. Not sure if this is a bug report or a feature request.
Thanks
VS Code version: Code 1.51.1 (e5a624b788d92b8d34d1392e4c4d9789406efe8f, 2020-11-10T23:34:32.027Z)
OS version: Windows_NT x64 10.0.18363
Remote OS version: Linux x64 4.4.0-18362-Microsoft
Remote OS version: Linux x64 4.4.0-18362-Microsoft

https://github.com/microsoft/vscode/wiki/Search-Issues


Fix collect tracking for EVM LP investors

sequenceDiagram
    box Purple Investment
    participant EVM as EVM Wallet
    participant CFG as CFG Wallet
    end
    box Green Pool
    participant IR as Invest/Redeem<br />Balance
    participant I as Invested
    end
    CFG ->> IR: InvestOrder
    CFG -->> IR: RedeemOrder
    note over IR,I: Epoch execution start
    activate I
    activate IR
    IR ->> I: InvestmentExecution
    I -->> IR:
    I ->> IR: RedemptionExecution
    IR -->> I:
    deactivate I
    deactivate IR
    note over IR,I: Epoch execution end
    alt if CFG Investor
        IR -->> CFG: InvestCollect
         IR -->> CFG: OrmlTransfer
    else if EVM Liquidity Pool
        IR -->> CFG: InvestCollect
        IR -->> EVM: EVMTransfer
    end

Introduce INVEST_LP_COLLECT and REDEEM_LP_COLLECT
Track userEscrow through (poolManager.investmentManager()).userEscrow()

INVEST_LP_COLLECT: EVM transfer from Escrow to investor
REDEEM_LP_COLLECT: EVM transfer from UserEscrow to investor
LP Event Payload:
poolId
trancheId
hash
timestamp
amount

Required for both invest and redeem collect tx
Complex because we cannot just rely on escrow => investor transfers, as this might also be the result of a tranche token transfer

REDEEM_LP_COLLECT: EVM transfer from UserEscrow to investor
Is moved to a follow up ticket #194


Create uber jar

As pushy-console is kind of a utility tool it would be nice to have it as one single file for easier sharing and running. I tried to make an uber jar with maven shade plugin but faced "Invalid signature..." errors due to signed BouncyCastle jars.
For now, I created a PR #15 that copies dependencies into one place and configures MANIFEST of pushy-console jar to make it executable as jar.

@bazi Could you clarify what would be different between an "uber jar" and what you've done in #15?

I found this: http://zhentao-li.blogspot.com/2012/06/maven-shade-plugin-invalid-signature.html
It feels a little like copying/pasting a magic voodoo spell into our POM, but it does get the job done. I'll open a pull request for more detailed discussion.


Add a method for getting a list of the defined groups

Would it possible to get back a list of the groups currently defined? Meaning, we could call something like FlowRouter.groups[ 'public' ] and get back all of the routes assigned to that group in an array? For example:
> FlowRouter.groups[ 'public' ];

[
  { name: 'login', path: '/login', ... },
  { name: 'signup', path: '/signup', ... }
]

If not already possible, could this be introduced in a future version?

I could assign this as a new feature. But we need to wait until 4.0 get released. Now FlowRouter is feature freezed, until we release 4.0

Cool, sounds good :)


GUI installer error: doesnt find common_constants.yaml

I tried to install the GUI on a windows machine.
The installer runs through and opens the command line at the end.
There, the error appears

I used the peptdeep-1.3.0-windows-amd64.exe

seems like the alphabase package was not bundled when building the installer

should be fixed by release 1.3.1: https://github.com/MannLabs/alphapeptdeep/releases/tag/v1.3.1


Swiper is adding 500KB to my bundle

This is a (multiple allowed):

[x] bug

[ ] enhancement

[ ] feature-discussion (RFC)

Swiper Version: 6.3.5.

Platform/Target and Browser Versions: Web.

What you did
Used ts-loader in Webpack to bundle my app with and without swiper imported.
Without: ~40Kb
With: ~550Kb
I'm using two extensions
import {Swiper, Pagination, Autoplay} from 'swiper';
Swiper.use([Pagination, Autoplay]);
Expected Behavior
The file size to be reasonable
Actual Behavior
My bundle size is unexpectedly big.

There is something wrong with your bundler config then. Fully compiled (and transpiled) not minified swiper bundle is 300Kb<EMAIL_ADDRESS>


Db migration fixes

Description
Fixed compatibility of existing db migration tests with new resource schema.
Self Check:
Strike through any lines that are not applicable (~~line~~) then check the box

[x] Attached issue to pull request
[x] Changelog entry
[x] Type annotations are present
[x] Code is clear and sufficiently documented
[x] No (preventable) type errors (check using make mypy or make mypy-diff)
[x] Sufficient test cases (reproduces the bug/tests the requested feature)
[x] Correct, in line with design
[x] End user documentation is included or an issue is created for end-user documentation (add ref to issue here: )

Processing this pull request

Merged into branches master, iso6 in 2f9a3ecdf6f03e89d92ee372fadd9db6df2bbc1e


fix(yaml): add missing formatting provider

ls.doFormat has another param with type CustomFormatterOptions, in this PR this is unused - I am not 100% sure about this.

Thanks for your contribution. We decided to not implement yaml-panguage-servers formatting API, because it’s just a poor wrapper around Prettier 2. If you want to format YAML code with Prettier, you can better use volar-service-prettier.


Wallpaper links?

Not an issue per se, but do you have links to the wallpapers?

Gravity falls
Second image


Change the front-page headline

Right now, this is what people see when they arrive at Binder:

Now that we support jupyter lab, RStudio, Julia, etc, I think that we should change the wording on the front page.
What do folks think about this? Any suggestions?

this has been fixed! it now reads "git repo"


Create edu.txt

Please, add miet/edu.txt to domains for getting VNL license for our university from website http://quantumwise.com/free-trial/free-vnl.

Hi @gddemin ,
I can't guarantee that the site in question uses this swot repo or even swot at all as their 'database'.
I will look into your eligibility for inclusion now, thank you for submitting

Looks good, merging

Thanks a lot, Peter! Hope it helps for our research! :)


ARXIVNG-551 update font sizes for result authors and abstract

Removed class is-size-7 from title and abstract; minor adjustments to style sheet.

Oh! Thanks Martin, will adjust to hotfix.


Parcel multiple targets with types workaround

Parcel can't export for multiple targets while emitting types for each, so I've made a workaround explained here: https://github.com/parcel-bundler/parcel/issues/8873#issuecomment-1596844972
See the issue itself here: https://github.com/parcel-bundler/parcel/issues/8082
The workaround is very hacky and we should probably use a different bundler.

Decided not to use this dirty solution.


Blocked syncing

This PR changes sync layer to only insert blocks into storage, not validate or attempt to commit them. After each sync round, a fast-forward is run. This eliminates high memory usage in testing, keep queue depth low for storage relatively, and should avoid putting too much pressure on the system. Lower sync pull rate is expected.

The snarkos-testing CI job fails, and it doesn't seem to be a fluke (a rerun didn't help). Other than that, I'm very much in favor of this change :+1:.

This should fix https://github.com/AleoHQ/snarkOS/issues/1176.

I'm still seeing some high memory usage but it's definitely lower than it was. Nodes don't need to be restarted as often to alleviate the issue.

@niklaslong I've observed in a node running for ~96 hours that it's been stable at 4.2 GB RSS for 25 min peers.

My long running node (started 17 hours ago) is showing 8GB currently, though the min peers is at 50. Perhaps I'm seeing https://github.com/AleoHQ/snarkOS/issues/1015.

My long running node (started 17 hours ago) is showing 8GB currently, though the min peers is at 50. Perhaps I'm seeing #1015.

Is the node syncing? Mine has been at canon tip for a few days.

That it is 👍

I filed https://github.com/AleoHQ/snarkOS/issues/1197 for an issue that might be introduced with this branch.


No information about managing U2F Authentication in Dashboard

Document: https://docs.wso2.com/display/IS550/Using+the+End+User+Dashboard#UsingtheEndUserDashboard-Editingyourprofile
There is no information available about U2F authentication in the doc under the topic of profile accessing. If there is any, please provide the reference.
Type/Docs
Affected/5.5.0-Alpha
Severity/Major
Priority/High

Added a tip to [1] with a reference for more info on U2F.
[1] https://docs.wso2.com/display/IS550/Using+the+End+User+Dashboard#UsingtheEndUserDashboard-Editingyourprofile


Failed to list *v1.Workflow: the server could not find the requested resource (get workflows.argoproj.io)

I'm trying to convert the Argo installation into a ksonnet config. My manifests are below. I've created the CRD but the workflow-controller is showing the following error
ERROR: logging before flag.Parse: E1228 05:50:42.630074       1 reflector.go:205] github.com/argoproj/argo/workflow/controller/controller.go:211: Failed to list *v1.Workflow: the server could not find the requested resource (get workflows.argoproj.io)

Any idea what could be causing this?
Here are the manifests for all the resources I'm deploying.
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: workflows.argoproj.io
spec:
  group: argoproj.io
  names:
    kind: Workflow
    listKind: WorkflowList
    plural: workflows
    shortNames:
    - wf
    singular: workflow
  scope: Namespaced
  version: v1alpha1
---
apiVersion: v1
data:
  config: 'executorImage: argoproj/argoexec:latest'
kind: ConfigMap
metadata:
  name: workflow-controller-configmap
  namespace: kubeflow-test-infra
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: workflow-controller-deployment
  namespace: kubeflow-test-infra
spec:
  selector:
    matchLabels:
      app: workflow-controller
  template:
    metadata:
      labels:
        app: workflow-controller
    spec:
      containers:
      - args:
        - --configmap
        - workflow-controller-configmap
        command:
        - /bin/workflow-controller
        env:
        - name: ARGO_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: IN_CLUSTER
          value: "true"
        image: argoproj/workflow-controller:latest
        name: workflow-controller
      serviceAccount: argo
      serviceAccountName: argo
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  labels:
    app: argo-ui
  name: argo-ui
  namespace: kubeflow-test-infra
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: argo-ui
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: argo-ui
    spec:
      containers:
      - env:
        - name: ARGO_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: IN_CLUSTER
          value: "true"
        image: argoproj/argoui:v2.0.0-alpha2
        imagePullPolicy: IfNotPresent
        name: argo-ui
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      serviceAccount: argo
      serviceAccountName: argo
      terminationGracePeriodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: argo-ui
  name: argo-ui
  namespace: kubeflow-test-infra
spec:
  ports:
  - port: 80
    targetPort: 8001
  selector:
    app: argo-ui
  sessionAffinity: None
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: argo
  namespace: kubeflow-test-infra
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: argo-cluster-role
  namespace: kubeflow-test-infra
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: argo
  namespace: kubeflow-test-infra

I was using the wrong docker image for the workflow controller; I wasn't using the v2 image.
Here are working manifests.
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: workflows.argoproj.io
spec:
  group: argoproj.io
  names:
    kind: Workflow
    listKind: WorkflowList
    plural: workflows
    shortNames:
    - wf
    singular: workflow
  scope: Namespaced
  version: v1alpha1
---
apiVersion: v1
data:
  config: 'executorImage: argoproj/argoexec:latest'
kind: ConfigMap
metadata:
  name: workflow-controller-configmap
  namespace: kubeflow-test-infra
---
apiVersion: extensions/v1beta1
kind: Deployment
labels:
  app: workflow-controller
metadata:
  name: workflow-controller
  namespace: kubeflow-test-infra
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: workflow-controller
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: workflow-controller
    spec:
      containers:
      - args:
        - --configmap
        - workflow-controller-configmap
        command:
        - workflow-controller
        env:
        - name: ARGO_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: argoproj/workflow-controller:v2.0.0-alpha2
        imagePullPolicy: IfNotPresent
        name: workflow-controller
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      serviceAccount: argo
      serviceAccountName: argo
      terminationGracePeriodSeconds: 30
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: argo-ui
  name: argo-ui
  namespace: kubeflow-test-infra
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: argo-ui
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: argo-ui
    spec:
      containers:
      - env:
        - name: ARGO_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: IN_CLUSTER
          value: "true"
        image: argoproj/argoui:v2.0.0-alpha2
        imagePullPolicy: IfNotPresent
        name: argo-ui
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      serviceAccount: argo
      serviceAccountName: argo
      terminationGracePeriodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: argo-ui
  name: argo-ui
  namespace: kubeflow-test-infra
spec:
  ports:
  - port: 80
    targetPort: 8001
  selector:
    app: argo-ui
  sessionAffinity: None
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: argo
  namespace: kubeflow-test-infra
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: argo-cluster-role
  namespace: kubeflow-test-infra
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: argo
  namespace: kubeflow-test-infra

Yes argoproj/argoexec:latest and argoproj/workflow-controller:latest tags are actually older than alpha2. We are actually long overdue for another version, which has a bunch of improvements. I think we can cut a new release by this week.

If you need earlier access to the resource templates (e.g. creating CRs as part of a workflow), or exit hooks, please use the following version for the time being: https://github.com/jessesuen/argo/releases/tag/v2.0.0-alpha3.jesse.ea1

Guys how come I get this error message now with :
Image: argoproj/workflow-controller:v2.4.2

github.com/argoproj/argo/workflow/controller/controller.go:157: Failed to list *v1alpha1.WorkflowTemplate: the server could not find the requested resource (get workflowtemplates.argoproj.io)

There is nothing like this in 2.4.2 of controller.go
I cannot figure out what is going on

@nrmitchi sorry it was a bug in helm chart https://github.com/argoproj/argo-helm/pull/138

Ah good to know. I didn't realize that the chart had started managing the CRD. I'll have to fix my deployment when the chart update is merged

I've had this problem before.
exec
kubectl create namespace argo
kubectl apply -n argo -f https://raw.githubusercontent.com/argoproj/argo/stable/manifests/install.yaml

then OK.


Material3 version update

While waiting for this to be merged, please this fork as a temporary dependency:
implementation 'com.github.NoelChew.ComposePreferences:preferences-material:0.1.5-5'


/articles: articles linked under "Or jump to a related question" should open in same tab

The context ("Keep reading!") means the user clicks on them as the next step in their reading journey, as opposed to being a tangent (as the inline links are).
Remember to remove the arrows.

@melissasamworth ?

Oh weird, I'm not sure how plurality (questions vs question) makes a difference there. I think this may be too small of a data point to make the change. Heh sorry I keep saying that. Just want more feedback and thinking of this round as MVP (not trying to be perfect because we still need more data to learn what perfect even is)


Slow queries on codespace attributes

Querying by codespace attributes is slow. I suggest to add an index to every codespace column (e.g. building.function, building.roof_type, ...). Please note that I mean the actual data column ('function'), not the codespace definition column ('function_codespace').

Well, you're free to create an index on all columns you want to filter on :)
I think, it's better to leave this up to the user. Every index requires disk space and - more important - affects write performance. So, we ship the 3DCityDB only with indexes on foreign key columns and spatial attributes (ok, and gmlid columns), which is a general recommendation for databases. One could argue that even some spatial indexes are not really necessary like the ones for terrain_intersection_curves.
In general, every database user should do some monitoring on his/her own to detect slow queries and create indexes.


Support non-static lifetimes for gap roles

With the introduction of GAP configuratino in #143, they only support 'static values for now. They should also support non-static lifetimes to allow passing in data read from configuration (for instance in flash).

I'm having a look at making the GattServer take ownership of the table atm

I think I've got it, will PR to show you now


chore: rename IsAlgClosure fields

to match the naming convention (rule 4).

Thanks! :tada:
bors merge


Remove website source code and docs folder in favor of byconity.github.io repo

I hereby agree to the terms of the CLA available at: [placeholder]
Changelog category (leave one):

Not for changelog (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
...
Detailed description / Documentation draft:

Remove website source code from this repository.
Website source code will be maintained at https://github.com/ByConity/byconity.github.io
...

If you are doing this for the first time, it's recommended to read the lightweight Contributing to ByConity Documentation guide first.
Close #170

Need to merge #152 first.

@pswai  Could you submit another commit to resolve the recent conflicting files?

@pswai  Please add the lasted docs files to the website repo, I think we can clean up these docs files now.


Create AsyncAPI data structures

[ ] Info Object
[ ] Servers Array
[ ] Security Object
[ ] Schema Object
[ ] Operation Object
[ ] Message Object
[ ] Security Schemes Object
[ ] Channel Parameters Object
[ ] Tags Object
[ ] External Docs Object
[ ] Channels Object
[ ] Components Object

is there a repo for this already ??

I'm taking care of the InfoObject for now.

Will the Go code be generated from the JSON Schema or written by hand?

I'm doing it manually now. Do you know some way to automate it? That would be awesome.

Looking at this repo: https://github.com/getkin/kin-openapi

I'm looking at https://github.com/atombender/go-jsonschema and https://github.com/a-h/generate. The last one works but doesn't perform any validation and doesn't support x- extensions. The first one is not yet there and can't even deal with additionalItems.

I was thinking of simply bootstrapping the boilerplate code using the existing node Codegen and new templates.

Oh! This is interesting. Want to give it a try yourself?

Yes if it can wait a day or two.

Oh sure, no rush 😄. I'll keep going with this to learn more about Go. Also, this code could serve you as a template.

Perfect.


Merge develop to master

Release

:tada: This PR is included in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Added space shortcut key to draw

This a fun little program! I found myself wanting to use the space bar to draw things, so I added it in in addition to the d key.
Feel free to reject the PR if you don't want it.

no issues, thanks for the PR :)


Deployer does not exit

Deployer version: Deployer 7.0.0-rc.2
Deployment OS: ubuntu impish

I use deployer to take backups as well as make deployments.
If my task starts a worker queue locally, deployer does not exit.
It only exits after I kill the worker process using another terminal.
Any way to allow starting a worker queue?

An example?

@antonmedv
Thanks for getting back, I tried to create a short example to narrow down the suspected culprit exec(), but was not successful in proving it.
Nonetheless, if you run core:test, it should exit quickly, however when I runLocally("php artisan core:test"), deployer does not exit. You will need Laravel in order to run my example. Sorry about this.
<?php

namespace App\Console\Commands;

use Illuminate\Console\Command;
use Illuminate\Support\Facades\Queue;

class CoreTest extends Command
{
    /**
     * The name and signature of the console command.
     *
     * @var string
     */
    protected $signature = 'core:test {--t|tries=1 : Number of times to check, useful if waiting for queue:worker to die}';

    /**
     * The console command description.
     *
     * @var string
     */
    protected $description = 'Ensure queue:work process is running';

    /**
     * Create a new command instance.
     *
     * @return void
     */
    public function __construct()
    {
        parent::__construct();
    }

    /**
     * Execute the console command.
     *
     * @return int
     */
    public function handle()
    {
        $tries = $this->option('tries');
        $job_count = Queue::size();
        $this->line("- There are <comment>$job_count jobs</comment> in the queue");
        $this->line("- Will check up to $tries time(s)");
        $path = base_path();

        $cmd = 'artisan queue:work --queue='.config('app.name').',default --tries=3';

        $pidfile = storage_path('app/queue/'.gethostname().'-work.pid');
        $stderr = storage_path('app/queue/stderr');
        $dirname = dirname($pidfile);
        if (! file_exists($dirname)) {
            mkdir($dirname, 0777, true);
        }

        $executed = false;
        while ($tries > 0) {
            $is_running = false;
            // $reason = "";
            if (file_exists($pidfile)) {
                $pid = file_get_contents($pidfile);
                $cleanedpid = strval(str_replace("\0", '', $pid));
                $fd = "/proc/$cleanedpid/cmdline";
                if ($cleanedpid != '') {
                    if (file_exists($fd)) {
                        // check that the running process cmdline matches what we want to run
                        $raw_cmd = file_get_contents("/proc/$pid/cmdline");
                        $running_cmd = trim(str_ireplace("\0", ' ', $raw_cmd));
                        if (stripos($running_cmd, $cmd) !== false) {
                            $is_running = true;
                        }
                    }
                }
            }
            if (! $is_running) {
                $this->line('- Starting queue:work process.');
                $this->line("- Running <comment>$cmd</comment>...");

                $command = "php -d register_argc_argv=1 {$cmd} >/dev/null 2>{$stderr} & echo $!";
                $number = exec($command);
                file_put_contents($pidfile, $number);

                $err = file_get_contents($stderr);
                if ($err != '') {
                    throw new \Exception($err);
                }
                $executed = true;
                break;
            }
            $tries--;
            sleep(1);
        }

        if (! $executed) {
            $this->error("- <comment>$cmd</comment> already running. Nothing to do.");
        }

        $this->info('- Exiting now');

        return Command::SUCCESS;
    }
}

task('test', function () {
    // the 3 lines below exit correctly, I'm not sure what the issue is now
    // $cmd = '-r "sleep(1000);';
    // $stderr = 'err.log';
    // runLocally("php -d register_argc_argv=1 {$cmd} >/dev/null 2>{$stderr} & echo $!");

    // the 3 lines below will not exit even though core:test has completed
    runLocally('php artisan queue:restart');
    sleep(3);
    runLocally('php artisan core:test');
});

OUTPUT
task test
[live] run locally php artisan queue:restart
[live] Broadcasting queue restart signal.
[live] run locally php artisan core:test
[live] - There are 0 jobs in the queue
[live] - Will check up to 1 time(s)
[live] - Starting queue:work process.
[live] - Running artisan queue:work --queue=App,default --tries=3...
[live] - Exiting now

I think this is because exec is never returned and blocking Deployer task.

@antonmedv exec() does return, see [live] - Exiting now

Yes, I see. For some reason runLocally thinks its not done. The rubLocally is actually just fancy wrapper around Symfony/Process. Can you try using it directly?

What is it that I should try? Call Symfony/Process myself? How about I try exec('php artisan core:test')?

Ok, exec('php artisan core:test'); did not work as well.

Any idea how to tackle this? Thank you.

If I remember correctly, & only forces background. Unfortunately it is still attached to the ssh session, so if you disconnect, the process is killed.
the only solution for this I know is screen

That means, you either don’t & and wait for the process‘ end or use screen, but have an endless running screen session + process

@Schrank
Do you know location of the code that is waiting for the process to end?
I tend to think that restarting the queue worker should be considered as part of deployment, just like bringing down and up the website.

I do only assumptions (!!)
Here is what I think:

Deployer is opening an ssh connection
Starting a process (or more than one)
Waiting for the ssh connection to be process free
Close it

and this is not deployer explicitly. It is deployer waiting for a „free“ ssh connection and the way & works it doesn’t happen.
So I guess the better way (cents just dropped here 😅) is to run your workers via serviced or systemd and call it to restart them. I would assume then the ssh connection is „free“ because systemd/serviced isn’t attaching them to ssh.
assumptions over assumptions. Hope it helps!

So what's the status? Let's close it for clean-up purposes.
If it is still the problem, let's reopen the issue.


Link errors when using amqp-cpp.lib from Nuget

I used Nuget in Visual Studio 2015 to install the x64 <IP_ADDRESS> and even <IP_ADDRESS> versions on Windows 10 but I get link errors:
1>amqp-cpp.lib(tcpconnection.obj) : error LNK2001: unresolved external symbol "int __cdecl AMQP::tcp::Errno(void)" (?Errno@tcp@AMQP@@YAHXZ)
1>amqp-cpp.lib(tcpconnection.obj) : error LNK2001: unresolved external symbol "char const * __cdecl AMQP::tcp::StrError(int)" (?StrError@tcp@AMQP@@YAQEBDH@Z)
1>amqp-cpp.lib(tcpconnection.obj) : error LNK2001: unresolved external symbol "int __cdecl AMQP::tcp::Close(unsigned __int64)" (?Close@tcp@AMQP@@YAH_K@Z)

I made sure that path to the lib is added to the project, but I still get the link errors.
I've uploaded the project with the package to here.
I've also uploaded the dump of the lib and I can see that indeed those symbols are missing (there is UNDEF near those symbols) from the lib:
1B4 00000000 UNDEF  notype ()    External     | ?Errno@tcp@AMQP@@YAHXZ (int __cdecl AMQP::tcp::Errno(void))
1B5 00000000 UNDEF  notype ()    External     | ?StrError@tcp@AMQP@@YAQEBDH@Z (char const * __cdecl AMQP::tcp::StrError(int))
1B6 00000000 UNDEF  notype ()    External     | ?Close@tcp@AMQP@@YAH_K@Z (int __cdecl AMQP::tcp::Close(unsigned __int64))

Is there another lib that I can download?

I having the same issue, any suggestion?

I am having the exact issue as well nuget package <IP_ADDRESS>.


[#50] [API] As a user, I can Login with a Gmail Account

close #50

What happened 👀
Registered user with Google Account registration can login with a Google Account and receive a user token.
Insight 📝

Return token from Doorkeeper::Application in OAuth callback.
Move redirect_url to ENV.
Add allow_forgery_protection to skip CSRF check for APIs register/login.

Proof Of Work 📹

@byhbt @malparty Thank you for the review on this PR. Please help review the latest update on this commit with Production working correctly. https://github.com/nimblehq/ic-rails-phong-bliss/pull/71/commits/5360b103e3fd7d7b789ed13dcc0b98f9b06e38ca 🙏


Use key and factors methods instead of direct RSA accessors

Please note that while we are transitioning from travis-ci to AWS CodeBuld, some tests are run on each platform. Non-AWS contributors will temporarily be unable to see CodeBuild results. We apologize for the inconvenience.
Issue # (if available): N/A
Description of changes:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report

:exclamation: No coverage uploaded for pull request head (rsa-accessors@b84c74d). Click here to learn what that means.
The diff coverage is n/a.


Only export environment with aenv.

No longer export the AWS environment variables in the current shell session. To export the environment to aws commands run the new shell wrapper aenv before executing commands like the EC2 CLI, awscli, etc. The awsam commands assh and ascp don't require any changes, as they will correctly load the environment as needed.
This is to protect against stuff like: https://iamakulov.com/notes/npm-malicious-packages/

Could keep the creds in a ~/.aws/credentials file and just have it export AWS_DEFAULT_PROFILE instead.

@rsalmond So it will export a credentials file too, but I think there are still older utilities (ec2cli) that only accept the KEY/SECRET environs.

@mheffner I just mean that in terms of exporting sensitive ENV vars there's no danger in exporting env var for the name of the profile. It makes sense to add an extra step to export KEY/SECRET but as you say most newer utils would still provide the current AWSAM experience of just working if the profile variable is exported by default.


Additional spanner integration tests.

For spanner GA.  Investigate whether these are already tested, and add tests as appropriate.  Link either PR # or line reference of existing test.

[x] Read single key
https://github.com/GoogleCloudPlatform/google-cloud-python/blob/6c602823e7101b40dfe5bfc614da633beb8accfb/spanner/tests/system/test_system.py#L840-L850

[ ] Read single key w/ index

[ ] Range read w/ single key

[ ] Range read w/ partial key

[ ] Range read w/ open/open range
https://github.com/GoogleCloudPlatform/google-cloud-python/blob/6c602823e7101b40dfe5bfc614da633beb8accfb/spanner/tests/system/test_system.py#L879-L906

[ ] Range read w/ open/closed range
https://github.com/GoogleCloudPlatform/google-cloud-python/blob/6c602823e7101b40dfe5bfc614da633beb8accfb/spanner/tests/system/test_system.py#L879-L913

[ ] Range read w/ closed/open range
https://github.com/GoogleCloudPlatform/google-cloud-python/blob/6c602823e7101b40dfe5bfc614da633beb8accfb/spanner/tests/system/test_system.py#L879-L899

[ ] Range read w/ closed/closed range
https://github.com/GoogleCloudPlatform/google-cloud-python/blob/6c602823e7101b40dfe5bfc614da633beb8accfb/spanner/tests/system/test_system.py#L879-L892

[ ] Bind string to null

[ ] Bind timestamp

[ ] Bind date

[ ] Bind string array

[ ] Bind empty string array

[ ] Query returning array of -Inf, +Inf, Nan

[ ] Write / read back null string

[ ] Write / read back empty array of string

[ ] Write / read back null array of string

[ ] Write / read back array of string with mixed null / non-null values

/cc @lukesneeringer
Alan: particular commit used because I'm not sure if master will shift the line number around.  Update in Progress

@chemelnucfin Thanks for the updates!  As we each begin adding one or more related tests, let's adopt the covention of marking the checkbox items with our handles (to avoid duplicating effort).

@chemelnucfin I had to uncheck some of the boxes :( -- the test description was missing "over index".

What is range read with partial key?

Never mind, got it.

Closed via PRs from above


chore: remove unused file from #1433

Deploy preview for netlify-cms-www ready!
Built with commit 0ae5bcd15945a35b8c38bc320d0c2da920a91b32
https://deploy-preview-1661--netlify-cms-www.netlify.com

Deploy preview for cms-demo ready!
Built with commit 0ae5bcd15945a35b8c38bc320d0c2da920a91b32
https://deploy-preview-1661--cms-demo.netlify.com


[feature request] Specific action on right click, double click

First congrats for the job: very useful plugin
Would be great to be able to configure specific behaviors on right click or double click.
For example something that would be very helpful for me would be to have:

full screen display on double click,
then going back to obsidian with a second double click

Cheers

This is a good suggestion one I like! put my vote

Same request here~, I would love to leave the single click as it is in Obisidian, and using double click to enter full screen display.

@laurentbonet  @accept82  @PandaraWen
Now, you can enter full-screen mode by clicking icon on the toolbar, and then, you can go back by clicking anywhere, and then exit previewing image by clicking non-image area.
I'll think about entering full-screen by double click.

@laurentbonet  @accept82 @PandaraWen
Hi you guys, it has added double click setting in latest version. The default function for double click is 'full screen'. You can upgrade to v1.3.1 and have a try.


fix(controller): Correct the order merging the fields in WorkflowTemplateRef scenario. Fixes #4044

Checklist:

[ ] Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this is a chore.
[ ] The title of the PR is (a) conventional, (b) states what changed, and (c) suffixes the related issues number. E.g. "fix(controller): Updates such and such. Fixes #1234".
[ ] I've signed the CLA.
[ ] I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
[ ] My builds are green. Try syncing with master if they are not.
[ ] My organization is added to USERS.md.
Fixes #4044

v2.11 backport is done


Enable use of RenderedContentInterface and small fix

hey @freekmurze - this change makes a quick fix for the 2.0 release related to extensions to help ensure things work properly. There were parts of the initial changes I made that were not covered by tests to catch the logic issue.
Quickfix
Make sure extensions can be given as class names (strings) or instances. So i've made adjustments to ensure extension load properly.
Specifically these lines would be affected:
https://github.com/spatie/laravel-markdown/blob/991ff79926e3eb6667ed0c2e6793a63825516cf6/src/MarkdownRenderer.php#L149
https://github.com/spatie/laravel-markdown/blob/991ff79926e3eb6667ed0c2e6793a63825516cf6/src/MarkdownRenderer.php#L78
New feature
This also adds a method that mimics the league convertToHtml method for users. Adding this method and covering it with tests was what caused me to encounter the error to begin with. Otherwise I would have opted to split them up.
This addition is important for users that want the FrontMatterExtension (or similar). Since these shouldn't return a string directly like existing methods do. So the new method returns a RenderedContentInterface object specific to the renderer. For instance, they can use any potential extension specific results. Like:
$result = app(\Spatie\LaravelMarkdown\MarkdownRenderer::class)->renderMarkdown($markdown);

// Grab the front matter:
if ($result instanceof RenderedContentWithFrontMatter) {
    $frontMatter = $result->getFrontMatter();
}

Thank you very much!


iOS Add2App profile mode can't attach/find observatory

flutter run --profile on a iPhone allows you to attach and use the observatory in a normal way.
However, running in a profile mode add2app scenario does not.

For some reason, it looks like DartServiceIsolate:NotifyServerState isn't getting called in this mode.

@dannyvalente - I was able to get this working eventually by what I suspect was some mix of cleans and restarts.  This makes me suspect that we're caching some build artifact incorrectly.
Can you try to delete the build directory in your Flutter application, and run things in profile mode again and see if it works?  If not, there's something else going on here that I'm missing.

@dnfield, thanks for the tip. I tried that and I think you're right as now I see the app outputting that the observatory is listening, however flutter attach fails to attach every time with the following:
Waiting for a connection from Flutter on Danny's iPhone 6...
Done.
Syncing files to device Danny's iPhone 6...
Error -32601 received from application: Method not found

Well that's progress at least.  So one issue is that we're not properly doing a rebuild (or just not doing a clean?) when we change modes in the add2app case.
For the method not found, does that consistently reproduce?  Do you get anything more if you run the application in verbose logging mode?

In this case flutter doctor -v?
I've disabled all of the fingerprinting done by the tool itself

Yes i can reproduce this consistently.
How do you run in verbose logging mode? Is this documented in the add2app documentation?

Ahh sorry, I keep forgetting - for add2app it should just be verbose already.
Do you get any interesting output in the Xcode console when you see the method not found in your attach window?

No I checked that and there was no output after it says that the observatory is listening. There were no exceptions or any output at all.

I haven't been able to reproduce this.  Are you able to reproduce it with a "fresh" add2app setup?

@dnfield Does this help? After the initial error, I hit the Hot Reload button in Android Studio, and this output came up. (Still nothing in the XCode output console though).
Initializing hot reload...
Syncing files to device Danny's iPhone 6...
Error 100 received from application: Feature is disabled
Unhandled exception:
{code: 100, message: Feature is disabled, data: {request: {method: _reloadSources, params: {pause: true, rootLibUri: file:///private/var/mobile/Containers/Data/Application/ACDEAD4D-D07B-489F-A8F0-174888788C48/tmp/flutter_moduleQZxSIn/flutter_module/lib/main.dart.incremental.dill, packagesUri: file:///private/var/mobile/Containers/Data/Application/ACDEAD4D-D07B-489F-A8F0-174888788C48/tmp/flutter_moduleQZxSIn/flutter_module/.packages, isolateId: isolates/865762522}}, details: Compiler is disabled in AOT mode.}}
#0      _rootHandleUncaughtError. (dart:async/zone.dart:1112:29)
#1      _microtaskLoop (dart:async/schedule_microtask.dart:41:21)
#2      _startMicrotaskLoop (dart:async/schedule_microtask.dart:50:5)
#3      _runPendingImmediateCallback (dart:isolate/runtime/libisolate_patch.dart:115:13)
#4      _RawReceivePortImpl._handleMessage (dart:isolate/runtime/libisolate_patch.dart:172:5)
I plan on trying with a newer version of Flutter to see if it helps. (I'm using Flutter 1.2.1 here)

Hot reload isn't supported in profile.  That's expected.

Can you try flutter attach --profile?

That seems to work.
flutter_module danny.valente$ flutter attach --profile
Waiting for a connection from Flutter on Danny's iPhone 6...
Done.
An Observatory debugger and profiler on Danny's iPhone 6 is available at http://<IP_ADDRESS>:1025/
For a more detailed help message, press "h". To detach, press "d"; to quit, press "q".
And in the observatory, I can get some information. I can't seem to get the CPU profile or metrics views to do much though. Is there something you want me to look at to to see that we're good? I can see the timeline view, which is good.

Woohoo!
/cc @liyuqian @cbracken - they'd have more information on using that to create a Skia trace.
This bug should stay open until we properly clean the assets between builds for Add2App.

Is there a way we could get this to work with Android Studio though? Or is there no way to get the Widget Inspector to work on profile builds? I suppose it's not something we should be trying to do with a profile build anyway...

Thanks for the help @dnfield by the way!!

@jmagman - is this of interest to you?
The root cause appears to be that add to app use cases on iOS are not properly clearing out old build artifacts between builds (or perhaps when switching build modes e.g. from debug to profile).

I can take a look!

The MyApp/my_flutter/.ios/Runner.xcodeproj Profile build configuration is using the Config/Release.xcconfig as its base, which still contains FLUTTER_BUILD_MODE=release instead of dropping back to parse the build config as "Profile".  So in my testing the observatory never even started when the host was running a Profile build config.
See:
https://github.com/flutter/flutter/blob/master/packages/flutter_tools/templates/module/ios/host_app_ephemeral/Config.tmpl/Release.xcconfig
https://github.com/flutter/flutter/blob/master/packages/flutter_tools/templates/module/ios/host_app_ephemeral/Runner.xcodeproj.tmpl/project.pbxproj.tmpl#L261
https://github.com/flutter/flutter/blob/master/packages/flutter_tools/bin/xcode_backend.sh#L62
https://github.com/flutter/flutter/pull/23387/files#diff-0d00f38df3883291f23cee8074ac075d.
However that's not getting me to the "Method not found" stage since flutter attach can't even connect to the observatory (that error message means the method is missing from the VM service).  When I fix that problem, I'm able to flutter attach and take screenshots swapping back and forth from debug and profile.  So I wasn't able to reproduce the exact reported issue.

I created https://github.com/flutter/flutter/issues/35298 to track the issue that the observatory doesn't start.  We can keep tracking the "Method not found" issue here.

The MyApp/my_flutter/.ios/Runner.xcodeproj Profile build configuration is using the Config/Release.xcconfig as its base, which still contains FLUTTER_BUILD_MODE=release instead of dropping back to parse the build config as "Profile".

This was a red herring since the host app was correctly passing the --profile flag to Flutter run.  See https://github.com/flutter/flutter/issues/35302.


Don't delete reservation without re-adding it

@bratseth please review and merge.

For the record:
Hehe, the test which was added to verify this did a mutation -- supposed to be idempotent -- twice, exactly to ensure its idempotency, but this was exactly what made the bug sneak past, since the second mutation corrected the error made in the first.

Thanks! Will update the test to cover this bug.


Upload types?

Hi,
This is not an issue, but more of a question of functionality.
Does this combination of the facebook-chat-api and fbash allow for the uploading of other attachments?
I'm considering this for uploading small mp3 files.
Just curious if you know before I go through the process of installing it.
Cheers,
Steve

Hi Steve,
You can currently use the savefile command with any non-image filetype. Here's an example with a .mp3 file:

After uploading your file to Messenger, just run savefile <filename> to save it in fbash's current directory with that filename. Let me know if this answers your question.

Fantastic. I was wanting to use a system like this (I may have to modify it slightly) to be able to automatically upload files and use the FB platform as the delivery platform. It'll just be a one way system at this stage. Thanks.

Hi Avikj,
I managed to get it working and I can get a file to come through to me, but I can't save it on the computer I'm issuing the commands from.
So while it's an audio file, it's coming through as an audio file with a play button.
However, I've also discovered that if I just zip the file and then send it, it comes through as a downloadable attachment at the browser end.
While I know you know how all these things work, it may be worth adding the requirements to getting the system working. eg, the facebook-chat-api, the latest version of nodejs and even possibly git? Just a thought.
Cheers,
Steve

All sorted.