Force static shapes through the graph?

Trying to convert a model to tflite for the Coral TPU, which only supports static shapes. Is there any way to force static shapes throughout the whole graph?
P.S. Magnificent work! Thanks!

Hi, thanks! Strictly speaking, TFLite only supports static shapes. I never tried Coral, but think it should be fine.

@AlexanderLutsenko when I inspected the converted model with the https://github.com/google-ai-edge/model-explorer , it indicated shapes of size -1, which I presume are dynamic. Am I doing something wrong?

Was it input tensor that had size -1? That should never occur unless you annotate some dimensions as dynamic in input_shapes argument. As for intermediate/output tensors, the result of certain ops has to be dynamic. For example:
class FilterBoxes(nn.Module):
    def __init__(self):
        super().__init__()
        self.threshold = 0.1

    def forward(self, boxes, scores):
        mask = scores > self.threshold
        return boxes[mask], scores[mask]

boxes = torch.normal(0, 1, size=(128, 4))
scores = torch.normal(0, 1, size=(128,))
pytorch_module = FilterBoxes().eval()

keras_model = nobuco.pytorch_to_keras(
    pytorch_module,
    args=[boxes, scores],
)

"1" in tensor shapes should be "-1",  Netron visualizes it wrong

Obviously, we do not know in advance how many scores will be greater that threshold. To avoid dynamic shapes, gt is usually replaced with topk:
    def forward(self, boxes, scores):
        top_scores, top_indices = torch.topk(scores, k=10, dim=-1)
        top_boxes = boxes[top_indices]
        return top_boxes, top_scores

That's about how much I can tell without looking at your model. If that's not enough, feel free to share the part of it that causes problem.

Hi!
I have a similar problem but it may just be a limitation of tensorflow or tflite.
I have a 2-stage CNN in pytorch where the first stage uses a scaled down image to predict the center of a crop. The second stage then takes the crop in full resolution as input to make a prediction. Both the images and crops are of fixed size (640x480 and 128x128)
I convert the model from pytorch to tflite and want to run it on a Android phone GPU. Everything works fine on my computer, I get identical output from pytorch and tflite. To make it run on Android GPU I had to make some changes to the cropping, and It now looks a bit hacky but the tf.lite.experimental.Analyzer.analyze() function says it is GPU comparable. However, when I try to run it on Android GPU it crashes and says that the size of the crop is dynamic and it only supports static-sized tensors.
I can see why it thinks the size is dynamic but it really isn't, can I convince it that is is static?

Simplified example code
    ```python
"""Convert CNN from pytorch to tflight.

Requirements:
- nobuco
- tensorflow-cpu==2.15.0

Recomended VS Code extensions:
- HTML Preview: for displaying nobuco conversion output
- vscode-netron: for displaying output graph
"""

import numpy as np
import torch
from torch import nn
import nobuco
import tensorflow as tf
from nobuco import ChannelOrder

class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.stage_1_conv = nn.Conv2d(3, 4, kernel_size=8, stride=8)
        self.stage_1_relu = nn.ReLU()
        self.stage_1_linear = nn.Linear(4 * 20 * 15, 2)
        self.stage_1_sigmoid = nn.Sigmoid()

        self.stage_2_conv = nn.Conv2d(3, 4, kernel_size=8, stride=8)
        self.stage_2_relu = nn.ReLU()
        self.stage_2_linear = nn.Linear(4 * 16 * 16, 5)
        self.stage_2_tanh = nn.Tanh()

    @torch.no_grad()
    def forward(self, image: torch.Tensor) -> torch.Tensor:
        # Cast and normalize image (not GPU compatible, but that is fine)
        image = image.float() / 255

        ##### Predict crop center (stage 1) #####
        x = self.stage_1_conv(image[:, :, ::4, ::4])
        x = self.stage_1_relu(x.flatten(start_dim=1))
        x = self.stage_1_linear(x)
        rel_crop_center = self.stage_1_sigmoid(x)

        ##### Extract crop #####
        image_size = torch.tensor((640, 480), dtype=torch.int32)
        half_crop_size = torch.tensor((64, 64), dtype=torch.int32)
        crop_center = (rel_crop_center * image_size).to(torch.int32)

        # Crop has to be within frame
        crop_center = torch.clamp(crop_center, half_crop_size, image_size - half_crop_size)

        # Hacky way to crop since TFLite GPU only supports slicing in 4 dimensions
        # and the only way I could make scalar tensor was with `min()`
        tl = crop_center - half_crop_size
        br = crop_center + half_crop_size
        top = tl.view(2, 1, 1, 1)[0:1, :, :, :].min()
        left = tl.view(2, 1, 1, 1)[1:2, :, :, :].min()
        bottom = br.view(2, 1, 1, 1)[0:1, :, :, :].min()
        right = br.view(2, 1, 1, 1)[1:2, :, :, :].min()
        crop = image[:, :, top:bottom, left:right]

        ##### Predict full resolution crop (stage 2) #####
        x = self.stage_2_conv(crop)
        x = self.stage_2_relu(x.flatten(start_dim=1))
        out = self.stage_2_tanh(self.stage_2_linear(x))

        return out

# Test pytorch model
print("=" * 20, "Pytorch model", "=" * 20)
pytorch_module = MyModel().eval()
dummy_input = torch.randint(0, 256, size=(1, 3, 640, 480), dtype=torch.uint8)
dummy_output = pytorch_module(dummy_input)
print("Input:  ", dummy_input.shape, dummy_input.dtype)
print("Output: ", dummy_output.shape, dummy_output.dtype)

# Convert the model with nobuco to keras.
print("=" * 20, "Nobuco conversion", "=" * 20)
keras_model = nobuco.pytorch_to_keras(
    pytorch_module,
    args=[dummy_input],
    inputs_channel_order=ChannelOrder.TENSORFLOW,
    outputs_channel_order=ChannelOrder.TENSORFLOW,
    save_trace_html=True,
)

# Convert the model to tflite.
print("=" * 20, "TFLite conversion", "=" * 20)
converter = tf.lite.TFLiteConverter.from_keras_model(keras_model)
tflite_model = converter.convert()

# Save the model.
with open("mymodel.tflite", "wb") as f:
    f.write(tflite_model)

# Analyze model
print("=" * 20, "Model analysis", "=" * 20)
tf.lite.experimental.Analyzer.analyze(model_content=tflite_model, gpu_compatibility=True)

# Test run tflite model
print("=" * 20, "TFLite model test", "=" * 20)
interpreter = tf.lite.Interpreter("mymodel.tflite")
my_signature = interpreter.get_signature_runner()
output = my_signature(args_0=tf.constant(dummy_input.permute(0, 2, 3, 1).numpy()))
pred = list(output.values())[0]
assert np.allclose(np.array(pred), np.array(dummy_output)), "Model output mismatch!"

```

@johan-sightic Well, that's unfortunate. But I see why TFLite fails to recognize the crop shape as static. Consider this:
y0 = center_y - h
y1 = center_y + h
crop = image[:, y0: y1]

As center_y and center_y are tensors, so are y0 and y1. When you invoke image[:, y0: y1], the indexing operator ([]) has no way of knowing that y0 and y1 are related and y1 - y0 = const. It wouldn't be a problem if the op accepted (center_y, h) as cropping parameters instead of (y0, y1). If you are to assemble such an op from existing Tensorflow primitives, brace yourself for some real jank:
@nobuco.traceable
def get_crop(x, center_y, center_x, h, w):operator
    return x[:, :, center_y - h:center_y + h, center_x - w:center_x + w]

@nobuco.converter(get_crop, channel_ordering_strategy=nobuco.ChannelOrderingStrategy.FORCE_TENSORFLOW_ORDER)
def converter_get_crop(x, center_y, center_x, h, w):
    def func(x, center_y, center_x, h, w):
        x = tf.roll(input=x, shift=-(center_y - h), axis=1)
        x = tf.roll(input=x, shift=-(center_x - w), axis=2)
        return x[:, :h*2, :w*2]
    return func

class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.stage_1_conv = nn.Conv2d(3, 4, kernel_size=8, stride=8)
        self.stage_1_relu = nn.ReLU()
        self.stage_1_linear = nn.Linear(4 * 20 * 15, 2)
        self.stage_1_sigmoid = nn.Sigmoid()

        self.stage_2_conv = nn.Conv2d(3, 4, kernel_size=8, stride=8)
        self.stage_2_relu = nn.ReLU()
        self.stage_2_linear = nn.Linear(4 * 16 * 16, 5)
        self.stage_2_tanh = nn.Tanh()

    @torch.no_grad()
    def forward(self, image: torch.Tensor) -> torch.Tensor:
        # Cast and normalize image (not GPU compatible, but that is fine)
        image = image.float() / 255

        ##### Predict crop center (stage 1) #####
        x = self.stage_1_conv(image[:, :, ::4, ::4])
        x = self.stage_1_relu(x.flatten(start_dim=1))
        x = self.stage_1_linear(x)
        rel_crop_center = self.stage_1_sigmoid(x)

        ##### Extract crop #####
        image_size = torch.tensor((640, 480), dtype=torch.int32)
        half_crop_size = torch.tensor((64, 64), dtype=torch.int32)
        crop_center = (rel_crop_center * image_size).to(torch.int32)

        # Crop has to be within frame
        crop_center = torch.clamp(crop_center, half_crop_size, image_size - half_crop_size)
        center_y, center_x = crop_center[0]

        crop = get_crop(image, center_y, center_x, 64, 64)

        x = self.stage_2_conv(crop)
        x = self.stage_2_relu(x.flatten(start_dim=1))
        out = self.stage_2_tanh(self.stage_2_linear(x))

        return out

One drawback of this approach, apart from being suboptimal, is that tf.roll is not included in TFLITE_BUILTINS. You'd need to download/compile TFLite binaries with extended operator set. Also, don't forget to enable Flex ops in TFLiteConverter:
converter = tf.lite.TFLiteConverter.from_keras_model(keras_model)
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS]
tflite_model = converter.convert()

Tested on an Android device with OpenCL and XNNPACK delegates, so it might work for you, too.

I see, thank you very muck for such a good answer, I will do some tests to see if this is a valid approach for us!

@johan-sightic Hey, I just found a much better solution:
@nobuco.traceable
def get_crop(x, center_y, center_x, h, w):
    return x[:, :, center_y - h:center_y + h, center_x - w:center_x + w]

@nobuco.converter(get_crop, channel_ordering_strategy=nobuco.ChannelOrderingStrategy.FORCE_TENSORFLOW_ORDER)
def converter_get_crop(x, center_y, center_x, h, w):
    def func(x, center_y, center_x, h, w):
        return tf.image.crop_to_bounding_box(x, center_y - h, center_x - w, h*2, w*2)
    return func

It translates to normal tf.slice, so no custom binaries required.

@johan-sightic Hey, I just found a much better solution:
@nobuco.traceable
def get_crop(x, center_y, center_x, h, w):
    return x[:, :, center_y - h:center_y + h, center_x - w:center_x + w]

@nobuco.converter(get_crop, channel_ordering_strategy=nobuco.ChannelOrderingStrategy.FORCE_TENSORFLOW_ORDER)
def converter_get_crop(x, center_y, center_x, h, w):
    def func(x, center_y, center_x, h, w):
        return tf.image.crop_to_bounding_box(x, center_y - h, center_x - w, h*2, w*2)
    return func

It translates to normal tf.slice, so no custom binaries required.

Thank you very much! This seems to work!
I'm not sure how much is actually run on cpu vs gpu because I don't quite understand the log output but at least it runs about 75% faster now :smile:

@tonystratum I'm closing the issue for now due to inactivity. Please reopen if it's still relevant.


Port for mdns service is hardcoded, but possible to change service port via environment variable.

See #1595.
Which actually leads me into another issue, that the mdns broadcast is hardcoded.
https://github.com/MarshallOfSound/Google-Play-Music-Desktop-Player-UNOFFICIAL-/blob/master/src/main/features/core/websocketAPI.js#L115

And then because I love increasing the scope of problems.
The port is hardcoded in the remote app.
https://github.com/GPMDP/google-play-music-desktop-remote/blob/master/src/stores/WebSocketStore.js#L6
I would like to point out though that this environment variable was only really ever meant for testing purposes 😆


Add rule to check that a single top level class name matches the file name

This rule adheres to both Kotlin and Android Kotlin style guides:
https://kotlinlang.org/docs/reference/coding-conventions.html#source-file-names
https://android.github.io/kotlin-guides/style.html#naming
In order to make this change possible I had to plumb through the fileName into each Rule. This will also allow us to add additional style checks in the future (i.e. no underscores in file names, directory path matches package path, etc.)
Regarding autoFormat, I don't think there's a good solution here. If a violation occurs, then it's impossible for the tool to automatically determine whether to use the file name or the class name. The only alternative I can think of to this problem is to have Ktlint prompt the user for which name to use every time a violation occurs, which could be built after #79 is complete.

Hi Jacob (and thank you for the PR).
Everything inside ktlint-core/ is considered to be part of the public API and as such all changes must be backward compatible.
Starting from<EMAIL_ADDRESS>file path is now accessible via
val path: String? = node.psi.containingFile.getUserData(KtLint.FILE_PATH_USER_DATA_KEY).
Could you please update the PR to use that instead?
Also,

ktlint-core/src/main/kotlin/com/github/shyiko/ktlint/core/ASTNodeExtensions.kt needs to be removed (core should stay minimal).
(ClassNameMatchesFileNameRule) emit(0, ...) instead of emit(topLevelClassNames.first().offset, ...) (this should allow to simplify rule to a few lines of code)
**/*.kts should not be checked.

Thanks for the feedback! (and adding the file path to userData :)) Updated the PR.

Perfect. Thank you so much, Jacob!


Update README.md

Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Perfect. Thanks @ejferr !!


[0.8.10] Scene image dithering is often incorrect

I haven't experimented enough with this to know when it doesn't correctly update the scene and apply image properties but I did note today that the image display changed significantly when I changed the DPI from 500 to 1000 and didn't change much when I changed it back.
I suspect that the image DPI is not applied when the the file (previously saved by MK with DPI set) is loaded.

That may have been the same root cause as in issue https://github.com/meerk40t/meerk40t/issues/2086. The same comment applies, not yet in place in the latest bugfix release but fixed in the codebase.


Only some specific characters are thicker than others in the same sentence.

http://www.eaccess.net/technique_info/pdf/mobile.pdf
Open the page 6.
In pdf.js only some specific characters are thicker than others in the same sentence. Or most of characters are not Gothic while they should be.

Closing since this is fixed in the latest version.


Authored component .dll -- processor architecture mismatch

We need to make sure a component can author winui types and have those be consumed by a winui app.
For "winui" we are focusing on the ProjectReunion version of WinUI. Currently, I have a component producing a winmd, and a c++ app consuming it and trying to use one of the types via ProjectReference
You can change the build configuration to build for x86 instead of x64, and you'll get past this error. But we should support x64 out of the box.
Repro: Build the solution here for x64
https://github.com/j0shuams/ReunionAuthoringSample
Describe the bug
The bug is a build failure:
"c:\gh\Repros\ReunionAuthoringSample\ReunionAuthoringSample.sln" (default target) (1) ->
"c:\gh\Repros\ReunionAuthoringSample\CppApp\CppApp\CppApp.vcxproj" (default target) (4:6) ->
  C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targe
ts(2204,5):

error MSB3271: There was a mismatch between the processor architecture of the project being built "AMD64" a
nd the processor architecture, "x86", of the implementation file "c:\gh\Repros\ReunionAuthoringSample\MyLibrary\bin\Deb
ug\net5.0-windows10.0.19041.0\MyLibrary.dll" for "c:\gh\Repros\ReunionAuthoringSample\MyLibrary\bin\Debug\net5.0-window
s10.0.19041.0\MyLibrary.winmd".

This mismatch may cause runtime failures. Please consider changing the targeted process
or architecture of your project through the Configuration Manager so as to align the processor architectures between yo
ur project and implementation file, or choose a winmd file with an implementation file that has a processor architectur
e which matches the targeted processor architecture of your project. [c:\gh\Repros\ReunionAuthoringSample\CppApp\CppApp
\CppApp.vcxproj]

Version Info
CsWinRT : 1.2.3
ProjectReunion (& co.) : 0.5.5
It would be good to try this same scenario but using a PackageReference.

Is this because the path of the winmd is wrong?  I wasn't able to repro after setting all my configs to x64, and removing/readding the winmd reference.
I think the path should be c:\gh\Repros\ReunionAuthoringSample\MyLibrary\bin\x64\Debug\net5.0-windows\10.0.19041.0\MyLibrary.winmd
I did however run into #812 but not this issue.

I saw this error with a command line build using msbuild, but not a VS build. This was using this repro (packaging scenario): https://github.com/angelazhangmsft/AuthoringPackagingScenarios

Did not repro on reunion 0.8.1


Unittest

I think your unittest didn't pass
did you check and traced what's wrong with it ??

We made changes, thank you.


Prep for next release

Don't merge this until we can pin the setup.py to a new version rather than a commit.

Jk we're already pinned to a commit


Unable to resolve path to module

I've followed the instructions in the docs, but eslint keeps telling me this error:
Unable to resolve path to module 'react-native-linear-gradient'
And when I apply a dark blue ->  slightly lighter blue, I get a white background and it adds roughly a
paddingRight: 30 ?

link error


Identified - We are currently investigating reports of missing build logs. The issue has been identified and a resolution is in progress. We will provide a further update when available. Mar 21, 09:02 UTC

Identified - We are currently investigating reports of missing build logs. The issue has been identified and a resolution is in progress. We will provide a further update when available.
Mar 21, 09:02 UTC

Duplicate of #11768 Identified - We are currently investigating reports of missing build logs. The issue has been identified and a resolution is in progress. We will provide a further update when available.
Mar 21, 09:02 UTC


⚠ Bofrost.at has degraded performance

In 87c0111, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 5509 ms

Resolved: Bofrost.at performance has improved in d91b750 after 45 minutes.


Updating the mapping in elasticsearch

Official documentation says, we can have a mapping for string type for each index, named after the MongoDB database and collection names as animals.kitten.
I tried to create mapping as:
$ curl -XPUT 'http://localhost:9200/animals.kitten/_mapping' -d '
{
"animals.kitten" : {
"properties" : {
"name" : {"type" : "string", "store" : true }
}
}
}'
But it throws the error as:
{
"error": "ActionRequestValidationException[Validation Failed: 1: mapping type is missing;]",
"status": 500
}
I dont know where I am wrong. I tried different alternatives which did not worked as well. Even though elasticsearch generate dynamic mapping for each field in index animals.kitten (i.e. GET localhost:9200/animals.kitten/_mapping is working fine), it would be great if I could manually insert the mapping or update it later.

@BlackRabbitt
I think you may be missing the mapping type from the URI to which you're sending the PUT request. The following works for me:
curl -XPUT http://localhost:9200/animals.kitten/_mapping/string -d'{
    "string": {
      "properties": {
        "name": {
          "type": "string",
          "stored": true
        }
      }
    }
}'

Then we can add documents to the kitten collection in the animals database in MongoDB, and this mapping will apply to those documents sent to Elasticsearch from Mongo Connector. See the PUT Mapping API for Elasticsearch.

@lovett89
As you suggest, I updated the mapping as below:
PUT http://localhost:9200/animals.kitten/_mapping/string
{
"string": {
"properties": {
"name": {
"type": "string",
"stored": true
}
}
}
}
And I tried to insert name in mongodb as follow:

use animals
db.kitten.insert({name:2})
(insert successfull)

Then, I searched for mapping as: GET animals.kitten/_mapping
The mapping was not what I was expecting. The response is:
{
"animals.kitten": {
"mappings": {
"string": {
"properties": {
"name": {
"type": "double"
}
}
}
}
}
}
Am I wrong anywhere? I think mongo-connector should think twice on indexing data to elasticsearch.

The "type" of "name" changed because a document with {"name": 2.0} was inserted, and the "string" type, as currently set up, has a dynamic mapping. This is just a property of dynamic mappings in Elasticsearch. Besides, Mongo Connector doesn't use the Elasticsearch mappings to enforce a schema on MongoDB documents (see https://github.com/10gen-labs/mongo-connector/issues/140). For that, check out the --fields option.
I'd suggest reading Elasticsearch's documentation on mappings to see what advantages mappings provide and how to set them up. Mongo Connector doesn't attempt to set up mappings for you, it merely moves documents from MongoDB to Elasticsearch, and Elasticsearch either creates a dynamic mapping automatically, or uses the one already set up at /<database.collection>/string.

Thank you. Now my question has been answered. And sure, I will follow your suggestion. Thanks again.


Cannot be used with virtual DOMs

It seems as Hammer can’t leave elements in peace. For touch events, this inline style is set on elements:
touch-action: pan-y; -webkit-user-select: none; -webkit-user-drag: none; -webkit-tap-highlight-color: rgba(0, 0, 0, 0);

This completely eliminates any hope for using this library with virtual DOMs. Sadly! :(

Use a better virtual Dom :p
But seriously, most VD implementations handle this just fine.  Also, the styles you are complaining about are necessary, hammer is functioning as a touch-action polyfill, something that will continue to improve overtime.

Any suggestions on which vdom to use then?

Any implementation that doesn't wipe out your element references without warning, or (alternatively) which doesn't diff your actual DOM.
Which implementation are you using that's giving you trouble?

@runspired I’m using Matt Esch’s virtual-dom.

Given that lib is so heavily react-inspired, there's probably a version of this which you need: https://facebook.github.io/react/docs/more-about-refs.html
The implementation you are using seems to have very sparse documentation, but I'm guessing you need to be attaching to rootNode: https://github.com/Matt-Esch/virtual-dom/blob/master/vdom/README.md#example
It looks like you then would need to copy rootNode.style to leftNode.style after initializing Hammer.

(Also, sucks that the patch is applied directly to rootNode, there should be another layer there managed for you for staging, surprised there isn't).

@runspired Am I wrong in my assumption that a new instance on Hammer is needed for each element we want to apply listeners to?

I'm not sure what you mean to ask.
A new instance of Hammer.Manager would be needed for each DOM tree / DOM region you want a unique manager for.

Maybe it’s be that doesn’t understand the docs well enough, but why would I need a separate manager for a tree/region? Why not just one manager for the whole DOM?

You could use one manager for the whole DOM, but then anywhere you wanted to use native scroll either horizontal or vertical would likely be prevented.

@runspired Also for, say a div with scroll in the DOM tree?

Correct, touch-action is a CSS property that controls whether the developer or the device is going to get to control the outcome of a touch.  https://developer.mozilla.org/en-US/docs/Web/CSS/touch-action

Okay, so I’m screwed because these styles are set imperatively by Hammer. Thank you for you explanation, @runspired .

You can probably get the particular virtual-dom implementation you are using to work by copying what Hammer does to your virtual-implementation so that it won't be overwritten on the next patch.

 I've been using virtual-dom with touch events also, but thinking that some kind of widget could wrap the Hammer events. This looks like a reasonable starting point. I'm looking into it now...

@danielnaab I managed to make a hammer.js driver for Cycle.js that uses virtual-dom.

Thanks @Frikki. I'm not using Cycle.js, but more of a Mercury-style approach. This isn't tested, but I put a widget together yesterday based on react-hammerjs: https://github.com/danielnaab/virtual-dom-hammerjs

@Frikki @danielnaab if in the process you identify something that is absolutely not able to be done in your current approaches or just annoyingly complicated to do, please report back.  I think we'd be willing to add something to 2.1 or 2.2 to make this easier if needed as long as it were a minimal change.

@runspired totally agree as long as its not a huge change all for making this work

@runspired Thanks. The Cycle.js Hammer driver is in early stages and need community feedback. If we find issues that cannot be solved by ourselves, I’ll definitely report back. Cycle.js embraces functional programming and isolation of side effects through the use of drivers. Hammer.js, on the other hand, seem quite "class"-oriented in its style.

@runspired virtual-dom-hammerjs is working for me as of now, although it's probably doing something wrong. I'm learning how to use Hammer as I go, which is maybe not the best approach :)
One nice thing that might help with object reuse is if it were possible to create a Hammer object that isn't parented to a node, and then add/remove the attached node at a later point. In practice it's not really needed, though.

@danielnaab an element-less manager instance and the ability to swap elements, and possibly even a dispatch method for sending an event to a manager might be things I'd consider.  I've bumped into situations where reusing the manager with a new element was ideal before.  @arschmitz thoughts?

These are things im open to in general but that would be 3.0


Feature request: add sortDirection to customSort function

Is your feature request related to a problem? Please describe.
Request for a sortDirection param in the customSort function.
I need it to achieve an outcome where some designated records are always set to be on the top.
Describe the solution you'd like
customSort(a, b, ..., sortDirection)
Describe alternatives you've considered
The recommendation provided in https://github.com/mbrn/material-table/issues/3054 by @tian-lan-landing .
  customSort?: (
    data1: RowData,
    data2: RowData,
-   type: "row" | "group"
+   type: "row" | "group",
+   sortDirection?: "desc",
  ) => number;

It would be the best if the sortDirection is "asc" | "desc".
And he's created an example. here.
Or Perhaps, anyone knows alternative ways to make certain records always on the top when the columns is clicked to sort.

Looks like an easy fix. Will do.

I added the commit, it will be in the next release.


Understanding the handling of diacretics

This is not an issue. But a question on tex.sx made me curious.
When I run biber --tool bib.bib on this bib-file
@misc{exampleone,
author = {Jos{\'{e}} and  H{\'{\i}}drica},
title = {Title}
}

then the accented chars in the authors are encoded like this:
  AUTHOR = {Jos U+00E9 and H U+0131 U+0301drica},

I understand why the {'{\i}} is converted to U+0131 U+0301, but why is the é  U+00E9 and not U+0065 U+0301? I have been now looking for an hour at recode.pm and recode_data.xml and can't see the reason. (Just to avoid a misunderstanding: it is good that é is U+00E9).

It's just down to the complexities of dealing with these things. There are special Unicode code points for these things but without lots of hard-coded special cases, it's quite tricky. Dotless i and j are a pain to deal with.


Execute special query caching hooks in the BaseModel

Part of slaxweb/framework#150
Execute the special hooks when selecting and updating as described in slaxweb/cache#2
Execute the select method hooks only if user has explicitly set the cache property to true. The property must be set to false afterwards.

Canceled.


add-Gigi-info

LGTM! 🦖

Codecov Report

Merging #2 (8f7fb9c) into master (da39c6a) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master       #2   +/-   ##
=========================================
  Coverage     72.15%   72.15%
  Complexity      399      399
=========================================
  Files            70       70
  Lines          1232     1232
  Branches        125      125
=========================================
  Hits            889      889
  Misses          311      311
  Partials         32       32

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update da39c6a...8f7fb9c. Read the comment docs.


How do I get access to the Draggable data once it is over a Droppable?

Copying the example from the documentation quick start:
import React from 'react';
import {useDroppable} from '@dnd-kit/core';

function Droppable(props) {
  const {isOver, setNodeRef} = useDroppable({
    id: 'droppable',
  });
  const style = {
    color: isOver ? 'green' : undefined,
  };

  return (
    <div ref={setNodeRef} style={style}>
      {props.children}
    </div>
  );
}

How do I conditionally style the droppable depending on the draggable's data?
function Droppable(props) {
  // ...
  const supportedTypes = ["foo", "bar"];
  const style = {
    color: isOver ? 'green' : undefined,
  };

  const draggableCurrentlyOver = ???;

  if (supportedTypes.includes(draggableCurrentlyOver.data.type)) {
    style.color = "red"
  }

  // ...
}

Solved: Not found in the docs but active is also exposed by the useDroppable hook.

Thank you for updating your issue with a solution. Saved me a headache. This should definitely be documented!


Properly resolve relative paths in CSS imports

Currently, WebCord cannot resolve relative imports because it simply constructs a new URL() with the relative URL which will fail.
This change passes context about the current file to parseImports and fetchOrRead to prepend the previous URL's directory name if creating a URL fails.

You don't have to force-push PR on changes to have a specific history, in fact it might be better to commit on top of the another. As long as things are around single thing to be implemented, I can always squash your commits later, and having the history of changes done to this PR is for sure useful if we consider to go back to  how things were before (isn't that main purpose of git?)


web: Add missing CSS.escape calls

When referring to <div id="my-id" class="my-class">:

my-id is an id
#my-id is a selector
my-class is a class name
.my-class is a selector

When an id or a class name is interpolated into a selector, it needs to be escaped with CSS.escape.
Previously:

#17200

Most of these selectors are cardcoded, but definitely worth doing so we don't have to think about that sort of detail. @evykassirer FYI since about half of these were in code you wrote a few months ago.

Thanks Tim! yeah I only realized we needed to do this when working with Anders earlier this week - thanks for bringing it up!


filter nested keys in without()

Hopefully I didn't break any use cases, but all tests pass.  I was unable to filter nested keys at all before these changes.

Coverage remained the same at 95.974% when pulling 48175d7dc62337e0cfcf3ffae3adf2df8733ab95 on dwburke:without-nested into b37c55abedc3310a135f8766b8c5a95762787a68 on njlg:master.

Thanks for the update.


chore: Code cleanup in DatabaseModal

SUMMARY
Some of my cleanup changes were overwritten during a recent rebase, this reintroduces those changes.
ADDITIONAL INFORMATION

[ ] Has associated issue: Clubhouse Ticket
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

Why are you changing package-lock.json? Was that intentional?

It changed when I ran npm ci, so I added it. In the past I've encountered merge conflicts with the package-lock if this happens and I don't add the change.

Why are you changing package-lock.json? Was that intentional?

It changed when I ran npm ci, so I added it. In the past I've encountered merge conflicts with the package-lock if this happens and I don't add the change.

You should be able to remove it. You shouldn't have to update package-lock unless you make changes to package.json.

This looks great!

🏷 ready-to-merge


Risk encounter disappeared after 4 days

Avoid duplicates

[x] Bug is not mentioned in the FAQ
[x] Bug is not already reported in another issue

Describe the bug
A risk encounter shown up on 11.08. did not show up after 4 days. Looking at the logs, I see that the risk match of the same exposure check changed to 0 on 15.08.
Expected behaviour
A risk encounter should disappear after 14 days, not after 4 days. If there might be circumstances where the MatchCount of an exposure check can change, this should be mentioned in the FAQ.
Steps to reproduce the issue
Looking at the exposure export, I see the same exposure check with a MatchCount of 1 from 11.08. to 14.08. From 15.08. until today (17.08.) the MatchCount is 0
Exposure Checks from 11.08.-15.08. (hash changed to "xyz")
Exposure Check on 11.08.:
      {
        "Hash" : "xyz",
        "RandomIDCount" : 2380,
        "MatchCount" : 1,
        "DataSource" : "de.rki.coronawarnapp",
        "Timestamp" : "2020-08-11 09:08:38 +0200"
      },

Exposure Check on 12.08.:
      {
        "Hash" : "xyz",
        "RandomIDCount" : 2380,
        "MatchCount" : 1,
        "DataSource" : "de.rki.coronawarnapp",
        "Timestamp" : "2020-08-12 09:17:17 +0200"
      },

Exposure Check on 13.08.:
      {
        "Hash" : "xyz",
        "RandomIDCount" : 2380,
        "MatchCount" : 1,
        "DataSource" : "de.rki.coronawarnapp",
        "Timestamp" : "2020-08-13 11:32:54 +0200"
      },

Exposure Check on 14.08.:
      {
        "Hash" : "xyz",
        "RandomIDCount" : 2380,
        "MatchCount" : 1,
        "DataSource" : "de.rki.coronawarnapp",
        "Timestamp" : "2020-08-14 12:17:56 +0200"
      },

Exposure Check on 15.08.:
      {
        "Hash" : "xyz",
        "RandomIDCount" : 2380,
        "MatchCount" : 0,
        "DataSource" : "de.rki.coronawarnapp",
        "Timestamp" : "2020-08-15 14:12:48 +0200"
      }

Technical details

iOS Version: 13.6.1 (13.6 until 16.8.2020)
Device: iPhone 7

Additional context
The risk encounter mentioned above was a 'low risk' encounter

The matchcount disappeared because after 14 days your collected random ids will be deleted by iOS. This mean the rpi was 11 days old this the reason for a low risk and after 14 days it was deleted this mean 0 risk exposure

This is expected behavior:
each package contains all the keys of all the ppl who uploaded their keys that day. The package which contained the key you matched with had 2380 keys and is therefore from the 10.08 (cf. here). It therefore contains keys for up to 14 days in the past from that date (i.e. 27.07). That your key disappeared on the 15.08 means, that you your encounter with the person who later tested positive must have been 14 days before that i.e. on the 01.08.
Keys older than 14 days are usually not relevant for determining infection risk and are therefore discarded as @thomasaugsten stated, that's the reason why your risk encounter disappeared.

Thank you for your quick responses and the additional information. That explains the behavior. Unfortunately, there was no information in the App about the age of the risk encounter. But it looks like this is already tracked in another issue

Unfortunately, there was no information in the App about the age of the risk encounter. But it looks like this is already tracked in another issue

Yes, this is tracked here: https://github.com/corona-warn-app/cwa-backlog/issues/23 🙂


[BUG] Custom VPC-DNS not working at VM (kubevirt)

Kube-OVN Version
v1.12.12
Kubernetes Version
Client Version: v1.30.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0
Operation-system/Kernel Version
"Ubuntu 22.04.4 LTS"
5.15.0-113-generic
Description
Right now, I can create isolated workload in Pod by using Subnet, VPC and Nat Gateway.
And thru VPC-DNS, my pod can reach domain name.
But somehow in KubeVirt VM, there are two problems

Route: somehow route cannot go to <IP_ADDRESS> automatically, i have to add somekind of route inside the vm thru cloud-init (solveable)
DNS : somehow in the vm cannot reach vpc-dns. In my case cannot reach dns pod

Steps To Reproduce

Create Kubevirt VM using script in Config VPC
Cannot check for DNS

Current Behavior
Default DNS not working inside VM.
ubuntu@devspace-vm:~/dekagpu-installation/$ k get svc -o wide -A|grep dns
kube-system   kube-dns                      ClusterIP   <IP_ADDRESS>       <none>        53/UDP,53/TCP,9153/TCP   43h   k8s-app=kube-dns
kube-system   slr-vpc-dns-dns-net1-ns1      ClusterIP   None             <none>        53/UDP,53/TCP,9153/TCP   42h   k8s-app=vpc-dns-dns-net1-ns1

When I try to solve from existing VM
ubuntu@ubuntu:~$ resolvectl status
Global
       Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (enp1s0)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: <IP_ADDRESS>
       DNS Servers: <IP_ADDRESS>
        DNS Domain: cluster.local devspace.svc.cluster.local
                    ns1.svc.cluster.local svc.cluster.local

DNS already correct but somehow cannot reach DNS server
ubuntu@ubuntu:~$ ping <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>) 56(84) bytes of data.
^C
--- <IP_ADDRESS> ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1018ms

ubuntu@ubuntu:~$ telnet <IP_ADDRESS> 53
Trying <IP_ADDRESS>...
telnet: Unable to connect to remote host: No route to host
ubuntu@ubuntu:~$

Expected Behavior
DNS should be working, maybe I missed something

There are two mistakes:

<IP_ADDRESS> is a service ip, you should never ping a service ip to check if the service is reachable;
You should use the vpc-dns vip defined in configmap vpc-dns-config, not <IP_ADDRESS>.

You should use another ip address, e.g. <IP_ADDRESS>.

my DNS at <IP_ADDRESS> ...my current ServiceIP for CoreDNS
ubuntu@ubuntu:~$ resolvectl status
Global
       Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (enp1s0)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: <IP_ADDRESS>
       DNS Servers: <IP_ADDRESS>
        DNS Domain: cluster.local devspace.svc.cluster.local
                    ns1.svc.cluster.local svc.cluster.local

If i change into <IP_ADDRESS>,
##
# coredns-vip is the IP address of the CoreDNS service.
# IP can be changes
##

apiVersion: v1
kind: ConfigMap
metadata:
  name: vpc-dns-config
  namespace: kube-system
data:
  coredns-vip: <IP_ADDRESS>
  enable-vpc-dns: "true"
  nad-name: ovn-nad
  nad-provider: ovn-nad.default.ovn

No response...
ubuntu@devspace-vm:~/dekagpu-installation/$ k exec -it pod/vpc1-pod -n ns1 -- ping www.google.com
ping: bad address 'www.google.com'
command terminated with exit code 1

how about nc -u -zv <IP_ADDRESS> 53 ？

This is my result
ubuntu@devspace-vm:~/dekagpu-installation/$ k exec -it pod/vpc1-pod -n ns1 -- nc -u -zv <IP_ADDRESS> 53
<IP_ADDRESS> (<IP_ADDRESS>:53) open
ubuntu@devspace-vm:~/dekagpu-installation/$ k exec -it pod/vpc1-pod -n ns1 -- ping www.google.com
ping: bad address 'www.google.com'
command terminated with exit code 1

and my config
##
# coredns-vip is the IP address of the CoreDNS service.
# IP can be changes
##

apiVersion: v1
kind: ConfigMap
metadata:
  name: vpc-dns-config
  namespace: kube-system
data:
  coredns-vip: <IP_ADDRESS>
  enable-vpc-dns: "true"
  nad-name: ovn-nad
  nad-provider: ovn-nad.default.ovn

Please check whether the vpc dns works:
k exec -it pod/vpc1-pod -n ns1 -- nslookup kubernetes.default.svc.cluster.local. <IP_ADDRESS>

If it works for vpc pods but does not work for the vpc vm, the problem may be related to kubevirt. Check your route in the vm:
ip route get <IP_ADDRESS>
nc -v -z -w1 <IP_ADDRESS> 54

Hi @zhangzujian
This is my config
apiVersion: v1
kind: ConfigMap
metadata:
  name: vpc-dns-config
  namespace: kube-system
data:
  coredns-vip: <IP_ADDRESS>
  enable-vpc-dns: "true"
  nad-name: ovn-nad
  nad-provider: ovn-nad.default.ovn

Results
ubuntu@ubuntu:~$ ubuntu@devspace-vm:~/dekagpu-installation/$ k exec -it pod/vpc1-pod -n ns1 -- nslookup kubernetes.default.svc.cluster.local. <IP_ADDRESS>
Server:         <IP_ADDRESS>
Address:        <IP_ADDRESS>:53

** server can't find kubernetes.default.svc.cluster.local.: REFUSED

** server can't find kubernetes.default.svc.cluster.local.: REFUSED

command terminated with exit code 1

in VM
ubuntu@ubuntu:~$ ip route get <IP_ADDRESS>
RTNETLINK answers: Network is unreachable

ubuntu@ubuntu:~$ ip route get <IP_ADDRESS>
<IP_ADDRESS> dev enp1s0 src <IP_ADDRESS> uid 1000
    cache

and
ubuntu@ubuntu:~$ nc -v -z -w1 <IP_ADDRESS> 54
nc: connect to <IP_ADDRESS> port 54 (tcp) timed out: Operation now in progress

ubuntu@ubuntu:~$ nc -v -z -w1 <IP_ADDRESS> 54
nc: connect to <IP_ADDRESS> port 54 (tcp) failed: Network is unreachable

ubuntu@ubuntu:~$ ip route get <IP_ADDRESS>
RTNETLINK answers: Network is unreachable

Check routes in the VM by:
ip addr show
ip route show

nc -v -z -w1 <IP_ADDRESS> 54

The port should be 53.
Did you change the dns vip in the vpc-dns-config? If so, please try to turn off the vpc dns and enable it again.

Run the following command to check ovn status:
kubectl ko nbctl lr-route-list vpc1
kubectl ko nbctl ls-lb-list <vpc-subnet-name>
kubectl ko trace ns1/vpc1-pod <IP_ADDRESS> udp 53

Hi @zhangzujian
after i figure out, is that

Kubevirt VM is not getting default route that has been change into <IP_ADDRESS>

After I change/add default gateway
ubuntu@ubuntu:~$ ip route show
default via <IP_ADDRESS> dev enp1s0
<IP_ADDRESS>/24 dev enp1s0 proto kernel scope link src <IP_ADDRESS> metric 100
<IP_ADDRESS> dev enp1s0 proto dhcp scope link src <IP_ADDRESS> metric 100

Default of dns still goes to coredns pod, instead of vpc-dns

Default from dhcp4
ubuntu@ubuntu:~$ resolvectl status
Global
       Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (enp1s0)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: <IP_ADDRESS>
       DNS Servers: <IP_ADDRESS>
        DNS Domain: cluster.local devspace.svc.cluster.local
                    ns1.svc.cluster.local svc.cluster.local

After changed 2 of these, DNS in kubevirt working well.
But the next question, is it possible to make it as default value (VIP of VPC-DNS) for dhcp4 ?
Thanks

I try to create another namespace and recreate the same use case, but somehow, default dns resolve for pod still the same coredns ip (<IP_ADDRESS>), instead of vpc-dns ip (<IP_ADDRESS>)
Here's the script
kind: Vpc
apiVersion: kubeovn.io/v1
metadata:
  name: vpc3
spec:
  namespaces:
  - ns3
  staticRoutes:
  - cidr: <IP_ADDRESS>/0
    nextHopIP: <IP_ADDRESS>
    policy: policyDst
  - cidr: <IP_ADDRESS>/0
    nextHopIP: <IP_ADDRESS>
    policy: policySrc
---
kind: Subnet
apiVersion: kubeovn.io/v1
metadata:
  name: subnet3
spec:
  vpc: vpc3
  cidrBlock: <IP_ADDRESS>/24
  protocol: IPv4
  namespaces:
    - ns3
  excludeIps:
  - <IP_ADDRESS>
  gateway: <IP_ADDRESS>
  gatewayType: distributed
  natOutgoing: false
  provider: ovn-nad.default.ovn
---
kind: VpcNatGateway
apiVersion: kubeovn.io/v1
metadata:
  name: gw3
spec:
  vpc: vpc3
  subnet: subnet3
  lanIp: <IP_ADDRESS>
  selector:
    - "kubernetes.io/hostname: kind-worker"
    - "kubernetes.io/os: linux"
  externalSubnets:
    - ovn-vpc-external-network
   ---
kind: IptablesEIP
apiVersion: kubeovn.io/v1
metadata:
  name: eip-ns3-1
spec:
  natGwDp: gw3
---

kind: IptablesSnatRule
apiVersion: kubeovn.io/v1
metadata:
  name: snat-ns3-01
spec:
  eip: eip-ns3-1
  internalCIDR: <IP_ADDRESS>/24
---
kind: VpcDns
apiVersion: kubeovn.io/v1
metadata:
  name: dns-net3-ns3
spec:
  vpc: vpc3
  subnet: subnet3

Everything already ok, but somehow ping to domain still failed
ubuntu@devspace-vm:~/dekagpu-installation/samples/diff-gw-3$ k exec -it pod/vpc3-pod -n ns3 -- nslookup kubernetes.default.svc.cluster.local <IP_ADDRESS>
Server:         <IP_ADDRESS>
Address:        <IP_ADDRESS>:53

Name:   kubernetes.default.svc.cluster.local
Address: <IP_ADDRESS>

ubuntu@devspace-vm:~/dekagpu-installation/samples/diff-gw-3$ k exec -it pod/vpc3-pod -n ns3 -- cat /etc/resolv.conf
search ns3.svc.cluster.local svc.cluster.local cluster.local devspace.svc.cluster.local
nameserver <IP_ADDRESS>
options ndots:5

ubuntu@devspace-vm:~/dekagpu-installation/samples/diff-gw-3$ k exec -it pod/vpc3-pod -n ns3 -- nc -v -z -w1 <IP_ADDRESS> 53
<IP_ADDRESS> (<IP_ADDRESS>:53) open

i've check using nbctl, dhcp options already in into the system
ubuntu@devspace-vm:~/dekagpu-installation/$ kubectl ko nbctl list dhcp-option
_uuid               : 3e73a722-ef51-4c21-ab26-3a44a0227816
cidr                : "<IP_ADDRESS>/24"
external_ids        : {ls=subnet3, protocol=IPv4, vendor=kube-ovn}
options             : {dns_server="<IP_ADDRESS>", lease_time="60", router="<IP_ADDRESS>", server_id="$random_mac", server_mac="$random_mac"}

DNS config in the subnet's dhcp config does NOT work for pods (including VMs), because pod's /etc/resolv.conf is generated by kubelet. You can try Pod's DNS Config to set the dns server.
If pod's DNS config does not for VMs, you need to find another way to set VM's DNS server. Maybe cloud-init is a solution.

I dont know why, since using VPC Gateway, my default DNS from coredns svc now unreachable.
Service for vpc-dns working well, but pod seems by default using <IP_ADDRESS> instead of assigned VIP vpc-dns <IP_ADDRESS>
ubuntu@devspace-vm:~/dekagpu-installation/samples/diff-gw-3$ k exec -it pod/vpc3-pod -n ns3 -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet <IP_ADDRESS>/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
150: eth0@if151: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1400 qdisc noqueue state UP
    link/ether 6a:85:46:0a:f8:16 brd ff:ff:ff:ff:ff:ff
    inet <IP_ADDRESS>/24 brd <IP_ADDRESS> scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::6885:46ff:fe0a:f816/64 scope link
       valid_lft forever preferred_lft forever

Network is reachable
ubuntu@devspace-vm:~/dekagpu-installation/samples/diff-gw-3$ k exec -it pod/vpc3-pod -n ns3 -- nc -u -zv <IP_ADDRESS> 53
<IP_ADDRESS> (<IP_ADDRESS>:53) open

ubuntu@devspace-vm:~/dekagpu-installation/samples/diff-gw-3$ k exec -it pod/vpc3-pod -n ns3 -- nc -u -zv <IP_ADDRESS> 53
<IP_ADDRESS> (<IP_ADDRESS>:53) open

But DNS unresolveable
ubuntu@devspace-vm:~/dekagpu-installation/samples/diff-gw-3$ k exec -it pod/vpc3-pod -n ns3 -- ping www.google.com
ping: bad address 'www.google.com'
command terminated with exit code 1

ubuntu@devspace-vm:~/dekagpu-installation/samples/diff-gw-3$ k exec -it pod/vpc3-pod -n ns3 -- cat /etc/resolv.conf
search ns3.svc.cluster.local svc.cluster.local cluster.local devspace.svc.cluster.local
nameserver <IP_ADDRESS>
options ndots:5

Don't use nc -u to check whether an UDP service is reachable. In stead, you should use nc -zv <IP_ADDRESS> 53 to check whether the TCP port is reachable.
<IP_ADDRESS> is the service ip of the default DNS service which is not accessable from other VPCs. That's why we introduced vpc dns.

Sorry for this. I've got the idea. And vpc-dns already works, if it set as default resolver.
Which currently is not. Pod DNS still refer to <IP_ADDRESS> right now.
That's why I asked regarding is it possible to change default dns. Because if not, vpc-dns is working, but it's usable, due to default DNS is not refer to vpc-dns VIP
As expected, <IP_ADDRESS> is not reachable
ubuntu@devspace-vm:~/dekagpu-installation/$ k exec -it pod/vpc3-pod -n ns3 -- nc -zv <IP_ADDRESS> 53
^Cpunt!
command terminated with exit code 130

ubuntu@devspace-vm:~/dekagpu-installation/$ k exec -it pod/vpc3-pod -n ns3 -- nc -zv <IP_ADDRESS> 53
<IP_ADDRESS> (<IP_ADDRESS>:53) open
ubuntu@devspace-vm:~/dekagpu-installation/$

pod's /etc/resolv.conf is generated by kubelet

Pod's DNS config (/etc/resolv.conf) is NOT controlled by CNI plugins.
We introduced vpc dns so that pods in custom vpcs can lookup domains via the vpc dns vip.
However, CNI plugins have NO ability to set pod's DNS server to the vip. You can do it by pod's dns config or by pod creation webhook.
If you are just testing vpc dns, maybe you can simply set the vpc dns vip to <IP_ADDRESS>.


backend/multi: add backends at end of list

This allows wlr_multi_for_each_backend to iterate on the backends in the
order where they have been added.

Thanks!


[Improvement] Ability to download mediation policies.

Description:
Currently, the api publishers could not check the content of the mediation policies at the time of applying them. This improvement suggests a solution for this by providing the ability to download the selected mediation policy.
If any modification should be done, the publisher can edit the policy and re-upload.
Suggested Labels:

Suggested Assignees:

Affected Product Version:
2.1.0
OS, DB, other environment details and versions:
Steps to reproduce:
Related Issues:

This issue is fixed by https://github.com/wso2/carbon-apimgt/pull/5114


TimeoutError: timed out during handshake

Description of the bug:
Problem:
The code works fine when running in Colab, but when I try to run it locally in my country, I keep encountering the TimeoutError: timed out during handshake issue. I've already tried using a VPN, but I still can't access WebSockets properly.
Does anyone have any good solutions to fix this?
Thank You!

TimeoutError                              Traceback (most recent call last)
Cell In[7], line 1
----> 1 await run(prompt='Hello?')
Cell In[6], line 66
63 if tools is None:
64   tools=[]
---> 66 async with (
67     connect(uri, additional_headers={"Content-Type": "application/json"}) as ws,
68 ):
69   await setup(ws, modality, tools)
70   await send(ws, prompt)
File c:\Users\maoxi\github\miniconda3\envs\finance_bot\Lib\site-packages\websockets\asyncio\client.py:485, in connect.aenter(self)
[484](file:///C:/Users/maoxi/github/miniconda3/envs/finance_bot/Lib/site-packages/websockets/asyncio/client.py:484) async def aenter(self) -> ClientConnection:
--> [485](file:///C:/Users/maoxi/github/miniconda3/envs/finance_bot/Lib/site-packages/websockets/asyncio/client.py:485)     return await self
File c:\Users\maoxi\github\miniconda3\envs\finance_bot\Lib\site-packages\websockets\asyncio\client.py:476, in connect.await_impl(self)
[472](file:///C:/Users/maoxi/github/miniconda3/envs/finance_bot/Lib/site-packages/websockets/asyncio/client.py:472)             raise SecurityError(f"more than {MAX_REDIRECTS} redirects")
[474](file:///C:/Users/maoxi/github/miniconda3/envs/finance_bot/Lib/site-packages/websockets/asyncio/client.py:474) except TimeoutError:
[475](file:///C:/Users/maoxi/github/miniconda3/envs/finance_bot/Lib/site-packages/websockets/asyncio/client.py:475)     # Re-raise exception with an informative error message.
--> [476](file:///C:/Users/maoxi/github/miniconda3/envs/finance_bot/Lib/site-packages/websockets/asyncio/client.py:476)     raise TimeoutError("timed out during handshake") from None
TimeoutError: timed out during handshake
Actual vs expected behavior:
No response
Any other information you'd like to share?
No response

met the same error here. waiting for the response

+1

+1

From which country are you trying to use Gemini?
Can you run !curl ipinfo.io?

China, but now I can get access somehow. I am trying to figure out why

China, but now I can get access somehow. I am trying to figure out why

How was it solved?

China, but now I can get access somehow. I am trying to figure out why

How was it solved?

I try to use websocket instead of websockets in my vitepress blog. This may be the reason?
// 运行对话
const run = async (prompt, modality = 'TEXT', tools = []) => {
  return new Promise((resolve, reject) => {
    const uri = `wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContent?key=${import.meta.env.VITE_GEMINI_API_KEY}`

    try {
      ws = new WebSocket(uri)
      ws.addEventListener('open', async () => {
        try {
          await setup(ws, modality, tools)
          await send(ws, prompt)
        } catch (error) {
          reject(error)
        }
      })

      ws.addEventListener('message', async (event) => {
        try {
          let response
          if (event.data instanceof Blob) {
            const text = await event.data.text()
            response = JSON.parse(text)
          } else {
            response = JSON.parse(event.data)
          }

          logger.value?.debug('收到消息:', response)

          const serverContent = response.serverContent
          if (serverContent) {
            const [turnComplete, audio] = handleServerContent(serverContent)
            if (turnComplete) {
              resolve()
              return
            }
          }

          const toolCall = response.toolCall
          if (toolCall) {
            await handleToolCall(ws, toolCall)
          }
        } catch (error) {
          logger.value?.error('消息处理错误:', error)
        }
      })

      ws.addEventListener('error', (error) => {
        logger.value?.error('WebSocket 错误:', error)
        reject(error)
      })

      ws.addEventListener('close', () => {
        logger.value?.debug('WebSocket 已关闭')
        resolve()
      })
    } catch (error) {
      reject(error)
    }
  })
}

Gemini is not available in China (cf. https://ai.google.dev/gemini-api/docs/available-regions#available_regions), so that's your issue.


loadProcess in outcomes.js does not handle integer values

TODO after #339 is merged.

Note that this also affects loadRaw

Also affects unload{Process,Raw}

Also affects add{Process,Raw}

Also affects various functions in codes.js

Also: update the variable names (see https://github.com/Open-Reaction-Database/ord-schema/pull/339#discussion_r485778799)

@connorcoley my solution in #350 is to look at the value in the text field and, if it can be parsed as an integer, use integer_value instead of float_value. This means that even numbers entered as e.g. 140.0 will be converted to integer_value in the Data message.
Is this what we want? Or do we want to expand the number radio button to float and integer?


Move pagination component from Altinn's design system

- [ ] Move component code and unit tests
- [ ] Refactor CSS classes (we are not using the BEM convention)
- [ ] Create Storybook MDX file

Pagination is created.


Add Corefx/master to ci4

cc: @mmitche @sunandabalu

@sunandabalu You used to need that.  I changed it recently so that a CI server has a 'default' utilities repo.  It can still be overridden.


Graphormer - Do Transformers Really Perform Bad for Graph Representation?

This blog post serves as a guide to understanding the Graphormer, which is a Transformer architecture which attains good performance in various Graph representation tasks. We also provide small code snippets for better understanding.

We have successfully received your blog post. Thank you for your submission!
If you are willing to review another blog post, you can follow the guidelines here.
We will compile all the reviews and provide feedback on your submission by July 15th. Thank you again!

Your blog post has been accepted, congratulations!
It will be featured on Gram-Blogpost on Monday, July 22nd. You will receive an email tonight with some feedback and instructions. You have until July 21st to make any improvements. If you are attending ICML in person, please inform us, as you can present your blog post as a poster at our workshop. Hopefully, see you next week in Vienna! :)

Hello,
We have gone through the review and made the requested changes. We have also added explanatory bpxes wherever requested.


[FEATURE REQUEST] Add License section in readme

This is usually done with a link to the license file to encourage users to read know where the license is as some newer users may not know

Pull request https://github.com/czM1K3/keychain-generator/pull/21 would resolve this issue


On nano 33 IOT, EmailSender Supports Only Encrypted Mail

EMailSender ignores the isSecure flag on this platform and probably others.

Hi,
isSecure is supported only by esp8266 for core < 2.4.2.
Now all platform automatically manage that flag.
Bye Renzo

The flag isn't managed in the sense that it doesn't work if I try to connect to port 25 which is not secure. If your statement is meant to say that it isn't an issue and that EMailServer is intended to support only encrypted mail, I can understand that. The documentation should say that in bold print.

Please post your code and explain your test.
Bye Renzo

My test code is copied from J-oseph with modifications to use my account on sendgrid.
This code fails.
/*
 * MADE BY J-oseph ON GITHUB
 * MOST OF THIS CODE IS NOT MY OWN
 * PLEASE CHECK OUT MY VIDEO https://youtu.be/GFDTMFBPCn4 TO SEE HOW TO SET EVERYTHING UP
 * AND TO FIND LINKS TO OTHER PEOPLE'S CODE
 * THIS CODE WORKS WITH ARDUINO NANO 33 IOT
 * USE WITH OTHER ARDUINOS AT YOUR OWN RISK ;)

 * Modified by Kenneth Lerman<EMAIL_ADDRESS>*/
#include <WiFiNINA.h>
#include <EMailSender.h>
#include "arduino_secrets.h"

char ssid[] = SECRET_SSID;
char pass[] = SECRET_PASS;
char eMailPass[] = SECRET_APIKEY;

char smtpServer[] = "smtp.sendgrid.net";
int smtpPort = 25;

char eMailUser[] = "apikey"; // account must have correct settings
char eMailRecipient[] =<EMAIL_ADDRESS>
const String TEXT = "T E X T"; // main body text of the email
const char SUBJ[] = "S U B J"; // subject of the email

WiFiClient client; // sets up the wifi client

void setup(){
  Serial.begin(9600);

  // all we do is connect to wifi and send and email
  setupWifi();

  sendEmail();
}

void loop() {}

void setupWifi(){
  // connects to the wifi
  Serial.print("Connecting to ");
  Serial.print(ssid);
  Serial.print("...");

  WiFi.begin(ssid, pass);
  while (WiFi.status() != WL_CONNECTED) {
    // will stay in a loop if it doesnt connect
    delay(2000);
    Serial.print(".");
  }
  Serial.println("Connected to WiFi");
  Serial.print("Local IP address: ");
  Serial.println(WiFi.localIP());
  Serial.println("");
  Serial.println("");
}

void sendEmail(){
    // sends an email
    // the name it will display who the email is from
    char senderName[] = "Sump Pump Controller";

    // the actual stuff that sends the email:
    EMailSender emailSend(eMailUser, eMailPass<EMAIL_ADDRESS>        senderName, smtpServer, smtpPort);

    EMailSender::EMailMessage msg;
    EMailSender::Response resp;
    Serial.println("Sending email...");
    msg.subject = SUBJ;
    msg.message = TEXT;

    resp = emailSend.send(eMailRecipient, msg);

    // all of this just prints the info
    Serial.println("Sending status: ");
    Serial.print(resp.status);
    Serial.println(resp.code);
    Serial.println(resp.desc);
    Serial.println("");
    Serial.print("FROM: ");
    Serial.println(eMailUser);
    Serial.print("TO: ");
    Serial.println(eMailRecipient);
    Serial.print("SUBJECT: ");
    Serial.println(SUBJ);
    Serial.print("DATA:");
    Serial.println(TEXT);
    Serial.println("");
}

It will work if I insert into EMailSender.h around line 217:
#define EMAIL_NETWORK_CLASS WiFiSSLClient
//!!!KL
#define KL
#ifndef KL
#define EMAIL_NETWORK_SSL_CLASS WiFiSSLClient
#define EMAIL_NETWORK_SERVER_CLASS WiFiServer
#else
#define EMAIL_NETWORK_SSL_CLASS WiFiClient
#define EMAIL_NETWORK_SERVER_CLASS WiFiServer
#endif

That is, if I use WiFiClient instead of WiFiSSLClient.
That's because WiFiSSLClient is always using SSL.
Thanks for looking at this.
I should comment that at this point, I'm up and running using the above change to the library, so there is no urgency for me.
Regards,
Ken

Hi,
I add a define that allows disabling SSL clients for the particular management of the SAMD board.
Give me a feedback about that.
Bye Renzo

That's just another patch that's probably better than mine.
If I were writing a requirements document for this program, one of the requirements would be:
At run time, the user shall be able to set whether SSL is used for each message sent.

I should be able to write and build a program where the user sets what server is used, what port is used, and whether it is encrypted or not. The default might be that some ports (such as port 25) are not encrypted and other ports are encrypted.


Add Unit Test on Pool Performance calculation

📄 Context
The Pool Performance should have the unit test to check whether the calculation is accurate. The unit test should also check for the handler behind it.
🎯 Definition of Done

[x] Unit test should cover the logic behind the calculation
[x] Test Coverage should be at least 60%

I linking that to the merged PR #6


Body invalid

After setting-up client getting Body invalid

If you got "Body invalid", that means that the client fail to collect stat data.
Which system is this client installed?

Ubuntu 14

Thank you for your so quickly reply. Ubuntu is supported.
It seems that the log is not help for find out the problem.
I will add more log the next version, would you try again after that?
You may try other client this time if you wish.

Sure
when i run ./client we are getting Body Invalid

Thank you!
Is this client 32 or 64 Bits?

64bit, i just tried in CentOS 64 but, same error.

Will you try this client?
wget https://github.com/likexian/stathub-go/releases/download/v0.7.2/client && chmod +x client
./client

-bash: ./client: cannot execute binary file
Version:- Ubuntu 12.04.5 x32

Body: -->  <--
Body: -->  <--
Body invalid
Version:- Ubuntu 14.04

The client collect empty stat, I will test it again on Ubuntu.
Thank you very much, @vijaygadde , I will release next version which you may try again.

I have find out why.
If the system have no SWAP, the client fail.
I will release the new version shortly, may you try again after that?

yes the server don't have any swap for now. however i will swap on and try,
yes please let me know when the new version released.

thank you very much, @vijaygadde .

after adding swap, it works.

yes, and the next version will work when swap is off.

Hello, the new version has released just now.
This version will work when swap is off, and add monthly bandwidth usage data.


Forms : Fix state references

Summary of changes

DialogRequester saves the references to the states that it is passed when a field requests a dialog. In case of a configuration change, new states are created but the DialogRequester is still referencing the old state.
This PR fixes this issue by making the DialogRequester accept a stateId instead of the state object. This stateId can then be used to fetch the actual state from the FormStateCollection.
The stateId mechanism is relying in the FormElement.hashCode() implementation which ensures that even after an orientation change the hashCode() will not change because the underlying core object is the same.

thanks @sorenoid


XeTeX compatibility

Hi, I'd like to request XeTeX compatibility be considered. Currently any attempt to compile a package results in the following "error":
Package microtype Error: Letterspacing currently doesn't work with xetex.

See the microtype package documentation for explanation.
Type  H <return>  for immediate help
error: halted on potentially-recoverable error as specified

The currently part makes me hope that it is possible to re-enable the compatibility.
Rationale
https://github.com/tectonic-typesetting/tectonic uses a customised fork of XeLaTeX. Aside from that it is an incredibly convenient typesetting system: it doesn't occupy tens of gigabytes, it fetches only the packages that are needed and frankly the choice to use a more modern version of LaTeX allows it to be built for other platforms.
Currently any document that requires microtype cannot be built with tectonic, necessitating a TeXLive installation.

First of all, you must be doing something to provoke this error message (either you're loading the package with the tracking option, or you're using the \textls command). Just loading microtype should not result in any errors, and it should run just fine with tectonic/xetex. So if you really get this error message for no apparent reason, something else would be wrong (for which I'd need a MWE).
As to whether it's possible to allow letterspacing with xetex: in principle yes, because xetex is able to add letterspacing to (OpenType) fonts via fontspec's LetterSpace option. I could integrate this into microtype's interface -- in fact this has long been on my todo list, and your report has now upped it... For the time being, you could use the interface provided by fontspec.

(either you're loading the package with the tracking option, or you're using the \textls command)

Exactly! I didn't notice it before making the MWE.

I'm also using tectonic and therefore a fork of xetex and would love to use the automatic font expansion option of microtype. Is it difficult to implement this for xetex?

I wonder if there can be a simple package that can convert the microtyping macros into XeTeX native instructions.

I'm also using tectonic and therefore a fork of xetex and would love to use the automatic font expansion option of microtype. Is it difficult to implement this for xetex?

Font expansion would have to be implemented in the engine itself, so there's nothing microtype can do. (Some time ago, there were some rumours that expansion would be made available in xetex, but nothing seems to have come of it. Since development is basically frozen, I wouldn't hold my breath.)
As for the original request (letterspacing): this is feasible, and I'll have a look.


Repo sync for protected CLA branch

The pull request is created from master637716448570593418 to master to fix git push error for protected CLA branch

@opbld15 : Thanks for your contribution! The author(s) have been notified to review your proposed change.


CPDLC message type appears to be missing

Hi,
First off, thanks for your work on this super handy package for Hoppie's ACARS!
While implementing this in a personal project, I kept wondering why I was unable to logon to a CPDLC station, e.g. KUSA.
I then found out that the request logon message needs to be sent as message_type: cpdlc, however it appears that this message type is missing from HoppieConnector.
I'm wondering if this is indeed correct, or whether I'm missing something here?
Thanks!

Hi there!
Glad to see my stuff is useful to others :)
The message type is indeed not implemented! My primary use case was as an ACARS ground station, and I hadn't looked into CPDLC yet at that point.
Are you parsing the CPDLC message contents in your application? A generic CPDLC message container type should cover that usecase, I believe.
Thanks for the feedback, I'll put this as a feature request!

You're more than welcome! :-)
Parsing and creating these CPDLC message payloads is the thing that I'm struggling with... I already tried creating a CPDLCMessage class based on how TelexMessage is implemented.
However, I then learned (the inner workings of CPDLC are still extremely new to me) that CPDLC messages must follow a specific format, which might look something like
/data2/6/19/N/WILCO

Unfortunately, I'm struggling to find out the meaning of those specifiers, all I know is that data2 refers to the payload format being used, and WILCO is the actual message being sent...
Do you by any chance happen to have an idea what the other stuff means?
Thanks!

OK... Was able to learn some more about the structure of these messages, but still not sure if my understanding is fully correct. Here's how it apparently works.
Given the following example message:
/data2/14//WU/CLD 1522 240826 EDDL PDC 005 OCN69FX CLRD TO @LICC@ OFF @23L@ VIA @DODEN9T@ CLIMB @500... [+59]

The / character is used as the delimiter between the various data fields.
data2 designates the data exchange format, this is always specified and always the same.
14 is the message ID. Can be randomly generated as far as I understand, doesn't have to be sequential or anything like that.
// If this were a response to a previously received message, the ID of the referenced message would go here. No value in this case as the PDC message is the start of a message dialog.
WU specifies the response choices given to the recipient, indicating how the message should be acknowledged. As far as I could figure out this can be one of:

NE: None Required

Y: Yes

N: No

WU: Wilco/Unable

AN: Affirm/Negative

R: Roger

The final part of the payload is the actual message content.

Consequently, the response message to the above PDC looks like this:
/data2/5/14/N/WILCO

Just figured I'd leave my findings here, would be awesome if this could be implemented as a message type.

I am not too familiar with the CPDLC format myself, but I did find a good source of documentation when writing this package: devHazz/hoppie-acars-docs. Maybe this helps?
Other than that there's docs by Hoppie himself, but you need to use webarchive to get to them: Hoppie's ACARS, CPDLC page (Archived April 2024).
As for the implementation, and going by the information provided in the links above, my idea would be to do some light pre-processing (i.e. splitting the message into min, mrn, ...) inside a CpdlcMessage. This would allow for some simple validation, too. Grabbing data from the plaintext message should probably end up as a RegEx in the application, anyway.
PS: I was still typing this as you posted your follow-up ;P Thanks for the info! It does line up with what I gathered so far. I will add the response choices to the pre-processing, probably an Enum type.

Check the dev_cpdlc-message-type branch whether this solution fits your usecase :)
Also, feel free to submit any "problematic" CPDLC messages as test cases when you encounter them in the wild!
Thanks for your help so far!

Haven't tested it yet but absolutely plan to do so tomorrow, but looking at the CpdlcMessage class this looks just perfect.
Also, really appreciate the docs you linked above, I didn't actually come across those pages on my own searches, the stuff that I did figure out came from me poking around in the public Hoppie's message log and correlating some data :-).
Huge thanks once again for implementing this, I'll run some tests ASAP and report back on how it went!
Cheers

Just had a chance to do some testing, and so far this is working perfectly!
>>> cnx.peek()
([(125, CpdlcMessage(from_name='KSFO', to_name='BVI4RK', min=9, rr=CpdlcResponseRequirement.WILCO_UNABLE, message='CLD TO<EMAIL_ADDRESS>@TRUKN2@ DEPARTURE. MNTN<EMAIL_ADDRESS>DPRTR ON<EMAIL_ADDRESS>SQUAWK<EMAIL_ADDRESS>EXPCT RWY<EMAIL_ADDRESS>', mrn=None)), (128, CpdlcMessage(from_name='KUSA', to_name='BVI4RK', min=11, rr=CpdlcResponseRequirement.NOT_REQUIRED, message='LOGON ACCEPTED', mrn=1))]

Have yet to do a full flight with it and will be sure to keep looking out for weird edge cases as well, but so far it's absolutely fantastic! 👍

Awesome! Thanks for the feedback! :)
I'll merge this in to master so it's available in the 0.2.0 release.


babel-plugin-minify-dead-code-elimination SequenceExpression use case

SequenceExpression in ExpressionStatement containing UnaryExpression(void 0) does not remove dead code.
I realize this is a small use case, but if an ExpressionStatement contains a SequenceExpression with void 0 it can be removed with no worry of side-effects.
  a, b, c, d, void 0;

should be:
  a, b, c, d;

Thanks for reading!
-Josh

Thanks for reporting, definitely a valid use case.

Thanks for replying!  I don't suppose I can help?

It's an unsafe transformation -
function f1() {
  return 1, 2, void 0;
}

function f2() {
  return 1, 2;
}

console.log(f1(), f2()); // will print undefined, 2

@boopathi when the parent is an ExpressionStatement it is completely safe.
Also when the sequence expression contains void 0 and it's not the last item it is safe too.
//Any other statement with an expression that contains a sequence expression
let g = (void 0, void 0, 1);

//ExpressionStatement (SequenceExpression)
void 0, 1, 2, 3,;

Example visitor that might be useful.
Check out this example here. It should be correct.
For some reason the transformation yields two sequence expressions but the general idea here is that some void 0 operations are definitely ignored and we can easily determine if it is safe to ignore them.

This is similar to https://github.com/babel/babili/pull/197/files where we remove undefined only when it's in a "safe" context.

Thanks guys!  This is really awesome.

@kangax do you detect to see if undefined is overwritten?  It's possible it might be unsafe, unlike void 0.

You're right, it's unsafe. I looked into fixing it in Babili, then realized it has to do with Babel's path.evaluate() incorrectly thinking that undefined is undefined. Filed a bug — https://github.com/babel/babel/issues/5204


Add *.s and *.ii format to .gitignore file

Add two format *.s and *.ii to .gitignore file

Hi,
I've no problem merging this, but out of curiosity, why are .s and .ii files being created?
-Adam

Hi,
I'm using dnp3 library, in another project, OpenRTU, you can see in my projects. I want to write a make file to build libraries i use, include dnp3. When build libraries finished, i have so many .s and .ii files in my dnp3 folder.

Probably the -save-temps option for gcc: very useful for debugging compile-time errors.
rlc
--
I am Ronald's iPhone and I approve this message (now that I've edited it extensively using autocorrect)

On Aug 12, 2016, at 11:03, Saber Mesgari<EMAIL_ADDRESS>wrote:
Hi,
I'm using dnp3 library, in another project, OpenRTU, you can see in my projects. I want to write a make file to build libraries i use, include dnp3. When build libraries finished, i have so many .s and .ii files in my dnp3 folder.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

I don't know. I just have a lot of these files and when i want to commit my own codes, git tell me you have uncommited code in dnp3 folder.

Cmake supports out-of-tree builds so that you don't have to worry about black listing every possible build output.

I would recommend two things:

that Adam merge your patch
that you build out-of-tree (e.g. in a .build directory) to avoid mixing source and generated files.

rlc
--
I am Ronald's iPhone and I approve this message (now that I've edited it extensively using autocorrect)

On Aug 12, 2016, at 11:17, Saber Mesgari<EMAIL_ADDRESS>wrote:
I don't know. I just have a lot of these files and when i want to commit my own codes, git tell me you have uncommited code in dnp3 folder.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

Thanks

No problem, the existing .gitignore is largely due to our long history with previous build systems like autotools. It's pretty harmless to keep it as is though.


session.run should not require backend when no options are passed

Describe the bug
the docs show that backend is only required in the with Session(...) block, but there is an error when removing options during session.run.
Note: this works when options={} is set
Steps to reproduce
service = QiskitRuntimeService(channel="ibm_quantum",
                               token=os.environ.get('QISKIT_TOKEN'),
                               instance=PROVIDER,
                               url=AUTH_URL)
with Session(service=service, backend=device) as session:
        job = session.run(program_id="hello-world",
                          inputs=program_inputs)

error
if you remove the backend kwarg from the Session context manager it fails:
        if self._service.channel == "ibm_quantum" and not backend:
>           raise ValueError('"backend" is required for ``ibm_quantum`` channel.')
E           ValueError: "backend" is required for ``ibm_quantum`` channel.

.tox/staging/lib/python3.8/site-packages/qiskit_ibm_runtime/session.py:111: ValueError

And if you remove the options dict:
        if channel == "ibm_quantum" and not self.backend:
>           raise IBMInputValueError(
                '"backend" is required field in "options" for "ibm_quantum" channel.'
            )
E           qiskit_ibm_runtime.exceptions.IBMInputValueError: '"backend" is required field in "options" for "ibm_quantum" channel.'

.tox/staging/lib/python3.8/site-packages/qiskit_ibm_runtime/runtime_options.py:93: IBMInputValueError

Expected behavior
session.run uses the backend within the session context instead of explicitly requiring it via options
Additional Information

qiskit-ibm-runtime version: 0.9.3
Python version: 3.10.9
Operating system: tested on both mac os and linux

I tried to reconstruct this failure, but it passed for me. I probably missed something. Can you post a complete test case that fails @asarver ?

And if you remove the options dict:

I wonder if you hit what I did in #879? It was a mistake on my part. Removing the options dict from session.run was fine as long as the session was started with the backend kwarg.

ah yeah, I cannot get this to fail now. Thanks @mriedem


Joining a meeting in new tab

Dear @joisarjignesh , How can we join each meeting in new tab? is there any solution?

Hi, Is there any solution for that? @SamuelWei @joisarjignesh

Do you have any idea? @joisarjignesh

@cod3r0k Most of your questions are not related to this API, this is the wrong place to ask basic php and html questions.
This API is there to help you make api calls to a bbb-server. Everything else needs to be done with the software that is using this api, probably a laravel project.
Some ideas:

Have a javascript in the frontend calling a laravel route, that return the join link. Recieve the url in js and open the url in a new window window.open("https://bbb-server.tld/bigbluebutton/api/....", "_blank"); This could be blocked by some web browsers as it is the same way as popups work
Have the user click on a link and open the page in a new tab xyz] }}" target="_blank">Join meeting xyz The link should go to a laravel controller, that is calling the bbb api and return a redirect to the join url or show a error message on failure.

@cod3r0k Most of your questions are not related to this API, this is the wrong place to ask basic php, html, js questions.
This API is there to help you make api calls to a bbb-server. Everything else needs to be done with the software that is using this api, probably a laravel project.
Some ideas:

Have a javascript in the frontend calling a laravel route, that return the join link. Recieve the url in js and open the url in a new window window.open("https://bbb-server.tld/bigbluebutton/api/....", "_blank"); This could be blocked by some web browsers as it is the same way as popups work
Have the user click on a link and open the page in a new tab xyz] }}" target="_blank">Join meeting xyz The link should go to a laravel controller, that is calling the bbb api and return a redirect to the join url or show a error message on failure.

Hi @SamuelWei , I find a good solution :)
I use this tag and my problem is solved! (onclick="$('form').attr('target', '_blank');")


Improve performance of COUNT aggregates

What is the problem the feature request solves?
The benchmarks in CometAggregateBenchmark show that COUNT is slower than Spark, but SUM is faster than Spark. There should not be so much difference between these two aggregates. I could not reproduce the performance difference in standalone DataFusion.
SUM
Grouped HashAgg Exec: single group key (cardinality 1048576), single aggregate SUM:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (SUM)                                                                    1672           1698          37          6.3         159.4       1.0X
SQL Parquet - Comet (Scan) (SUM)                                                             1913           1993         112          5.5         182.5       0.9X
SQL Parquet - Comet (Scan, Exec) (SUM)                                                        669            798         113         15.7          63.8       2.5X

COUNT
Grouped HashAgg Exec: single group key (cardinality 1048576), single aggregate COUNT:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (COUNT)                                                                    1796           1827          43          5.8         171.3       1.0X
SQL Parquet - Comet (Scan) (COUNT)                                                             1810           1853          61          5.8         172.6       1.0X
SQL Parquet - Comet (Scan, Exec) (COUNT)                                                       2827           2867          56          3.7         269.6       0.6X

Describe the potential solution
No response
Additional context
No response

With the latest DataFusion, I see that count is actually faster than sum.
> select l_shipmode, sum(l_linenumber) from 'lineitem.parquet' group by 1;
+------------+------------------------------------+
| l_shipmode | sum(lineitem.parquet.l_linenumber) |
+------------+------------------------------------+
| AIR        | 257182624                          |
| REG AIR    | 257116345                          |
| MAIL       | 257171951                          |
| FOB        | 257164573                          |
| RAIL       | 257195346                          |
| SHIP       | 257156584                          |
| TRUCK      | 257165514                          |
+------------+------------------------------------+
7 row(s) fetched.
Elapsed 1.136 seconds.

> select l_shipmode, sum(1) from 'lineitem.parquet' group by 1;
+------------+---------------+
| l_shipmode | sum(Int64(1)) |
+------------+---------------+
| RAIL       | 85734456      |
| AIR        | 85729153      |
| SHIP       | 85718196      |
| TRUCK      | 85719689      |
| REG AIR    | 85710420      |
| MAIL       | 85714788      |
| FOB        | 85711200      |
+------------+---------------+
7 row(s) fetched.
Elapsed 1.039 seconds.

> select l_shipmode, count(l_linenumber) from 'lineitem.parquet' group by 1;
+------------+--------------------------------------+
| l_shipmode | count(lineitem.parquet.l_linenumber) |
+------------+--------------------------------------+
| REG AIR    | 85710420                             |
| FOB        | 85711200                             |
| TRUCK      | 85719689                             |
| SHIP       | 85718196                             |
| AIR        | 85729153                             |
| RAIL       | 85734456                             |
| MAIL       | 85714788                             |
+------------+--------------------------------------+
7 row(s) fetched.
Elapsed 1.027 seconds.

> select l_shipmode, count(*) from 'lineitem.parquet' group by 1;
+------------+----------+
| l_shipmode | count(*) |
+------------+----------+
| REG AIR    | 85710420 |
| FOB        | 85711200 |
| TRUCK      | 85719689 |
| MAIL       | 85714788 |
| SHIP       | 85718196 |
| RAIL       | 85734456 |
| AIR        | 85729153 |
+------------+----------+
7 row(s) fetched.
Elapsed 1.122 seconds.

When running a cluster, I cannot reproduce this issue
scala> spark.time(spark.sql("select l_shipmode, sum(1) from lineitem group by 1").collect)
24/07/31 16:53:59 WARN CometSparkSessionExtensions$CometExecRule: Comet cannot execute some parts of this plan natively (set spark.comet.explainFallback.enabled=false to disable this logging):
CometHashAggregate
+-  AQEShuffleRead [COMET: AQEShuffleRead is not supported]
   +- CometSinkPlaceHolder
      +- CometExchange
         +- CometHashAggregate
            +- CometScan parquet

Time taken: 1572 ms
res3: Array[org.apache.spark.sql.Row] = Array([AIR,85729153], [MAIL,85714788], [RAIL,85734456], [SHIP,85718196], [TRUCK,85719689], [REG AIR,85710420], [FOB,85711200])

scala> spark.time(spark.sql("select l_shipmode, count(1) from lineitem group by 1").collect)
24/07/31 16:54:06 WARN CometSparkSessionExtensions$CometExecRule: Comet cannot execute some parts of this plan natively (set spark.comet.explainFallback.enabled=false to disable this logging):
CometHashAggregate
+-  AQEShuffleRead [COMET: AQEShuffleRead is not supported]
   +- CometSinkPlaceHolder
      +- CometExchange
         +- CometHashAggregate
            +- CometScan parquet

Time taken: 1633 ms
res4: Array[org.apache.spark.sql.Row] = Array([AIR,85729153], [MAIL,85714788], [RAIL,85734456], [SHIP,85718196], [TRUCK,85719689], [REG AIR,85710420], [FOB,85711200])

I managed to reproduce with a real-world query in a cluster.
Count
scala> val df = spark.sql("select cs_sold_date_sk, cs_item_sk, cs_order_number, count(*) from catalog_sales group by cs_sold_date_sk, cs_item_sk, cs_order_number")
df: org.apache.spark.sql.DataFrame = [cs_sold_date_sk: int, cs_item_sk: int ... 2 more fields]

scala> spark.time(df.write.format("noop").mode("overwrite").save())
24/08/05 16:13:44 WARN CometSparkSessionExtensions$CometExecRule: Comet cannot execute some parts of this plan natively (set spark.comet.explainFallback.enabled=false to disable this logging):
 OverwriteByExpression [COMET: OverwriteByExpression is not supported]
+- CometHashAggregate
   +- CometExchange
      +- CometHashAggregate
         +- CometScan parquet

24/08/05 16:14:03 WARN CometSparkSessionExtensions$CometExecRule: Comet cannot execute some parts of this plan natively (set spark.comet.explainFallback.enabled=false to disable this logging):
CometHashAggregate
+-  AQEShuffleRead [COMET: AQEShuffleRead is not supported]
   +- CometSinkPlaceHolder
      +- CometExchange
         +- CometHashAggregate
            +- CometScan parquet

Time taken: 134709 ms

Sum
scala> val df = spark.sql("select cs_sold_date_sk, cs_item_sk, cs_order_number, sum(cs_quantity) from catalog_sales group by cs_sold_date_sk, cs_item_sk, cs_order_number")
df: org.apache.spark.sql.DataFrame = [cs_sold_date_sk: int, cs_item_sk: int ... 2 more fields]

scala> spark.time(df.write.format("noop").mode("overwrite").save())
24/08/05 16:16:53 WARN CometSparkSessionExtensions$CometExecRule: Comet cannot execute some parts of this plan natively (set spark.comet.explainFallback.enabled=false to disable this logging):
 OverwriteByExpression [COMET: OverwriteByExpression is not supported]
+- CometHashAggregate
   +- CometExchange
      +- CometHashAggregate
         +- CometScan parquet

24/08/05 16:16:58 WARN CometSparkSessionExtensions$CometExecRule: Comet cannot execute some parts of this plan natively (set spark.comet.explainFallback.enabled=false to disable this logging):
CometHashAggregate
+-  AQEShuffleRead [COMET: AQEShuffleRead is not supported]
   +- CometSinkPlaceHolder
      +- CometExchange
         +- CometHashAggregate
            +- CometScan parquet

Time taken: 11439 ms

Debug logging with stats
Sum
  AggregateExec: mode=Final, gby=[col_0@0 as col_0, col_1@1 as col_1, col_2@2 as col_2], aggr=[sum], metrics=[output_rows=7919602, elapsed_compute=1.479441583s]
  AggregateExec: mode=Final, gby=[col_0@0 as col_0, col_1@1 as col_1, col_2@2 as col_2], aggr=[sum], metrics=[output_rows=7928167, elapsed_compute=1.50249403s]
  AggregateExec: mode=Final, gby=[col_0@0 as col_0, col_1@1 as col_1, col_2@2 as col_2], aggr=[sum], metrics=[output_rows=7917152, elapsed_compute=1.521509782s]
  AggregateExec: mode=Final, gby=[col_0@0 as col_0, col_1@1 as col_1, col_2@2 as col_2], aggr=[sum], metrics=[output_rows=7922306, elapsed_compute=1.525740231s]
  AggregateExec: mode=Final, gby=[col_0@0 as col_0, col_1@1 as col_1, col_2@2 as col_2], aggr=[sum], metrics=[output_rows=7917057, elapsed_compute=1.841284372s]

Count
  AggregateExec: mode=Final, gby=[col_0@0 as col_0, col_1@1 as col_1, col_2@2 as col_2], aggr=[count], metrics=[output_rows=9359786, elapsed_compute=54.775684465s]
  AggregateExec: mode=Final, gby=[col_0@0 as col_0, col_1@1 as col_1, col_2@2 as col_2], aggr=[count], metrics=[output_rows=9360626, elapsed_compute=54.641645191s]
  AggregateExec: mode=Final, gby=[col_0@0 as col_0, col_1@1 as col_1, col_2@2 as col_2], aggr=[count], metrics=[output_rows=9359130, elapsed_compute=54.842657479s]
  AggregateExec: mode=Final, gby=[col_0@0 as col_0, col_1@1 as col_1, col_2@2 as col_2], aggr=[count], metrics=[output_rows=9360684, elapsed_compute=54.877841075s]

Looks like the issue is at count implementation?

The two queries take almost identical time with DataFusion standalone, and we are using their count implementation


ENH: Add document-composing tools.

The only public function is create_run. It returns closures over other
functions, passing internal shared state to manage foreign keys, counters,
and some inter-document validation.
Submitted for early review. Do not merge.
Example usage:
import event_model
start_doc, create_descriptor, create_resource, create_stop = event_model.create_run()
print(start_doc)
descriptor_doc, create_event = create_descriptor(data_keys={'motor': {'shape': [], 'dtype': 'number'}, 'image': {'shape': [512, 512], 'dtype': 'number', 'external': 'FILESTORE:'}}, name='pri
mary')
print(descriptor_doc)
resource_doc, create_datum = create_resource(spec='TIFF', root='/tmp', resource_path='stack.tiff', resource_kwargs={})
print(resource_doc)
datum_doc = create_datum(datum_kwargs={'slice': 5})
print(datum_doc)
event_doc = create_event(data={'motor': 0, 'image': datum_doc['datum_id']}, timestamps={'motor': 0, 'image': 0}, filled={'image': False}, seq_num=1)
print(event_doc)
stop_doc = create_stop()
print(stop_doc)

Note: I'm passing the whole document down into the closure for now, here and elsewhere. Eventually I think it would be better to pass down only the parts we need, for both safely (avoid shared mutable state where possible) and ability to reuse pieces of this API independently. For example, ophyd will want to run create_resource and create_datum without having run create_run or having access to a start document.

This is great. Could we also have a context-managed equivalent of create_run that generates stop_doc and runs a callback?

Something like
``python
with create_run as bundle_of_stuff:
...

Having that for all of the generations makes sense to me too...

Having `create_run` return a named tuple or SimpleNamespace would make sense to me.

I am partial to defined in-line functions, but partial makes more sense incase we want those low-level functions else where?

Could we also have a context-managed equivalent of create_run that generates stop_doc and runs a callback?

Having that for all of the generations makes sense to me too...

Sure, I'll try building context managers on top of the functions and see how it looks.

Having create_run return a named tuple or SimpleNamespace would make sense to me.

Yeah, I thought about a namedtuple. Will do.

I am partial to defined in-line functions, but partial makes more sense incase we want those low-level functions else where?

Right. I started with in-line functions and quickly changed course for that reason. For example, ophyd will need to be able to use create_resource without knowing the RunStart yet, maybe putting in {'run_start': None} as a placeholder. It might also be useful to be able to access these things individually in tests, I guess, but it's not that big a deal to just generate them. Meh.

:+1:

For example, ophyd will need to be able to use create_resource without

maybe the Resource factory should just be stand alone them?  How terrible would it look to pass a resource factory into trigger (I think pretty terrible, but could be convinced otherwise).

What's the rationale for defining functions locally? It makes them impossible to individually document and test. It seems a bit too clever to my taste.
If the goal is to make sure they are definitely not part of the public API, I think leaving them out of __all__ is good enough for now. Is there another reason?

I would like to propose merging this in its current form and experimenting with context managers separately, aiming to eventually move experiments up stream when we find a good pattern.

merging as it has been approved and I would like to use it for suitcase_csv tests


[Don't merge] Use the format name as document_type

Currently all documents from publisher are sent to the publishing-api with format placeholder.
format is being deprecated and replaced by document_type and schema_name (RFC 41). This PR will make publisher use Artefact#kind (the format name in publisher) for the document_type. This allows us to determine the type of content from the content item.
Possible values for document_type / kind in publisher are:

answer
business_support
campaign
completed_transaction
guide
help_page
licence
local_transaction
place
programme
simple_smart_answer
transaction
video

https://github.com/alphagov/govuk_content_models/blob/master/app/models/artefact.rb#L67-L79
I've checked that these don't clash with existing formats in the publishing-api:
[3] pry(main)> ContentItem.where(document_type: %w[answer business_support campaign completed_transaction guide help_page licence local_transaction place programme simple_smart_answer transaction video]).count
=> 0

This LGTM, why is it marked as Don't Merge?

👍

@boffbowsh I wanted some extra eyes on this to make sure this was the right thing to do! Happy for this to get merged now.