Question about rectification result

Hello,
I`ve been trying to visualize rectified events by using rectification maps that you provided but I ended with some strange blank lines (upper images) in contrast to normal visualization (lower images). Did I miss some operations that I have to provide first?
Thanks a lot,
Konrad

Hi @KondziuK
How are you visualizing the rectified events here? Are you performing nearest neighbor assignment of the rectified pixel coordinates or performing interpolation?

Hi @magehrig
Iam using nearest neigbor - I am casting rectified coordinates to int. I`ve perform rectification like this:
data = Sequence(seq_path,mode=mode,delta_t_ms=delt_t_ms, num_bins=num_bins)
rectify_map_left = data.rectify_events(data.event_slicers['left'].events['x'][:], data.event_slicers['left'].events['y'][:], 'left')
data.event_slicers['left'].events['x'] = rectify_map_left[:, 0]
data.event_slicers['left'].events['y'] = rectify_map_left[:, 1]
To visualize I`ve used slighty changed EventReader class and render function with new polarity assigment:
mask1 = (x>=0)&(y>=0)&(W>x)&(H>y)
p_new = p[mask1]
mask[y[mask1],x[mask1]]=p_new

Yet another possibility is to use backward warping as explained in another issue: https://github.com/uzh-rpg/DSEC/issues/14#issuecomment-841348958

Let me know if the issue still persists. I am closing this issue for now.

Hi,
Yes, the issue still exists. I tried to implement render function basing on the equation that you provided but I am stuck. I am ending with very big values in interp_weights and I don`t know how then map them into values {-1,0,1} just as it was in function render. I tried to experiment with adding mask like it was in render function but still no good results. Here is the function that I tried somehow to create:
def render2(x ,y , pol , H, W):
  x0 = x.astype(int)
  y0 = y.astype(int)
  img = np.full((H,W,3), fill_value=255,dtype='uint8')
  mask = np.zeros((H, W), dtype='float')
  value = 2 * pol - 1

  for xlim in [x0, x0 + 1]:
      for ylim in [y0, y0 + 1]:
          mask1 = (xlim < W) & (xlim >= 0) & (ylim < H) & (ylim >= 0)
          interp_weights = value * np.abs(1 - (xlim - x)) * np.abs(1 - (ylim - y))
          mask[y0[mask1], x0[mask1]] += interp_weights[mask1]

  ## Here missing good mapping
  img[mask==0]=[255,255,255]
  img[mask==-1]=[255,0,0]
  img[mask==1]=[0,0,255]
  return img

This should work for you:
def render_rect(x: np.ndarray, y: np.ndarray, pol: np.ndarray, t: np.ndarray, H: int, W: int) -> np.ndarray:
    x = x.squeeze()
    y = y.squeeze()
    pol = pol.squeeze()
    t = t.squeeze()
    assert x.size == y.size == pol.size
    assert H > 0
    assert W > 0
    img_acc = np.zeros((H,W), dtype='float32').ravel()

    pol = pol.astype('int')
    x0 = x.astype('int')
    y0 = y.astype('int')
    t = t.astype('float64')
    value = 2*pol - 1

    t_norm = (t - t.min())/(t.max() - t.min())
    t_norm = t_norm**2
    t_norm = t_norm.astype('float32')
    assert t_norm.min() >= 0
    assert t_norm.max() <= 1

    for xlim in [x0,x0+1]:
        for ylim in [y0,y0+1]:
            mask = (xlim < W) & (xlim >= 0) & (ylim < H) & (ylim >= 0)
            interp_weights = value * (1 - np.abs(xlim-x)) * (1 - np.abs(ylim-y)) * t_norm

            index = W * ylim.astype('int') + \
                    xlim.astype('int')

            np.add.at(img_acc, index[mask], interp_weights[mask])

    img_acc = np.reshape(img_acc, (H,W))

    img_out = np.full((H,W,3), fill_value=255,dtype='uint8')

    # Simple thresholding
    #img_out[img_acc > 0] = [0,0,255]
    #img_out[img_acc < 0] = [255,0,0]

    # With weighting (more complicated alternative)
    clip_percentile = 80
    min_percentile = -np.percentile(np.abs(img_acc[img_acc < 0]), clip_percentile)
    max_percentile = np.percentile(np.abs(img_acc[img_acc > 0]), clip_percentile)
    img_acc = np.clip(img_acc, min_percentile, max_percentile)

    img_acc_max = img_acc.max()
    idx_pos = img_acc > 0
    img_acc[idx_pos] = img_acc[idx_pos]/img_acc_max
    val_pos = img_acc[idx_pos]
    img_out[idx_pos] = np.stack((255-val_pos*255, 255-val_pos*255, np.ones_like(val_pos)*255), axis=1)

    img_acc_min = img_acc.min()
    idx_neg = img_acc < 0
    img_acc[idx_neg] = img_acc[idx_neg]/img_acc_min
    val_neg = img_acc[idx_neg]
    img_out[idx_neg] = np.stack((np.ones_like(val_neg)*255, 255-val_neg*255, 255-val_neg*255), axis=1)
    return img_out

Sometimes you still see the effects of rectification and forward warping but it is less visible.
It is up to you how you want to visualize events. There is no right or wrong. If you really care about removing rectification artefacts from the visualization you should use backward warping as mentioned above.

Hi @magehrig ,
thanks for your help, this works great!


Google Cloud Speech Support Speaker Diarization

Google Speech API V1p1beta1 now supports a bunch of new recognition configurations.
I have forked this library in order to be able to pass values for these new parameters to the API.
I've added enableSpeakerDiarization and diarizationSpeakerCount to my fork, though there may be other new parameters the API now accepts that would be useful for other users of this library.
Also, the response object needs to include speaker_tag for the enabling of the diarization to be useful.
Here is the commit from my fork to help explain what I added to gain access to this new feature: https://github.com/aloeai/google-cloud-ruby/commit/d3cd520c5c5ec90e639a5d3f9188c78253fefc3f

@dazuma These attributes look to have been added to v1p1beta1 3 weeks ago in https://github.com/googleapis/googleapis/commit/475d72b7405c92f06d7f2d4aba866278eb5ad8e9. Shouldn't these changes have already been propagated to this repo by now?

Is there an ETA on when these options will be rolled in here?

I believe these were added in #2227. We are in the process of automating the generation of the ruby files when the underlying protobuf files are updated. Once that process is complete and we are confident everything is up to date we will release. I'm thinking later this week, or early next week.

This is included in the most recent release.
http://googlecloudplatform.github.io/google-cloud-ruby/docs/google-cloud-speech/latest/Google/Cloud/Speech/V1p1beta1/RecognitionConfig.html#enable_speaker_diarization-instance_method


terraform apply fails to setup hcloud_server.lb

What happened:
Followed Hetzner install guideline https://github.com/kubermatic/kubeone/blob/master/docs/quickstart-hetzner.md  but it fails to connect to hcloud_server.lb on terraform apply
hcloud_server.lb: Still creating... (1m20s elapsed)
hcloud_server.lb (remote-exec): Connecting to remote host via SSH...
hcloud_server.lb (remote-exec):   Host: <IP_ADDRESS>
hcloud_server.lb (remote-exec):   User: root
hcloud_server.lb (remote-exec):   Password: false
hcloud_server.lb (remote-exec):   Private key: false
hcloud_server.lb (remote-exec):   SSH Agent: false
hcloud_server.lb (remote-exec):   Checking Host Key: false
hcloud_server.lb: Still creating... (1m30s elapsed)
hcloud_server.lb: Still creating... (1m40s elapsed)

What is the expected behavior:
terraform apply succeeds.
How to reproduce the issue:
Follow https://github.com/kubermatic/kubeone/blob/master/docs/quickstart-hetzner.md up to terraform apply
Anything else we need to know?
I was able to fix this by adding private_key = "${file("/root/.ssh/id_rsa")}" line in the connection sections of main.tf.  I'm not familiar with Terraform and I don't know what's the conventional way of setting up this, but anyways, it will be great to document I think.
Information about the environment:
KubeOne version (kubeone version): 0.6
Operating system: Windows 10 WSL Ubuntu
Provider you're deploying cluster on: Hetzner
Operating system you're deploying on:

Hi, thanks for letting us know.
One of ether should be provided, private_key in terraform (and it should be unencrypted) or you encrypted SSH key should be in the ssh-agent. I'm not really sure if it's possible to use ssh-agent in Windows 10 WSL Ubuntu, but most likely yes, it should be accessible.
P.S.
Windows 10 WSL Ubuntu oh wow, didn't expected that :) 👍  Currently we testing k1 only on linux and macos.

#460 will supersede this issue.

update — we have some new docs regarding SSH requirements: https://github.com/kubermatic/kubeone/blob/master/docs/ssh.md


Force IndexFormat  value to lowercase

when using the index format as below, the code will not show any error, yet it will not log to elasticsearch because the indexing format should be lowered-case. This should be handled somewhere.
    var logger = new LoggerConfiguration()
            .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200"))
            {
                IndexFormat = "TESTING",
            }).CreateLogger();

Is fixed in dev branch, will go to master soon. Tx.


Question : What is the use of to_b64() and from_b64() ? Why are they used?

to_b64 is "to base64", from_b64 is "from base64"
These, are helper utilities to convert a python dictionary object (well, anything that can be json.dumps) into a serialized string without special characters.
They are used to wrap the inputs to the "prompt" (lambdaprompt definition of prompt) that takes in a few inputs and provides completion.
See these lines for the prompt_kwargs creation
https://github.com/approximatelabs/sketch/blob/b4d5f257e1d9d0da423562f0d80ac343624b7ede/sketch/pandas_extension.py#L152
This is now immediately usable by the requests lib, when calling a prompt remotely
response = requests.get(
    f"{url}/prompt/{prompt.name}",
    params=prompt_kwargs,
)

Because the fields in prompt_kwargs are processed to be strings using the to_b64 function, they now are able to be passed as params to the prompt completion endpoint without any messy serialization.
In summary: the point is to serialize and deserialize the information that is used by the prompt into a format that makes it possible to send over requests so that the prompt can be completed by the remote lambdaprompt server (when SKETCH_USE_REMOTE_LAMBDAPROMPT==True)


Find an answer to “What is Solid?”

Is it not better to discuss here this https://github.com/orgs/solid/teams/explaining-the-vision-panel/discussions ? I would like to understand the Vision etc. first as well which we started discussing there.

Not sure, never worked with panels before. However, I didn’t get a notification for the thing you posted there, so that’s not a good sign. I’d hence suggest issues.

Trying this as a discussion https://github.com/orgs/solid/teams/explaining-the-vision-panel/discussions/4

My take:
Solid is an ecosystem of technologies that fosters innovation and empowers people by offering precise control over data across application and organization boundaries.

Counter
Solid is an ecosystem of [] working to provide people with the freedom to choose who has access to their information and what software and services they can use with it.

In general, I think it is important to emphasize both sides: the people whose data this is about, and those doing things with that data. Today's problems aren't limited to people: Facebook is not just my problem, it's also the problem of whoever wants to develop a nice Web application that needs people to log in and provide a couple of basic pieces of data.
And also, Solid is not just about personal data; it equally holds for company data.
When in doubt, it's probably useful to think about the Web. The Web is not just also a place for people to find information, it's also where companies do business.

Would it be better to refer to them as 'web users' rather than people? That could cover organisations and groups with singular identities too

Just going back to the embodiment noun, I find 'eco-system' a little too vague unless we can add specificity around what it's an ecosystem of. I like that it makes the scope for what Solid can constitute really broad but for people to get what it is I think we'd need to give them more of a cue to its actual manifestations.

'web users' rather than people

I'm not a big fan of calling people "users" 🙂
(Why is it that only drug dealers and software vendors refer to their customers as "users" 😉)
Also because this is not just about people on the Web. Is about people.
Including people going to a supermarket or a bank, because these also involve personal data.

I find 'eco-system' a little too vague unless we can add specificity around what it's an ecosystem of

See my suggestion above: "around technology that…".
Can also be "ecosystem of people, data, apps" (have used that before), but that phrasing would have caused repetition in my definition.

I'm not a big fan of calling people "users" 🙂
(Why is it that only drug dealers and software vendors refer to their customers as "users" 😉)
Ha how very true! :)

Ultimately, this all centres around identities doesn't it? What the individuals, businesses and organisations all have in common is a unique identity to which data is attributed. Is there a way to phrase it around that somehow?

Also because this is not just about people on the Web. Is about people.
Including people going to a supermarket or a bank, because these also involve personal data.
OK so we really are looking that expansively? In that case we definitely have to express all this in everyday terms.

have to express all this in everyday terms

Not necessarily; we can also just leave some things open, and not express them. As long as what we express doesn't exclude it, we're good.
My definition still meets those criteria, for instance. (It might of course still have other deficits.)

Tried to reflect the points made:
"Solid is an ecosystem of people, standards and software ensuring you have control over your data and the freedom to choose and change the software and services you use."

Tried to reflect the points made:
"Solid is an ecosystem of people, standards and software ensuring you have control over your data and the freedom to choose and change the software and services you use."

A question worth reflecting on is "ecosystem" the main point we want to get across about what Solid is. Because could it be:
"Solid is technology that enables you to have control over your data and the freedom to choose and change the software and services you use."

I guess it ultimately comes down to what Solid aims to provide. Is it software solutions for everyone to use? Is it standards for other to follow in creating solutions? Is it a collective of people driving change? Being all three seems a bit of a stretch...

I guess it ultimately comes down to what Solid aims to provide. Is it software solutions for everyone to use? Is it standards for other to follow in creating solutions? Is it a collective of people driving change? Being all three seems a bit of a stretch...

To say it is "Technology" classifies it in way that most people would be familiar dumping a lot of concepts into: including expertise of people, hardware, software, design patterns, protocols, math etc.. .
"Ecosystem" does imply a community in an environment interacting. That implies there is some governance which the W3C influence of course is. The technology , in its many forms, is the output of that. So that would imply Ecosystem comes first.

How about:
"Solid is an Ecosystem [feels like "of something" requires] working to provide technology that enables you to have control over your data and the freedom to choose and change the software and services you use."

I like the idea of an ecosystem very much, but in the time I spent before joining Inrupt, I found it really, really hard to get people to grasp the idea. So, I'm not sure it would be understood.
It might be that we should constrain ourselves to what are our technical contributions. We might then try to explain what social and economical impacts we anticipate, but I think that might be overstating our prophetic abilities ;-) Instead, perhaps we should say that we do this in the hope that society will take this to bring about such and such social changes, emphasizing that is an extension of the project's goals.

How a message house might shape up.

I like the idea of an ecosystem very much, but in the time I spent before joining Inrupt, I found it really, really hard to get people to grasp the idea. So, I'm not sure it would be understood.

Do you meant they didn't grasp the idea that Solid is a community or a *movement working on a technology to put people in control of their data?

It might be that we should constrain ourselves to what are our technical contributions. We might then try to explain what social and economical impacts we anticipate, but I think that might be overstating our prophetic abilities ;-) Instead, perhaps we should say that we do this in the hope that society will take this to bring about such and such social changes, emphasizing that is an extension of the project's goals.

Like "Solid is a technical specification and architecture that separates data from software so that there is more freedom, choice and control over how it is used"
or
Like "Solid is a technical specification and architecture that enables you to place data in a store of your choice and under your control, and for software to be able to easily provide useful features without needing to take control of your data."

It might be that we should constrain ourselves to what are our technical contributions. We might then try to explain what social and economical impacts we anticipate, but I think that might be overstating our prophetic abilities ;-)

@kjetilk I guess in that regard we have our Vision and Mission (which I won't get into in this Issue) which give us scope to talk of much broader impact without implication of Solid delivering it all on its own.

Just applying the message house to :
"Solid is an ecosystem around technologies that foster innovation and empower people by offering precise control over data across application and organization boundaries."

fyi
I just used the following in an email to a Government Department in the UK:
"Solid is a technology that builds on the current Web standards that enables people to have control over their data and the freedom to choose and change the software and services they use."

Hmm feels like we're swirling around lots of different versions now but not sure we've answered some fundamental questions yet.
Before we look at any more lines @pjworrall can I suggest we break the statement down a bit so each aspect can be debated? To my mind, it consists of at least 3 aspects:
Solid is [the form it takes] that [what it does] for [who it serves]
If we can just focus on answering the first one - [the form it takes] - to begin with then we can move on to the rest and build an accurate statement.
There are concerns about the word 'eco-system' but are we all agreed that Solid takes more than just a singular form e.g. tool, standard, convention, community and is in fact a combination of multiple forms?

Do you meant they didn't grasp the idea that Solid is a community or a *movement working on a technology to put people in control of their data?

To the extent that I can decipher things from the blank stares I was getting, I think that the problem was that they had a very functional view of technology, in the sense that technology companies operate within a certain domain and things they introduce has certain range of expected outcomes. Also, the domain is managed by the leadership and the board, and the range is understood in terms of markets and market share, regulations and such. In that world, both domain and range are rather constrained things, but instrumental to understand these boundaries for them. So, they were primarily interested in the domain that it would be operating within, and how I had analyzed the range.
An ecosystem is an entirely different thing, the domain is "everything" in the sense that it affects everything, social life, all commercial life, governments, and the range of expected outcomes for a commercial entity is much harder to see. So, I had a total failure in trying to bridge my own ecosystem thinking with that much more constrained view.
So, my concerns are mainly around two points:

We have to acknowledge that the actual social impact is unknown to us. We have expectations, visions and dreams, sure, and that is why we're doing this in the first place (or I certainly am). Nevertheless, what will actually happen after introduction of the technology is uncertain, and properly equipped social and political scientists are probably better suited to study it than we are ourselves. Therefore, I think we should have some humility when formulating the answer to the "what is"-question.
Secondly, there are those that need a precise definition, but do not have the frame of mind to accept a definition of a technology that is mainly motivated from its social and commercial impact on whole societies.

I think that explaining the expected social and commercial impact is better suited when giving a talk on Solid, than in here.

I agree.

Solid is [the form it takes] that [what it does] for [who it serves]
If we can just focus on answering the first one - [the form it takes] - to begin with then we can move on to the rest and build an accurate statement.

If we are not too sure Ecosystem is concrete enough them maybe the form it takes is:
..technical standards, design patterns and software tools..

Solid: Individuals controlling their data
[This is the main point, in 4 words.]

Mitzi added some quotes from timbl on here https://github.com/solid/Explaining-the-Vision-Panel/tree/master/Tim-describing-Solid which I think are useful in framing this conversation. The form Solid takes being a 'standard', for instance.
We also have https://solid.mit.edu/ where it's referred to as 'a (proposed) set of conventions and tools'.

"Solid is a standard" :)

(then need to insert what standard should do)


Setting Angle of Annotation

Hi,
I am trying to implement a scene: there are several shops on a street, and each shop has an annotation in front of them, the direction of the notation is fixed relative to the face of the shop. So user can only see clearly the annotation when they walk right in front of the shop.
Currently it seems all annotations are facing directly to the user, no matter that is the relative position between the user and the shop.
It would be great if there is a feature or a way to achieve this implementation, as it gives user more immersive experience.

Main purpose of this library is the ability for the user to open app in the middle of the city and see what is around him. If annotations don't face him then he can't read it.
Regarding your suggestion:
For this to work, library will have to know orientation of every shop, unless you provide it, I don't see a way to get it. If you have orientations of those shops than you can make ARAnnotation subclass, add orientation property, and in ARAnnotationView subclass use that to set CATransform.
It is an interesting concept but usable only in specific situations.


[BUG] error on GPU

Thanks for sharing the awesome work.
I got the following error when running the example on GPU
python example.py
2024-01-13 21:46:44.846591: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-01-13 21:46:44.846624: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-01-13 21:46:44.848009: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-01-13 21:46:44.854341: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-01-13 21:46:45.757644: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-01-13 21:46:47,012 - numexpr.utils - INFO - Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2024-01-13 21:46:47,013 - numexpr.utils - INFO - NumExpr defaulting to 8 threads.
Traceback (most recent call last):
  File "/home/jma/Documents/sanity_test/mamba/mamba/MambaTransformer/example.py", line 17, in <module>
    out = model(x)
  File "/home/jma/Documents/sanity_test/mamba/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/jma/Documents/sanity_test/mamba/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/jma/Documents/sanity_test/mamba/mamba/MambaTransformer/mamba_transformer/model.py", line 213, in forward
    x = mamba(x) + x
  File "/home/jma/Documents/sanity_test/mamba/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/jma/Documents/sanity_test/mamba/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/jma/Documents/sanity_test/mamba/lib/python3.9/site-packages/zeta/nn/modules/simple_mamba.py", line 118, in forward
    y = self.ssm(x)
  File "/home/jma/Documents/sanity_test/mamba/lib/python3.9/site-packages/zeta/nn/modules/simple_mamba.py", line 158, in ssm
    y = self.selective_scan(
  File "/home/jma/Documents/sanity_test/mamba/lib/python3.9/site-packages/zeta/nn/modules/simple_mamba.py", line 205, in selective_scan
    x = deltaA[:, :, i] * x + deltaB_u[:, :, i]
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

To Reproduce
import torch
from mamba_transformer import MambaTransformer, MambaTransformerblock

x = torch.randn(10, 25, 512).to("cuda:0")
model = MambaTransformerblock(
    dim=512,
    heads=8,
    depth=4,
    dim_head=64,
    d_state=512,
    dropout=0.1,
    ff_mult=4
).to("cuda:0")

# # Pass the input tensor through the model and print the output shape
out = model(x)

print(f"{x.shape=}, {out.shape=}")

@JunMa11 so you're trying to run the model on cuda but there are other parts of the model running on the cpu, the problem comes from the modules inside MambaTransformer.
I can fix this by making it more customize-able, until then you can keep trying things out.


Index Lifecycle Management Policy API v7

This is a cherry-pick of @aarontami's https://github.com/olivere/elastic/pull/1105 on release-branch.v7.

Wonderful. Thank you! I hope to get 7.0.2 out over the weekend.


failed to create database like "demo-test" if "-" in db name

https://github.com/openGemini/opengemini-client-go/blob/40fd91311772c9bc3e8f80939a865d787d037bba/opengemini/database.go#L34-L34

https://github.com/openGemini/opengemini-client-go/blob/40fd91311772c9bc3e8f80939a865d787d037bba/opengemini/database.go#L88
failed to create database like "demo-test" if "-" in db name.
How to elegantly solve db names that require quotation marks and those that do not require quotation marks.

closed by #47


express checkout button image loading

we have installation of <IP_ADDRESS> with 2 languages english and greek .
The button displays the image ''us'' even if we are at greek language and the reason is because greek language iso code is ''el'' and no ''GR'' as the greek country code . If i understand correct the value for choosing the correct image is language code and no country code .
Awaiting for your answer thanks

Hello,
Thank you for reporting us this issue !
We will fix it in the next version of the module.
Thank you for your help and have a nice day !
Best regards,
202 ecommerce

Hello,
We've sent the new version of the module v5.2.0 to PrestaShop for validation.
It will be available on PrestaShop addons next week.
Thank you and have a nice day !


Mesh.interpolate() using another mesh rather than a raster

Hi there,
Is it possible to interpolate the depths from another mesh rather than a raster? I've made some manual edits to some depths on an older mesh, and I want to include them in a new mesh, but can't make the edits in the same way. Is there some work around I might be able to do by using the mesh depths to edit the original raster?
Cheers,
Tom

OK, so here is my understanding of your requirement:

You have an old mesh that extends up to coastline (0m) which has all elevations above (-0.5m) set to -0.5
You have a new mesh that extends up to 1m that has correct elevation values (at least from 0m to 1m)
You'd like your new mesh elevations to be modified such that

All its nodes that fall within the multipolygon of the old mesh get their elevation interpolated from the old mesh
All its nodes that fall within the region of the old mesh that has -0.5 value have elevations equal to -0.5
All the nodes that fall out of the multipolygon of the old mesh have their values from the original new mesh (or be interpolated form rasters?)

If that's the case my suggestion is to do masking and setting values at jigsaw_msh_t level, which is basically a numpy array with special dtype. OCSMesh provides some utility functions to help with that such as:
https://github.com/noaa-ocs-modeling/OCSMesh/blob/8e2d43e4e3d74f04780dec9913debe7811c4e779/ocsmesh/utils.py#L193
https://github.com/noaa-ocs-modeling/OCSMesh/blob/8e2d43e4e3d74f04780dec9913debe7811c4e779/ocsmesh/utils.py#L781-L786
https://github.com/noaa-ocs-modeling/OCSMesh/blob/8e2d43e4e3d74f04780dec9913debe7811c4e779/ocsmesh/utils.py#L1283
All of which work on jigsaw_msh_t objects. Then these could be used along with numpy slicing and masking to get the desired results:
new_msht = newmesh_obj.msh_t
old_msht = oldmesh_obj.msh_t

old_shape = ocsmesh.utils.get_mesh_polygons(old_msht)
vert_idx = ocsmesh.utils.get_verts_in_shape(new_msht, old_shape)
new_msht.value[vert_idx] = ...

Or something along these lines (including interpolation, etc.)

Thanks Soroosh, will give it a go

@TPCollings any luck with using OCSMesh for this specific case?

I'm closing this ticket assuming either the issue was resolved or is no longer relevant. Please feel free to reopen it if needed.


There are always errors in log even though all data is synced successfully

Here are the logs, I'm a little concerned about whether it will corrupt my files. Is this normal or I have configured something wrong? Thanks in advance.
2023/4/4 16:28:53->HTTP:GET to:/_changes?style=all_docs&feed=longpoll&heartbeat=30000&since=181-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M5MT4XKMBubpxomGicgq4Yh_Y8FiDJ0ACk_kNNCQObYpZoYmmRZI6uJwsAfSgrOA&limit=250 -> failed
2023/4/4 16:28:53->TypeError:Failed to fetch
2023/4/4 16:28:53->replication paused
2023/4/4 16:28:55->replication paused
2023/4/4 16:30:08->HTTP:GET to:/_changes?style=all_docs&feed=longpoll&heartbeat=30000&since=181-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M5MT4XKMBubpxomGicgq4Yh_Y8FiDJ0ACk_kNNCQObYpZoYmmRZI6uJwsAfSgrOA&limit=250 -> failed
2023/4/4 16:30:08->TypeError:Failed to fetch
2023/4/4 16:30:08->replication paused
2023/4/4 16:30:09->replication paused
2023/4/4 16:31:22->HTTP:GET to:/_changes?style=all_docs&feed=longpoll&heartbeat=30000&since=181-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M5MT4XKMBubpxomGicgq4Yh_Y8FiDJ0ACk_kNNCQObYpZoYmmRZI6uJwsAfSgrOA&limit=250 -> failed
2023/4/4 16:31:22->TypeError:Failed to fetch
2023/4/4 16:31:22->replication paused
2023/4/4 16:31:24->replication paused

Sorry for being late!
HTTP 504 error means Gateway timed out.
Self-hosted LiveSync watches the remote database for LiveSync by long pooling. If you are behind the proxy, the timeout of the gateway might be shorter than the one the LiveSync waits for, or the proxy detects the no-transferring and disconnect them.
Could you please try enabling Use timeouts instead of heartbeats?

I try to enable the option but the errors remains. Then I find it is CDN that causes the problem, when I stop using it everything work fine, though I don't know the specific reson.
Thanks for your great work again!

@WinkeyLin I am facing the same issue. Can you please elaborate what you mean by CDN?


Self-started bloop server doesn't start

Version(s)
0.1.0, x86 mac binary downloaded from releases tab.
Describe the bug
scala-cli bloop start fails every time. However, trying to compile some sources successfully starts bloop.

Stack trace
java.lang.Exception: Server didn't start
  scala.build.blooprifle.internal.Operations$.$anonfun$startServer$2(Operations.scala:153)
  scala.build.blooprifle.BloopRifleLogger.$anonfun$runnable$1(BloopRifleLogger.scala:12)
  java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
  java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
  java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  java.lang.Thread.run(Thread.java:833)
  com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:597)
  com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:194)

To Reproduce

Install scala-cli
scala-cli bloop start
profit

Expected behaviour
A server starts just like when running a build :)

I didn't check the Scala CLI sources yet, but it seems scala-cli bloop start tries to use the system JVM to start Bloop. Locally, with Java 11 as system JVM, it says:
$ java -version
openjdk version "<IP_ADDRESS>" 2020-11-04
OpenJDK Runtime Environment AdoptOpenJDK (build <IP_ADDRESS>+1)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build <IP_ADDRESS>+1, mixed mode)

$ scala-cli bloop start -v -v -v -v
…
Error: VM option 'UseZGC' is experimental and must be enabled via -XX:+UnlockExperimentalVMOptions.
Error: The unlock option must precede 'UseZGC'.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
…

while the following works for me:
$ eval "$(cs java --env --jvm temurin:17)"
$ java -version
openjdk version "17" 2021-09-14
OpenJDK Runtime Environment Temurin-17+35 (build 17+35)
OpenJDK 64-Bit Server VM Temurin-17+35 (build 17+35, mixed mode, sharing)

$ scala-cli bloop start
Bloop server started.

I guess we should ensure that our JVM 17 is used by default by scala-cli bloop start.
@kubukoz What does java -version says on your system?

openjdk version "11.0.11" 2021-04-20 LTS
OpenJDK Runtime Environment Zulu11.48+21-CA (build 11.0.11+9-LTS)
OpenJDK 64-Bit Server VM Zulu11.48+21-CA (build 11.0.11+9-LTS, mixed mode)

Ideally JAVA_HOME or whatever is in the PATH of the scala-cli binary would be used, this way I can scope the usage of a JVM 17 for scala-cli's usage only.

I'm having the same problem on Apple M1.
And I agree with @kubukoz - I already run Java 17 natively built for arm64, and I wish scala-cli would just use that.
velvetbaldmime@Antons-Air /v/f/m/d/T/tmp.plBHFF7i> java -version
openjdk version "17.0.2" 2022-01-18
OpenJDK Runtime Environment Temurin-17.0.2+8 (build 17.0.2+8)
OpenJDK 64-Bit Server VM Temurin-17.0.2+8 (build 17.0.2+8, mixed mode)

velvetbaldmime@Antons-Air /v/f/m/d/T/tmp.plBHFF7i> scala-cli --version
0.1.0
velvetbaldmime@Antons-Air /v/f/m/d/T/tmp.plBHFF7i> scala-cli test.scala
Bloop daemon status: not running
Starting Bloop 1.4.19 at /Users/velvetbaldmime/Library/Caches/ScalaCli/bloop/daemon using JVM /Users/velvetbaldmime/Library/Caches/Coursier/arc/https/github.com/adoptium/temurin17-binaries/releases/download/jdk-17%252B35/OpenJDK17-jdk_x64_mac_hotspot_17_35.tar.gz/jdk-17+35/Contents/Home/bin/java
Error: java.lang.Exception: Server didn't start
For more details, please see '/private/var/folders/my/drt5584s2w59ncgxgxpq7v040000gn/T/tmp.plBHFF7i/.scala-build/stacktraces/1643998345-954539311384192108.log

Currently I can't find a way to use 0.1.0 so I dropped it down to 0.0.9
It's particularly important on M1 because I don't want compilation to run through emulator - and scala-cli will see os.arch as x86 if the binary is running under rosetta.

@keynmol When using standard commands, --bloop-jvm should work, so something like this should run fine on M1:
$ scala-cli --bloop-jvm system test.scala

Seems we need to handle the architecture more carefully though… (and also default to the system JVM when it's at least Java 17)

Related: https://github.com/coursier/coursier/issues/2335

@alexarchambault nope, seems like the same issue:
velvetbaldmime@Antons-Air /v/f/m/d/T/tmp.YhCQZV6k [1]> scala-cli --bloop-jvm system test.scala
Bloop daemon status: not running
Starting Bloop 1.4.19 at /Users/velvetbaldmime/Library/Caches/ScalaCli/bloop/daemon using JVM /Library/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home/bin/java
Error: java.lang.Exception: Server didn't start
For more details, please see '/private/var/folders/my/drt5584s2w59ncgxgxpq7v040000gn/T/tmp.YhCQZV6k/.scala-build/stacktraces/1644836015-15645112221190994336.log'

@keynmol At least it tries to use your JVM 😅
Can you run the same command with -v -v -v -v, and paste the output here?

 I didn't notice and was too quick to post a rebuttal :P
After a lot of jars:
Checking for a running Bloop server at /Users/velvetbaldmime/Library/Caches/ScalaCli/bloop/daemon ...
Attempting to connect to Bloop server /Users/velvetbaldmime/Library/Caches/ScalaCli/bloop/daemon ...
Connection attempt result: None
Error occurred during initialization of VM
Multiple garbage collectors selected
Exception in thread "main" java.lang.Exception: Server didn't start
	at scala.build.blooprifle.internal.Operations$.$anonfun$startServer$2(Operations.scala:153)
	at scala.build.blooprifle.BloopRifleLogger.$anonfun$runnable$1(BloopRifleLogger.scala:12)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.lang.Thread.run(Thread.java:833)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:597)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:194)

which led me to my custom bloop config in ~/.bloop/bloop.json:
...
"javaOptions": ["-Xmx4G", "-XX:+UseShenandoahGC"]
...

After I removed that option it ran fine.
I feel like when using "system" bloop scala-cli should respect user's GC settings.

--bloop-jvm system works for me, but it'd obviously be great if it wasn't required. Can scala-cli just check if there's a JDK17+ compatible jvm on its PATH and use just that?

I believe I've encountered it before...
Caused by: libdaemonjvm.server.LockError$ZombieFound: Cannot connect to process 1905

@julienrf have you tried killing that process? I believe it fixed it for me in the past.
kill -9 1905

The process that had the PID 1905 was the NFS server. For the record, I did a systemctl restart nfs-server.service that fixed the issue.
However, I believe it would be great to improve the error reporting.


Unable to install NuGet package InTheHand.Net.Bluetooth 4.0.5 into .NET Framework 4.8 project

Created a brand .NET Framework 4.8 Console App in Visual Studio 2019. Tried to install NuGet package InTheHand.Net.Bluetooth 4.0.5. Got an error:
Package 'MSBuild.Sdk.Extras 2.0.54' has a package type 'MSBuildSdk' that is not supported by project 'ConsoleApp'.
But 4.0.4 installs just fine, without any errors.

Same here in Windows Forms application.

This is fixed in 4.0.6


Provide a way for maintainers of ports to check that they're using equivalent HTML markup

What
Provide a way for maintainers of ports to ensure that, for given scenarios, their port generates markup equivalent to that of GOV.UK Frontend.
Why
Whilst anyone can easily re-use the stylesheets and JavaScript from GOV.UK Frontend, most of our users have to maintain their own abstractions of the HTML markup used for a component.
It's important for that markup to be equivalent to the markup used in GOV.UK Frontend so that it renders the same and has the same accessibility. This is especially true when those abstractions are published as packages to be re-used by others.
Further detail
Possible implementations might include:

example ‘fixtures’ that provide the expected HTML output for a given set of inputs, optionally with a test suite that makes assertions using those fixtures and some sort of test harness implemented by the maintainer – a little like snapshot testing
a test suite that makes assertions about the HTML for a given input, for example asserting that elements have specific attributes or relationships

Questions:

How would fixtures relate to the component YAML files?
Do these supplement or replace any of our existing tests? (Related: If using fixtures, do we generate them as part of the release process or as we make changes?)
Do we release tests as a separate package or as part of the existing GOV.UK Frontend package?
How thoroughly do we test components that are included by other components?
What about less widely supported features, like {% call %}?
How do we support ports that have deviated from the API (e.g. if they chose to implement sanitisation differently)
How do we ensure coverage?
What about scenarios where class names are changed (e.g. using local identifiers in React)

This would be really useful for a Jinja port I've recently started: https://github.com/LandRegistry/govuk-frontend-jinja
The code in the repo above is generated by an automated script that creates Jinja equivalent macros for each of the Nunjucks ones. This is currently run within each frontend app itself at build time. I'd rather this was an installable package from PyPi to help manage version upgrades, remove some additional complexity from the app, and have a robust set of tests to prove compatibility. Ideally, I want to render the HTML output using a single set of inputs, for both the original Nunjucks and Jinja ported macros, then diff and fail tests if they're not identical.
As you can see, not got very far with this yet so feedback/contributions very welcome!

In govuk-react-jsx I render the nunjucks templates using the yaml example data in each component, and then do the same for the react components. I then use https://github.com/Prettyhtml/prettyhtml to smooth over minor rendering differences and https://github.com/markedjs/html-differ to diff the results. It's not the prettiest code but you can see it here:
https://github.com/surevine/govuk-react-jsx/blob/master/tests/utils/govuk-frontend-diff.js
which is then called from each component's tests:
https://github.com/surevine/govuk-react-jsx/blob/master/src/govuk/components/accordion/accordion.test.js
I then get test output like this (in this case, I have removed the role="banner" from the header to demonstrate)

This works well for the most part. The only issue I have encountered is cases where the example yaml data doesn't fully exercise the component. There's also a few superficial differences in the output but I managed to configure the html differ to ignore these.
Having a set of pre-rendered fixtures would certainly simplify my test suite and I could concentrate on just the diffing. Or indeed a test suite where I pass in my html and everything is handled for me.
I might have a go at abstracting out the work I did on govuk-react-jsx and see if it can help @mash out above - as a proof of concept almost.

@matthew-shaw have you seen https://github.com/alphagov/govuk-frontend-jinja? (maintained by a separate team at GDS, not the Design System team)
@andymantell I'm not sure you tagged the right person there!
We think the first step will be to create a proposal that we can share with the community – once we've done that we'll make sure you get the chance to feed back. These comments will be really helpful in shaping that proposal, so thank you both!

@36degrees The govuk-frontend-jinja https://github.com/alphagov/govuk-frontend-jinja repository uses a port of a script I wrote back at HM Land Registry to automatically port the Nunjucks across to Jinja using some regular expressions and other tomfoolery. I believe they've taken it a bit further than I had originally and fixed some issues I hadn't managed to. But it's similar conceptually.
Before I left HMLR though I was growing increasingly frustrated with the automated porting script and had basically concluded that it wasn't a great way to do things - there are so many subtle differences between Nunjucks and Jinja that the script became harder to maintain than just manually porting between the two! @matthew-shaw and I have chatted about this at length and he is working on the successor to this which is a set of manual ports supported by a test suite exactly as you described in this ticket. (i.e. the same model I use for govuk-react-jsx)
I think if I have a crack at packaging up my test suite it might be an interesting exercise to see how easily it can be applied to @matthew-shaw's work
Not proposing you use my code particularly, but it might shed some light on any potential issues with such a solution.

We (Department of Finance Northern Ireland) have a c# port in development of the nunjucks macros.
A way to test that the output is largely the same as the nunjucks macros would be very useful.
Definitely going to look into govuk-frontend-template-spec.

@DanScottNI is your C# port open source?

As @andymantell says, the govuk-frontend-jinja package is now published on PyPi. I've also deployed a little demo Flask app to Heroku with a full component specimen of every example from the GOV.UK Design System. This is now also being used internally within HM Land Registry reference/template implementations we call "skeleton apps" and I'm working on getting our service teams to adopt it.
This has already been a very worthwhile exercise. Using Andy's test suite, we now have 190 tests for every single component in every single permutation from the design system. In doing that, we've found some bugs with our original port that have now been fixed, so we have a provably higher quality output than before. This means we can be certain of 100% markup parity and compliance with the latest, and every subsequent, GOV.UK Frontend release. This has also enabled me to remove a bunch of complexity from our skeleton frontend application, and make that into an open source component that the cross government community can make use of too.
The published package is currently at release 0.2.1 and I intend to leave it in a beta stage until at least the next release of GOV.UK Frontend. Once I can prove that doing an update to the latest GOV.UK release is quick and accurate I'll bump it to version 1.0.0 and mark it as production ready. In all likelihood very little will change, and having that comprehensive test suite will ensure its quality.

Hi all, I made govuk-frontend-template-spec, just to let you know I've subbed to this thread and happy to help where I can. I'm not working on govuk projects at the moment, but If anyone wants to collaborate on it to make it a decent stop gap solution until there's an official solution then please ping me.

@DanScottNI is your C# port open source?

Not yet. Still very early stages of development, and we're still in the early stages of open sourcing things.

@DanScottNI is your C# port open source?

Not yet. Still very early stages of development, and we're still in the early stages of open sourcing things.

I have an ASP.NET Core Tag Helper port of most of these components at https://github.com/gunndabad/govuk-frontend-aspnetcore that may be of interest. Still early days so (very) light on docs.

We’re going to start including fixture files in the published GOV.UK Frontend package.
If you maintain your own macros, templates or partials you can use the fixtures to check that your components output the same HTML that GOV.UK Frontend uses, for example:

when you first create the macro, template or partial for a component
whenever you update to a newer version of GOV.UK Frontend

We’d like your feedback on this proposed change. You can:

💬  add comments to this issue
✉  send feedback to<EMAIL_ADDRESS>👍 +1 this comment if you agree with the proposed changes and have no other feedback to give

Please provide feedback by 5pm on Tuesday 4 August.
Problem
Whilst you can easily re-use the stylesheets and JavaScript from GOV.UK Frontend, it’s likely that you have to maintain the HTML markup used for each component yourself.
At the minute there’s no automated way for you to tell whether your HTML is the same as the HTML that GOV.UK Frontend uses.
You may already be using the examples in the component YAML files as test fixtures, but:

you have to manually fetch the YAML files from the src directory in GitHub, because they’re not included in the published package
the examples in the YAML file are not comprehensive, and do not cover all of the ways the component can be used
you have to include Nunjucks somewhere in your development or test environment to generate the expected HTML using the GOV.UK Frontend macros

Proposal
We’ll make the examples provided for each component more comprehensive, so that every component option is represented by at least one example.
We’ll then publish the examples in a fixtures.json file for each component, which will contain an array of objects representing each example with the following properties:

name
options – the options passed to the component
html – the HTML as rendered by the component’s Nunjucks macro

For example, in the published package you’d find govuk/components/button/fixtures.json with examples that look like this:
[
  {
    "name": "default",
    "options": {
      "text": "Save and continue"
    },
    "html": "<button class=\"govuk-button\" data-module=\"govuk-button\">\n  Save and continue\n</button>"
  },
  // […]
]

You can use these fixture files in your own tests to check that your HTML is the same as the HTML that GOV.UK Frontend uses.
For example, in pseudo-code:
fixtures = load_json_file(“govuk/components/button/fixtures.json”)

for fixture in fixtures
	expected_ouput = fixture.html
	actual_output = render_my_button(fixture.options)
	expect_equal(actual_output, expected_output)
end

You may need to normalise the expected and actual HTML to allow for differences in whitespace or indentation.
You can try a pre-release of GOV.UK Frontend that includes a fixtures.json file for the button component only by running:
npm install alphagov/govuk-frontend#634eeca76ee6f4399cba54cc13a11bd38f5c450a

We're going to start including fixture files in the published GOV.UK Frontend package.
If you maintain your own macros, templates or partials you can use the fixtures to check that your components output the same HTML that GOV.UK Frontend uses, for example:

when you first create the macro, template or partial for a component
whenever you update to a newer version of GOV.UK Frontend

We'd like your feedback on this proposed change. You can:

💬  add comments to this issue
✉  send feedback to<EMAIL_ADDRESS>👍 +1 this comment if you agree with the proposed changes and have no other feedback to give

Please provide feedback by 5pm on Tuesday 4 August.
Problem
Whilst you can easily re-use the stylesheets and JavaScript from GOV.UK Frontend, it's likely that you have to maintain the HTML markup used for each component yourself.
At the minute there's no automated way for you to tell whether your HTML is the same as the HTML that GOV.UK Frontend uses.
You may already be using the examples in the component YAML files as test fixtures, but:

you have to manually fetch the YAML files from the src directory in GitHub, because they're not included in the published package
the examples in the YAML file are not comprehensive, and do not cover all of the ways the component can be used
you have to include Nunjucks somewhere in your development or test environment to generate the expected HTML using the GOV.UK Frontend macros

Proposal
We'll make the examples provided for each component more comprehensive, so that every component option is represented by at least one example.
We'll then publish the examples in a fixtures.json file for each component, which will contain an array of objects representing each example with the following properties:

name
options – the options passed to the component
html – the HTML as rendered by the component's Nunjucks macro

For example, in the published package you'd find govuk/components/button/fixtures.json with examples that look like this:
[
  {
    "name": "default",
    "options": {
      "text": "Save and continue"
    },
    "html": "<button class=\"govuk-button\" data-module=\"govuk-button\">\n  Save and continue\n</button>"
  },
  // […]
]

You can use these fixture files in your own tests to check that your HTML is the same as the HTML that GOV.UK Frontend uses.
For example, in pseudo-code:
fixtures = load_json_file("govuk/components/button/fixtures.json")
for fixture in fixtures
	expected_ouput = fixture.html
	actual_output = render_my_button(fixture.options)
	expect_equal(actual_output, expected_output)
end

You may need to normalise the expected and actual HTML to allow for differences in whitespace or indentation.
You can try a pre-release of GOV.UK Frontend that includes a fixtures.json file for the button component only by running:
npm install alphagov/govuk-frontend#634eeca76ee6f4399cba54cc13a11bd38f5c450a

@vanitabarrett Are the current YAML files going to remain, or do the new fixtures.json files replace them?
Reason I ask is that I use the examples in the YAML files to build the storybook demos on govuk-react-jsx. If they're disappearing I'll do something else instead... (I.e. I'm not saying you have to keep them, just thinking ahead).
As for the fixtures themselves, it sounds perfect.

@36degrees @vanitabarrett having a read through, that sounds spot on. I'll try do a React PoC at some point.
Some things to think about:

how should we (govuk-frontend authors and users) deal with html options? For some frameworks, such as React, you wouldn't pass html as a  property (you would use the children prop containing React components rather than raw html), so it may be hard to adhere to a fixture that includes an html option. Perhaps frameworks would skip these fixtures? but if that's the case it would be good to ensure that fixtures with a text option were always available as an alternative when demonstrating a feature unrelated to the html option.
please can you confirm that the fixtures.json will be part of the npm release? I assume this is the case, but I know the last time I looked, the tests are on github but not in the npm package
would it be possible to add a fixtures index somewhere? This would mean that if you have a component library that intends to have 100% coverage, and a new component is added, then this could get picked up by tests.

e.g.
fixtures.json
{
  "button": "./govuk/components/button/fixtures.json",
  "etc.": "...etc"
}

Thank you for the comments @penx @andymantell
Some parts of the proposal were perhaps not as clear as they could be:

fixtures.json files will be included in the npm release. All fixture files will be stored at a common path: /package/govuk/components/[component-name]/fixtures.json. We expect maintainers to iterate through these to ensure 100% coverage
YAML files in /src will remain, but we recommend replacing your use of them (where possible) with the new fixtures. We treat /src as internal to the team, so files like the component yaml could be changed in future and this wouldn’t normally be treated as a breaking change.

@penx Our aim is to make these fixtures as generic as possible. In the same way that maintainers may need to normalise the HTML to allow for differences, we expect any differences in framework behaviour to be handled by maintainers. If there’s anything we can do to make the process smoother, let us know!
The examples in GOVUK Frontend should only pass html where we specifically want to check the HTML works correctly. If you want to skip these, you could skip any tests that pass html to a component.

The examples in GOVUK Frontend should only pass html where we specifically want to check the HTML works correctly

👌 if this is consistently followed then this works for me, thanks!

I like this idea, and will experiment with the button example in our port.
A few questions:

Can this fixture be included for the components which aren't listed in the Design System website, but which are used by other components (eg govukFieldset and govukLabel)?
Any thoughts on how we could specify and test the equivalent of the caller option, which some of the components accept (and which govukFieldset kinda expects).
Can the JSON files be published at a standard location (tagged with the version), so that ports don’t have to use npm to require them? (As lots of languages have their own package manager, and using two package managers alongside each other can be a pain). I can see some ports wanting to commit the fixtures into their repo, and just use a script to update it from a known location when supporting a higher govuk-frontend version.
How would this file be maintained? I'm presuming that the file would be auto-generated from some other source, such as the existing YAML files?

Minor suggestion:

Make the top-level of the JSON file be an object, rather than an array, and move all the examples under an "example" key. Mainly to allow additional keys / data to be added in the future without breaking backwards-compatibility.

Thanks for your comment @frankieroberto !
We will generate fixtures for all components in GOVUK Frontend, so the fieldset and label will have fixtures. The fixtures are generated from the YAML files, so anything with a YAML file will get a corresponding fixtures file.
Any components that support caller should also have another option, for example: passing HTML instead. We would suggest taking that approach instead.
If you want to avoid npm, the fixtures will be on Github in a standard place for each component at: package/govuk/components/[component-name]/fixtures.json. I believe the package directory also contains a package.json which has the current version. Is that the kind of thing you were imagining?
Can you give a little more detail or an example of the backward-compatibility you mentioned?

@vanitabarrett thanks for your response.

We will generate fixtures for all components in GOVUK Frontend, so the fieldset and label will have fixtures. The fixtures are generated from the YAML files, so anything with a YAML file will get a corresponding fixtures file.

Great. Would these be under the existing examples key in the YAML, or are you proposing a separate key for the test fixtures? (This doesn’t affect people using the JSON file, but it'd be good to understand what the expectations are for contributors of new components)

Any components that support caller should also have another option, for example: passing HTML instead. We would suggest taking that approach instead.

That makes sense, but the govukFieldset component currently doesn't have an html option. The caller is technically optional, but without it you can only have a fieldset with a label, not any content.
You could either add an html option to the component, or another idea would be to have a special argument name for the caller, eg __caller__.

If you want to avoid npm, the fixtures will be on Github in a standard place for each component at: package/govuk/components/[component-name]/fixtures.json. I believe the package directory also contains a package.json which has the current version. Is that the kind of thing you were imagining?

Sounds good.

Can you give a little more detail or an example of the backward-compatibility you mentioned?

If the JSON file looked like this:
{
  "examples": [
    {
      "name": "default",
      "options": {
        "text": "Save and continue"
      },
      "html": "<button class=\"govuk-button\" data-module=\"govuk-button\">\n  Save and continue\n</button>"
    }
  ]
}

rather than this:
[
  {
    "name": "default",
    "options": {
      "text": "Save and continue"
    },
    "html": "<button class=\"govuk-button\" data-module=\"govuk-button\">\n  Save and continue\n</button>"
  }
]

then that allows you to add additional top-level keys to the JSON file, without breaking any existing code which is simply iterating through the examples array:
{
  "name": "Button",
  "url": "https://design-system.service.gov.uk/components/button/",
  "version": "3.8.0",
  "deprecated": false,
  "examples": [
    {
      "name": "default",
      "options": {
        "text": "Save and continue"
      },
      "html": "<button class=\"govuk-button\" data-module=\"govuk-button\">\n  Save and continue\n</button>"
    }
  ]
}

(It's hard to predict whether this’ll be needed or not, but may as well allow it as a possibility. It’s also recommended by the GDS API guidance, I just discovered)

Thanks for clarifying @frankieroberto, I think your suggestions sound sensible! :+1:

Great. Would these be under the existing examples key in the YAML, or are you proposing a separate key for the test fixtures? (This doesn’t affect people using the JSON file, but it'd be good to understand what the expectations are for contributors of new components)

Yes, the fixtures files will be generated from the examples in the YAML files. We’ll need to do a bit of work to ensure the current examples are comprehensive enough before generating the first set of fixtures. I think we’ll probably review the existing documentation for contributors to see if anything needs tweaking there too.

This has been included in v3.9.0 of GOVUK Frontend 🎉


If telemetry server is down or slow, envoy crashes

Discussed on slack: to repro create a high load (2-3000 qps) on ingress, and kill the mixer pods.
The ingress shows a number of error messages that report can't be reached, and crashes (with
'signal: killed' message).
It is likely the mixer plugin doesn't have the right deadline or can't shed load. Telemetry should fail open.

For 1.0.2

It is likely due to OOM. What do we expect for this other than not failing, should mixer write pending reports temporarily to disk, or just drop the pending reports?

Dropping is fine. This should only happen if no other upstream mixers are available.
I don’t think retry policy is set or used on this code path.

Dropping but recording how much got dropped. Endlessly accumulating failed RPCs seems like an anti-pattern.
/ref https://github.com/istio/istio/issues/5095

I'll try this one

/remove stale

Moving to P1 pending testing feedback

@duderino I'm going to close this, as we have adopted loadshedding by default for the mixer services. Moving forward, we are pursuing a future architecture without Mixer, so this issue is not likely to receive any further direct attention.

@douglas-reid sorry I forgot to circle back on this one. bug fixes that went into 1.1 fixed at least one of the crashes and considering there was no additional commentary on this bug that may have been the primary source.


++Enable sending image in GCM notifications

Existing behavior sends it under the data field which doesn't behave correctly with the native image notifications.
Fixes https://github.com/jazzband/django-push-notifications/issues/559

To deliver an image to the recipient in FCM's notifications payload, you must use the "image" key.
I hope there is a quick release.
https://firebase.google.com/docs/cloud-messaging/android/send-image

Thanks, it is so easy. Waiting for the relase!

Can anyone confirm that this is working? I have tried passing image=url... and as extra={"image": url} but I cant seem to get either to work.

Yes @astrike30 I can confirm it works with extra={"image": url} where url is well-formed & publicly accessible. Sometimes, the FCM network fails showing the image for some random reasons, so make sure to test on multiple devices if one doesn't seem to work.


Requirement on containerPort to capture traffic is too onerous

We've seen many customers report issues on 0.8 around the new requirement to use containerPort declarations to capture traffic. This requirements was introduced as a workaround for the security issues with the proxy config in 0.7 that allowed a proxy to be used as a trampoline.
This is yielding a sub-optimal user experience and I believe we can have a more complete solution for this problem without requiring user config intervention.
A variety of solutions have already been discussed
@PiotrSikora @rshriram @costinm @andraxylia
I believe this is blocking for 1.0

We are hearing about this problem often from customers, and even from within the team.
Even though this behaviour was introduced to fix a security problem, it has introduced another one.
Users can inadvertently (or maliciously) opt out of the mesh by not using the correct containerPort setting. If there is a standard RBAC policy that applies to all services, simply by forgetting to set containerPort that service is left open.
This is certainly blocking for 1.0.

As mentioned elsewhere - Pilot simply needs to stop generating wildcard listeners, and then we can go back to intercepting all the traffic.

For which we need the filter chain match on destination ip ranges.

Otherwise generating 100s of listeners one per service and port combo is not feasible given that we will have 100s of RDS resource requests - one per http connection manager. Any change in mixer filter or other filter will cause mass reload of all these listeners.

I think technicalities in configuration server should not prevent us from getting the model right. xDS server can hold responses as needed, to prevent mass reloads, or to wait until all RDS resources are requested.  In fact, I claim it's actually better to split RDS into smaller resources. Currently, in the worse case of all services listening on port 80, we have one giant RDS object that forces mass reload. By splitting into many RDS objects for each service VIP, the management server can be smarter and stage the update.

You still can’t get rid of wildcard listeners when you add service entries or have stateful sets.

Is it ok to trampoline from a random pod's sidecar to an external service? I think it's not as bad, since an external service does not use the source's credentials. It is similar to sharing an egress between pods.
Stateful sets are a separate problem, and using wildcard listeners for them is a work around, not a final solution.

Can someone clearly describe what the original issue was with an example and how that cryptic pr fixed the issue ?

Removing wildcard listeners will also break various cases in multicluster/meshexp - we can't know the full list of all IPs a service can use across the mesh.
And is likely to impact performance and scalability - since the number of matches in envoy will drastically increase.

And yes, statefull sets would also break if we remove the wildcard ports, as well as any non-k8s env where service VIPs are not used.

Mandar: I hope you are not proposing removing the option for user to exclude ports from interception - or disabling the interception completely via annotations. Mandatory policy and firewalls are far more complex then capturing and breaking all ports - it requires closing all other ways a user could break out.

We should keep exclusion options, as they are needed many times.
However it should be an explicit opt out option.

@rshriram the problem is when traffic is sent to a pod with a spoofed host. The in and out listeners are mixed together, there is no separation. The incoming traffic is matched against a "<IP_ADDRESS>:port" listener with the spoofed host, and sent to the outbound route to the microservice for the spoofed host.
The fix in the inboundPorts does not let that traffic enter Envoy, so it is not trampolined. There is a small price to pay, which is to declare the ports.
The alternative to not having wildcard listeners is to split the in and out listeners. For this we need support in Envoy.

If a user types curl -H serviceB serviceA and the request reaches service B instead of service A, then that's a UX problem. The expectation is that service A VIP receives traffic no matter what host is specified. Therefore, we should respect that, and utilize VIP whenever it is available.

Right, with the current inboundPort exclusion the traffic will reach serviceA, which will see it is spoofed and it contains hostB instead of hostA. We do not want to have serviceA exposed as external used as a trampoline to reach serviceB which is internal only.

don't understand.. we setup inbound listeners to listen on the pod IP only. And the http connection manager in that listener can only route to local app and no where else, because we don't have any other virtual hosts in the inbound path.
so, even if I send a request to a pod IP with a spoofed host, it would end up in the local app and not go out.
if I send a request to a service VIP with a spoofed host from a pod with a sidecar (outbound), then it goes to the spoofed host as per HTTP specs.
On the inbound path, the you always see the pod IP. So, the fix for requiring a container port doesn't make sense nor do the alternatives being discussed here. Am I missing something?

@andraxylia It's not spoofing. Imagine "serviceB" was just the external name for serviceA (e.g. coming through ingress).

I'd like to understand this better as well. I agree with @rshriram that inbound listeners are setup to listen on the pod IP only. With regard to containerIP, the following is extracted from k8s API specs https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.10/:
ports
ContainerPort array
patch type: merge
patch merge key: containerPort 	List of ports to expose from the container. Exposing a port here gives the system additional information about the network connections a container uses, but is primarily informational. Not specifying a port here DOES NOT prevent that port from being exposed. Any port which is listening on the default "<IP_ADDRESS>" address inside a container will be accessible from the network. Cannot be updated.
So trying to understand what's expected from setting containerPort in the deployment.

@baodongli the traffic is sent to that PodIP, but the original dest IP of the traffic is re-written as the traffic is trampolined.
Correct, k8s spec does not mandate the containerPorts to be listed, Istio requires it, so it can add the ports to the list of inboundPorts.

Is this realistic for 1.0 ?
Is there a clear design - and understanding of the tradeoffs and what will break ?

@rshriram  Scenario is:
Service A - port 8000
Service B - HTTP, port 9000
Attacker connects to a pod on Service A, on port 9000, and makes a request with "Host: b". It hits the *:9000 listener, which forwards to service B (with A's credentials).
This is only a problem for outbound path - inbound is fine ( connecting to A on port 8000 correctly goes to the inbound listener ).

What setting container port does is changes iptables capture, so only the ports that are used by the app are captured by the inbound table.
That means an app connecting to Service A pod on port 9000 will not be intercepted by iptables/envoy, so request will be rejected.

For tracking, if the change is made please reopen (and test):
https://github.com/istio/istio/issues/5865

Based on discussions with @rshriram and @costinm, this issue is 1) not 1.0 blocking 2) is non-trivial, risky and time-consuming (not something that can be done in 1.0 timeline). So we are making this a P0 for 1.1 milestone instead.

@jasmine-jaksic this should be decided/vetted by the TOC, since the TOC set the initial P0 priority.

@jasmine-jaksic I was under the impression that the agreed-upon solution for this is separating inbound/outbound listeners and reverting iptables back to intercepting all the traffic by default.
On the Envoy side, the code is the same as for #2468 (another P0), one the Istio side, it will require small changes to the iptables script (to use 2 ports instead of 1) and then Pilot changes to match it.

@PiotrSikora and @andraxylia This seems very risky but let's discuss the details in tomorrow's 1.0 stand-up.

I looked a bit into this - TPROXY does not use the redirect port, so the previously agreed solution doesn't work for that. I'm trying to fix TPROXY for 1.0 - it appears REDIRECT will not work for UDP (which we want to support), and most docs I found indicate TPROXY scales better and has less bugs than redirect.
I think we need to re-evaluate the implementation, given the goal of supporting firewall rules we will need
to have more dynamic configuration of the iptables, so instead of REDIRECT on different ports, I suggest

Init container will have a small app that will query pilot and get the list of inbound ports to intercept and list of inbound ports to block
Sidecar container to have NET_ADMIN capabilities, and contact pilot to dynamically update iptables
Long term define an API for dynamic port support

This moves us closer to real firewall support, works for TPROXY as well as REDIRECT - and avoids the need for user to define all containerPort (which we should still recommend).

Any update on this?

Once on-demand xDS is available, this will reduce the number of listeners in one Envoy to the ones actually needed, and we can change the outbound listeners for http to be VIP:port instead of wildcard <IP_ADDRESS>:port. This will solve the trampoline problem.
(We currently use wildcard for http just to reduce the number of listeners. Tcp and https already use VIP:port listeners)

@silentdai could you describe your current progress and the remaining work?

[x] iptables redirect inbound traffic and outbound traffic to separate listeners
[x] inbound listener and outbound listener share no bind_to_port=false listener
[ ]  redefine readiness probe
By then containerPort is not required when user opt-in above

pending test results, of course.
On Mon, Aug 26, 2019 at 3:53 PM John Howard<EMAIL_ADDRESS>wrote:

@lambdai https://github.com/lambdai this is done now, right? Will be
shipped in 1.3?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/6259?email_source=notifications&email_token=AAAUR2S2K3K6XNZYSH64ME3QGRNGLA5CNFSM4FE344AKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5F5OFQ#issuecomment-525063958,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAUR2X5AAR5GHSW4EW7PTTQGRNGLANCNFSM4FE344AA
.

Shipped in 1.3!


Dropdownbox extension problems in IE11

Hello,
I've extended the Dropdownbox control, so i can use the tabindex properly.
In Chrome and Firefox it works just fine. In IE11, it fires the following 4 events on rendering:
focus event on  contains: true -
Event fired: 'selectstart'
Event fired: 'select'
Event fired: 'focusin'
As far as i can see, the problem is caused by the onAfterRendering function that is also being extended.
However, the events are being fired in IE11 only, even if there is nothing in the onAfterRendering function declared.
Here's an example: http://jsbin.com/wovimarewu/1/
Kind Regards,
Wolfram Ganz

Hi,
you are overwriting the onAfterRendering function.
Try to apply the prototype onAfterRendering like you did in the init
http://jsbin.com/vurewekoco/1/
BR,
Tobias


This package is not compatible with nodejs 6.x, unlike the engines in package.json claims

The package uses Object.entries which is not implemented in nodejs 6.x (tested with 6.11.1).

You might fix it by using require('object.entries').shim(); [https://www.npmjs.com/package/object.entries

We need to run CI tests against node6.

We published v0.10.1 that fixes issue: https://github.com/GoogleChrome/puppeteer/releases/tag/v0.10.1


Create troubleshooting doc on versioning_operation_template_changes.md

You can try these changes at https://cadlplayground.z22.web.core.windows.net/typespec-azure/prs/847/
Check the website changes at https://tspwebsitepr.z22.web.core.windows.net/typespec-azure/prs/847/


envoy.code.check(0.5.2): Adjust path of fuzz configuration

This is an update pending https://github.com/envoyproxy/envoy/pull/27610


Using 'Toggle command palette' action from palette doesn't close palette

Windows Terminal version (or Windows build number)
1.9.1445.0
Other Software
No response
Steps to reproduce

Open the command palette
Search for: Toggle command palette
Hit enter to use that action

Expected Behavior
I'd expect the palette to close when using the Toggle command palette item in the palette. This is also what happens when you use the toggle shortcut while the palette is open.
Actual Behavior
The entered search text is cleared, and the palette remains open. The code responsible seems to be:
https://github.com/microsoft/terminal/blob/b2c2a4c159b7e918ea8aabc9e22f83d58a479002/src/cascadia/TerminalApp/AppActionHandlers.cpp#L369-L380
When the command palette is open, and you use the shortcut to toggle its visibility, CommandPalette().Visibility() correctly returns Visibility::Visible and the palette is closed. On the other hand, when using the entry in the palette, CommandPalette().Visibility() returns Visibility::Collapsed, which causes the palette to remain open.

Huh. Good catch!

I'll see if I can track down what's causing this.


Removes all AddField functions from MessageEmbeds

This PR replaces the old deprecated addField function with the newer addFields function in all instances of the function being used in an embed. Also makes the role logging look a bit nicer IMO, can be rejected.
Closes issue #172

Why do you want to move away from the builder pattern?


Add New File option to File Menu Items

I looked for this on starting the app when trying create a new tutorial.

/move to satyrnjs/satyrn


Why this cookbook?

There is another cookbook on the supermarket to manage ulimits.
Why this one? What limitations does the other ulimit cookbook have?

I wasn't a fan of the attribute naming convention, I felt my way was cleaner, and it also allows group ulimit settings using the @group syntax.  The other cookbook also didn't support 'legacy' systems, and assumed that all systems supported ulimit configs in /etc/security/limits.d (not the case with Centos 6 and earlier, which only looks in /etc/security/limits.conf)

Great, thanks!


FlTechincs - Graphic design error

ENVIROMENT: production
COMPANY: FlTechincs
URL: FlTechincs scarpers.peviitor.ro
Browser: Google Chrome
Device: laptop
OS: MacOS
Precondition:
The site must be open in Google Crome browser.
Steps:

Click on Firme from footer.
Enter Wiley in the search bar.
Click the button under the company's logo.
Minimize the page.
Maximize the page.

Expected result:
The graphic on the cover of the page returns to its original form
Actual result:
The graphic on the cover of the page does not returns to its original form
https://github.com/peviitor-ro/Scrapers-UI/assets/137428958/c49253c8-a587-4cb5-afce-d35964ca6771

fixed on pull request 434.
Re-test after pull request is merged.


Add function filter example

Add two simple examples for filter with function (odata v2)

I didn't manage to understand all the nuances of entityset wrapper or client direct, and I didn't yet manage to test more on my own.
I will keep looking into it to :+1:

Hope it will be enough for now :smiley:


Turn on IP Filtering

Next step is to see it fail, then try server.use-forward-headers=true per https://www.geekyhacker.com/2019/12/10/spring-boot-security-restrict-requests-to-ip-address-range/

@mpidcock FYI I added some tests to validate the IP filtering and X-Forwarded-For handling

applies to story #185022878


White screen on startup

I've installed draft on my reMarkable running version <IP_ADDRESS> and whenever I turn on the device, all I see is a white screen. I'm still able to press the buttons in the launcher, I can even see a faint outline of them because the display haven't done a full update, and everything works fine if I use button-capture-reMarkable to exit xochitl, but on the initial boot, the launcher is completely blank.
Let me know if I can provide anything to help solving this problem. Thanks!

Apologies, I didn't even see this issue pop up. I believe this is closed by #14 but please let me know if there are any further issues!


CET prevents the game from registering/connecting with GOG galaxy & GOG.

Version 1.24.1 with a completely fresh game install (nuked everything, including prior saves, then reinstalled).
If CET is enabled and loaded, the "My Rewards" tab on the main menu says I am not connected.   In addition this means achievement tracking is disabled which is more of a problem.
If I prefix "dis_" to version.dll, thereby disabling it from being loaded, the game connects to GOG.
Further if the dll hook/mod loader is named correctly but cyber_engine_tweaks .asi has _dis appended to the name, disabling it, the game is able to connect to GOG.
Is this intentional or some sort of unexpected side effect of CET hooking and overloading parts of the game's internal API?   I looked through the log files for any kind of exception, error, or even a warning but didn't find anything.

About to close this as i saw issue #663 and testing now.


No Cancellation Token on Azure Function Method

In the previous framework, I could pass a cancellation token to an Azure Function Method:
        [FunctionName("my-function")]
        public async Task<IActionResult> MyFunction(
        [HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = null)] HttpRequest req,
        CancellationToken cancellationToken)
        {
             ...
        }

if I try and do that now, an error occurs when calling the method with the cancellation token:
        [FunctionName("my-function")]
        public async Task<HttpResponseData> MyFunction(
            [HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = null)] HttpRequestData req,
            FunctionExecutionContext executionContext),
            CancellationToken cancellationToken)
        {
             ...
        }

This is the error that occurs when calling the method:
[2021-02-05T01:44:09.069Z] System.Private.CoreLib: Exception while executing function: Functions.migrate. System.Private.CoreLib: Result: Failure
Exception: Sequence contains no elements
Stack:    at System.Linq.ThrowHelper.ThrowNoElementsException()
[2021-02-05T01:44:09.070Z]    at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionExecutionContext context)
[2021-02-05T01:44:09.071Z]    at Microsoft.Azure.Functions.Worker.FunctionBroker.InvokeAsync(InvocationRequest invocationRequest) in D:\a\1\s\src\DotNetWorker\Broker\FunctionBroker.cs:line 44.

Will support be added back in for the tokens?

Is there any update on this?
I'd like to understand the reasoning behind removing the cancellation token, especially in the case of HttpTriggerFunctions where using a CancellationToken makes A LOT of sense.
This is a bit of a deal breaker personally for me as I feel like this is a step backwards and is preventing me from upgrading to net core 5.

@tidusjar, you can try this suggestion https://github.com/Azure/azure-functions-host/issues/6911#issuecomment-842528631 by injecting the IHostApplicationLifetime into the function and watching the  ApplicationStopping cancellation token.

@tidusjar, you can try this suggestion https://github.com/Azure/azure-functions-host/issues/6911#issuecomment-842528631 by injecting the IHostApplicationLifetime into the function and watching the  ApplicationStopping cancellation token.

Thanks for the suggestion, but this does not resolve my issue. My issue is regarding when a Http request is cancelled, there Is currently no way to handle this
The cancellation token you mentioned is regarding the application lifecycle. But I appreciate the pointer.

@tidusjar, I always understood the cancellation token being passed in the function method was the operating system notifying the function is about to terminate.
I think what you're looking for is HttpContext.RequestAborted based on this thread.

This is why the SDK should provide cancellation token rather than developers come up with their own solutions and recreated the same thing time and time again. From all the things that are being improved with the isolated worker, this feels like a step in the wrong direction. @fabiocav any plans to address this?

To be surfaced to the .NET isolated worker this depends on a host update related to this feature https://github.com/Azure/azure-functions-host/issues/7785, and requires further design decisions and prioritization, so for now the documentation here is up to date. Note that even though the cancellation token isn't flowing into the out of proc function, a graceful period should still be given to your function as per this work in Consumption and Premium plans.

@tidusjar, I always understood the cancellation token being passed in the function method was the operating system notifying function it is about to terminate.
I think what you're looking for is HttpContext.RequestAborted based on this thread .

The stackoverflow page doesn't help here because there is http context to which we could link the token.

I'm getting a NullReferenceException in the host's code when I add a cancellation token to a timer trigger:
[2022-05-07T22:58:51.934Z] Executed 'Functions.MyFunction' (Failed, Id=6b958381-1d1f-43f5-b701-abb6daaab3c4, Duration=6409ms)
[2022-05-07T22:58:51.935Z] System.Private.CoreLib: Exception while executing function: Functions.MyFunction. System.Private.CoreLib: Result: Failure
Exception: System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.)
[2022-05-07T22:58:51.936Z]  ---> System.NullReferenceException: Object reference not set to an instance of an object.
[2022-05-07T22:58:51.936Z]    at lambda_method1(Closure , MyFunction , Object[] )
[2022-05-07T22:58:51.937Z]    at Microsoft.Azure.Functions.Worker.Invocation.VoidTaskMethodInvoker`2.InvokeAsync(TReflected instance, Object[] arguments) in D:\a\1\s\src\DotNetWorker.Core\Invocation\VoidTask
MethodInvoker.cs:line 22
[2022-05-07T22:58:51.937Z]    --- End of inner exception stack trace ---
[2022-05-07T22:58:51.938Z]    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
[2022-05-07T22:58:51.938Z]    at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
[2022-05-07T22:58:51.938Z]    at System.Threading.Tasks.Task`1.get_Result()
[2022-05-07T22:58:51.939Z]    at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.<InvokeAsync>b__6_0(Task`1 t) in D:\a\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:
line 32
[2022-05-07T22:58:51.939Z]    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
[2022-05-07T22:58:51.940Z]    at System.Threading.Tasks.Task.<>c.<.cctor>b__272_0(Object obj)
[2022-05-07T22:58:51.941Z]    at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
[2022-05-07T22:58:51.942Z] --- End of stack trace from previous location ---
[2022-05-07T22:58:51.942Z]    at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
[2022-05-07T22:58:51.943Z]    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
[2022-05-07T22:58:51.944Z] --- End of stack trace from previous location ---
[2022-05-07T22:58:51.945Z]    at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor
.cs:line 44
[2022-05-07T22:58:51.946Z]    at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\1\s\src\DotNetWorker.Core\Out
putBindings\OutputBindingsMiddleware.cs:line 13
[2022-05-07T22:58:51.946Z]    at Microsoft.Azure.Functions.Worker.GrpcWorker.InvocationRequestHandlerAsync(InvocationRequest request, IFunctionsApplication application, IInvocationFeaturesFactory invocationF
eaturesFactory, ObjectSerializer serializer, IOutputBindingsInfoProvider outputBindingsInfoProvider) in D:\a\1\s\src\DotNetWorker.Grpc\GrpcWorker.cs:line 167
Stack:    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
[2022-05-07T22:58:51.947Z]    at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
[2022-05-07T22:58:51.948Z]    at System.Threading.Tasks.Task`1.get_Result()
[2022-05-07T22:58:51.948Z]    at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.<InvokeAsync>b__6_0(Task`1 t) in D:\a\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:
line 32
[2022-05-07T22:58:51.949Z]    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
[2022-05-07T22:58:51.949Z]    at System.Threading.Tasks.Task.<>c.<.cctor>b__272_0(Object obj)
[2022-05-07T22:58:51.950Z]    at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
[2022-05-07T22:58:51.951Z] --- End of stack trace from previous location ---
[2022-05-07T22:58:51.951Z]    at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
[2022-05-07T22:58:51.952Z]    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
[2022-05-07T22:58:51.953Z] --- End of stack trace from previous location ---
[2022-05-07T22:58:51.953Z]    at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor
.cs:line 44
[2022-05-07T22:58:51.954Z]    at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\1\s\src\DotNetWorker.Core\Out
putBindings\OutputBindingsMiddleware.cs:line 13
[2022-05-07T22:58:51.954Z]    at Microsoft.Azure.Functions.Worker.GrpcWorker.InvocationRequestHandlerAsync(InvocationRequest request, IFunctionsApplication application, IInvocationFeaturesFactory invocationF
eaturesFactory, ObjectSerializer serializer, IOutputBindingsInfoProvider outputBindingsInfoProvider) in D:\a\1\s\src\DotNetWorker.Grpc\GrpcWorker.cs:line 167.

(admittedly, it might not make much sense to add a cancellation to a time-triggered function, but still, it shouldn't fail like this... at the very least, if it's not supported, it should indicate it clearly rather than throw a NRE)

I have tried the sample code in Release1.9.0 and I still don't see the feature is working. I invoke the azure function and cancel it but the cancellation token is not being catched.

@mjallaq88 Could you please open a new issue with relevant repro information? If you can share a minimal repro (a github repo would be great), we can take a look. Thanks!


[AIRFLOW-900] Double trigger should not kill original task instance

This update the tests of an earlier AIRFLOW-900.
Please accept this PR that addresses the following issues:

https://issues.apache.org/jira/browse/AIRFLOW-900

Testing Done:

Unittests updated

Will merge when unit tests pass.

Codecov Report

Merging #2146 into master will increase coverage by 0.6%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           master    #2146     +/-   ##
=========================================
+ Coverage   66.61%   67.21%   +0.6%
=========================================
  Files         142      142
  Lines       10783    10783
=========================================
+ Hits         7183     7248     +65
+ Misses       3600     3535     -65

Impacted Files
Coverage Δ

airflow/models.py
86.91% <0%> (+0.05%)
:white_check_mark:

airflow/www/views.py
67.32% <0%> (+0.14%)
:white_check_mark:

airflow/utils/dag_processing.py
86.58% <0%> (+0.81%)
:white_check_mark:

airflow/hooks/dbapi_hook.py
84.09% <0%> (+1.51%)
:white_check_mark:

airflow/ti_deps/deps/trigger_rule_dep.py
93.54% <0%> (+1.61%)
:white_check_mark:

airflow/jobs.py
73.35% <0%> (+1.85%)
:white_check_mark:

airflow/settings.py
92.95% <0%> (+2.81%)
:white_check_mark:

airflow/utils/helpers.py
43.56% <0%> (+2.97%)
:white_check_mark:

airflow/executors/init.py
58.06% <0%> (+3.22%)
:white_check_mark:

airflow/executors/base_executor.py
94.64% <0%> (+3.57%)
:white_check_mark:

... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bb39078...9f36fce. Read the comment docs.


move PERL5LIB setting further up for all cmds

PERL5LIB not set early enough, so download_web_deps fails silently with missing dependencies

This closes #42 , right? Thank you!

I have another change that makes the failure visible..   coming shortly..


FlatButton remains highlighted when swiping out.

Observed Behavior: Tap and hold on a FlatButton, then slide off it. The Button's InkWell(?) remains highlighted, even if you let go of the tap after that.
Expected Behavior: In addition to unhighlighting onTapUp, the highlight should disappear if the pointer move takes it out of the FlatButton's borders.

Fixed.


Update design system to version 23.0.1

This is an auto-generated PR.
Design system release: https://github.com/ONSdigital/design-system/releases/tag/23.0.1

Closing as 23.0.1 breaks runner.

Should we delete closed PRs?


Use a link as a placeholder for the 'Link to project' form field

Woops, def doesn't make sense right now. Thanks for the fix! :)

@RobertSandu I just added you as a contributor to the README, let me know if you'd rather not be on there. Thanks your contribution :)


Add flip convenience method to AtomicBoolean.

I find ab.flip(true) reads better than ab.compareAndSet(expect = false, update = true).

Thanks @Wogan, but please add a test.

Hopefully these tests are in line with what you expect.


Feature Suggestions

I love this library, but here are a couple things I've run into that I think would make it better:

relative path interfaces should have a parameterless GetAbsolutePath() method that gets an absolute path relative to the working directory, which is a pretty common operation

FileInfo / DirectoryInfo properties can be cache the created instance on the first get call and return that for subsequent calls instead of making a new instance each time

Any interest in having these items added? I could assist with implementing them.
Unrelated aside: I would have much preferred if you could use this library without the string extensions, but rather with methods like: DirectoryPath.Create(string path), AbsoluteDirectoryPath.Create(string path) etc...or just using constructors. It feels like a bad code smell to pollute the string class with methods that only apply to paths and not to any other strings.

Please feel free to improve this lib. So far this lib fits perfectly our needs and it is bug free as far as we know (from millions of logged usages in prod) so it is stable and we are not investing more dev in it.
Concerning the string extensions, they apply only in source file where you have using NDepend.Path, so this limits a lot the problem you describe, and we like the ctor-less extension method extension (I guess it is a matter of habits).


Core: shared Clock ADR

What this PR changes/adds
Decision record for shared clock implementation. The implementation is already merged upstream.
Linked Issue(s)
#284
Checklist

[ ] added appropriate tests?
[ ] performed checkstyle check locally?
[ ] added/updated copyright headers?
[ ] documented public classes/methods?
[ ] added/updated relevant documentation?
[ ] added relevant details to the changelog? (skip with label no-changelog)
[ ] formatted title correctly? (take a look at the CONTRIBUTING and styleguide for details)

Codecov Report

Merging #290 (024e613) into feature/1364-clock-adr (63e4596) will not change coverage.
The diff coverage is n/a.

@@                   Coverage Diff                   @@
##           feature/1364-clock-adr     #290   +/-   ##
=======================================================
  Coverage                   67.62%   67.62%
=======================================================
  Files                         719      719
  Lines                       15957    15957
  Branches                     1044     1044
=======================================================
  Hits                        10791    10791
  Misses                       4692     4692
  Partials                      474      474

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 63e4596...024e613. Read the comment docs.


Release gql_http_link stable version

The latest release of gql_http_link is a couple of months old and not compatible with the current stable versions of gql_link and gql_error_link. It would be nice if a new stable release could be made.

Thank you, I wanted to publish with the latest batch of public releases but forgot to bump the version number.
Will do later today.

releases gql_htpp_link 0.4.4