Use job and _param labels cause double instance in report

v. 0.17.0
Using config like this
[{
	"targets": ["<IP_ADDRESS>"],
	"labels": {
		"job": "my_job",
		"__param_module": "my_module",
	}
}]

in the targets file "my_job.json'" cause creation of two instances with the same parameters in the same job in the Prometheus metrics report. Workaround: delete job label.

Can you share your prometheus.yml?

global:
   scrape_interval:     15s # By default, scrape targets every 15 seconds.
   evaluation_interval: 15s # By default, scrape targets every 15 seconds.

rule_files:
   - 'rules/*.rule'
   - 'rules/s3/*.rule'

scrape_configs:
...
  - job_name: 'dns_exporter_snmp'
    file_sd_configs:
    - names: ['/opt/prometheus/tgroups/snmp*_targets.json']
    relabel_configs:
      - source_labels: [__address__]
        regex: (.*?)(:80)?
        target_label: __param_address
        replacement: ${1}
      - source_labels: [__param_address]
        regex: (.*)
        target_label: instance
        replacement: ${1}
      - source_labels: []
        regex: .*
        target_label: __address__
        replacement: <IP_ADDRESS>:9116  # SNMP exporter.

UPDATE: despite I delete job label, Prometheus creates two jobs for the target:

job_name: dns_exporter_snmp
job_name: my_job

You're not doing anything odd, so that sounds like a bug that we'll need to verify and look in to.
I presume there's no other files matching that glob? That could explain it.
Does this also happen with 0.18.0?

uiuiui, you're right! there was another definition for this target hardcoded in the config. sorry.

No problem.


Input triggers twice

It appears that every date selection triggers the input event twice.  You can see this in the sandbox by adding console.log

Hi!
This is intended. Each time a date is selected, the combination of start and end date is sent out. What's expected?

Hi Praveen
Many thanks for responding and thanks for creating the date range picker.
Yes, I receive start and end date in the range field.  Not sure if it is intended, but my query is that each time I select either start or end date, is it correct that the input event fires twice ?
Regards
Mike Moncrieff
Computer Leverage
From: Praveen Puglia<EMAIL_ADDRESS>Sent: 17 September 2018 08:59 AM
To: praveenpuglia/vuetify-daterange-picker
Cc: outoppie; Author
Subject: Re: [praveenpuglia/vuetify-daterange-picker] Input triggers twice (#29)
Hi!
This is intended. Each time a date is selected, the combination of start and end date is sent out. What's expected?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/praveenpuglia/vuetify-daterange-picker/issues/29#issuecomment-421908504 , or mute the thread https://github.com/notifications/unsubscribe-auth/AMEblVq8SS0RK2ogjYnlR75CXSaOgf-Lks5ub0hBgaJpZM4Wq-4A .Image removed by sender.

Yes! that's how I designed it. I should probably also add an option to give Apply and Cancel buttons which will ensure the event gets fired only when Apply is clicked.

Thanks once again for the quick response.  I have adjusted my code accordingly.


undefined: runconfig.Config

Hi,
I tried to build docker-backup by:
go install github.com/discordianfish/docker-backup
and
docker build .
Both failed with error:
evgen@vps257621:~$ go install github.com/discordianfish/docker-backup
# github.com/discordianfish/docker-backup/backup
go/src/github.com/discordianfish/docker-backup/backup/backup.go:24: undefined: runconfig.Config
go/src/github.com/discordianfish/docker-backup/backup/backup.go:25: undefined: runconfig.HostConfig

Yes, this probably should be updated and the dependencies vendored. It's been a while.


Using pytorch data-loading with jraph.

Hi,
Thanks for the great library. I created this Toy example of how to combine PyTorch geometric data-loading and Jraph. Let me know if it is useful to make it a tutorial/example, and also if more descriptions are needed.
Link: https://drive.google.com/file/d/1_X2su92_nS52RNl4m-WYvmkvUSrFE4xQ/view?usp=sharing

Thank you, I really appreciate it - I will give it a read over.
If you're happy, I would really like to add this as an example or link to this from the central github repo.

Hi,
Sure I will refactor it.
Also, I like your idea of just keeping to the data-loading, and leaving the modeling part. So I will make it that way.
Thanks.

Hi @jg8610
Can you check now, if it is better? kind of removed all the rest of the code.

Thanks! I was OOO last week will have a look today.

Hi,
Done as per your suggestions. Do we have a similar notebook for tf.data.dataset options? if so can you share it with me?

The OGBG Molpca notebook uses tf data, but in order to use tfdata well you need to do a bit of work to add your dataset to the central repository.

But, this is great! I will add a link to this colab in the readme next week.

There seems to be a bug in this notebook (perhaps due to a newer version of pytorch-geometric). When creating the ToyDataset object in cell 5, the following error is raised: TypeError:  Can't instantiate abstract class ToyDataset with abstract methods get, len. I don't know exactly what the purpose of the methods len and get is in Dataset, because they don't replace __len__ and __getitem__. Adding the following two methods to ToyDataset seems to at least solve the issue:
def len(self):
    return len(self)

def get(self, idx):
    return self[idx]

Just adding empty (pass) methods len and get also works..


Start loading image before it comes into view?

Is it possible to have images start loading before they come into view?  Right now, it seems like it begins to load the image as soon as it is scrolled into the view.  Is it possible to start loading the image 100px or 200px (any custom value) ahead of time?  this would help minimize the amount of time the placeholder is seen while the real image is downloaded.  Is there some sort of padding setting to allow this?

You can do that by setting the rootMargin option of the intersection-options prop https://github.com/alexjoverm/v-lazy-image/blob/master/README.md#props

So, this would load it 500px before reaching the viewport:
<v-lazy-image :intersection-options="{ rootMargin: '500px' }"/>

Is there a way to globally apply this instead of every v-lazy-image component?


Disable MockTelemetryPlugin by default for integ tests

Description
With this plugin enabled I see significant increase in memory usage during ITs. This is due to a static map that collects Span entries per write/replication request and is not cleared until after the test execution. Tests with higher counts of independent writes are at risk of running oom or experiencing slow down / warnings related to this.
I've run two reported flaky tests on repeat with this disabled #12801 and #12836 and do not see failures.
Disabled the plugin by default until this is resolved given it is still under FeatureFlag.
Related Issues
related to https://github.com/opensearch-project/OpenSearch/issues/12615
Resolves https://github.com/opensearch-project/OpenSearch/issues/12801
Resolves https://github.com/opensearch-project/OpenSearch/issues/12836
Check List

[x] New functionality includes testing.

[x] All tests pass

[x] New functionality has been documented.

[x] New functionality has javadoc added

[x] Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
[x] Commits are signed per the DCO using --signoff
[x] Commit changes are listed out in CHANGELOG.md file (See: Changelog)
[x] Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@Gaganjuneja fyi

Can we hold on this PR. I will try to fix within this week only. Otherwise we can go ahead with this PR.

I would prefer not to wait.  This could incorrectly flag a lot of tests as flaky that would take more time to analyze.  We can re-enable once the fix is ready.

UNSTABLE

https://github.com/opensearch-project/OpenSearch/issues/10155


[Features] Text search

Is your feature request related to a problem? Please describe.
I need possibility to filter data by text serch
Describe the solution you'd like
Text search methods like startsWith, endsWith, contains, etc
Describe alternatives you've considered
One can use collections alternatives, but native mechanism would be better.

You can use regex to search text
Example:
https://github.com/marioreggiori/objectdb/blob/main/example/lib/main.dart#L57


Add "install all" button

"install all" buttons will be very useful during installation because manual installation is terrible

maybe


requirements.txt missing in package on PyPi

Hi there,
Thanks for creating this package. I noticed that the package on PyPi misses the requirments.txt, which causes the install to fail. Installing it via pip from GitHub works though.
Cheers
#8 5.180       ERROR: Command errored out with exit status 1: #8 5.180        command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-resolver-c833913s/ariadne-jwt/setup.py'"'"'; __file__='"'"'/tmp/pip-resolver-c833913s/ariadne-jwt/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-q69q_jk3 #8 5.180            cwd: /tmp/pip-resolver-c833913s/ariadne-jwt/ #8 5.180       Complete output (5 lines): #8 5.180       Traceback (most recent call last): #8 5.180         File "<string>", line 1, in <module> #8 5.180         File "/tmp/pip-resolver-c833913s/ariadne-jwt/setup.py", line 21, in <module> #8 5.180           with open('requirements.txt', 'r') as fin: #8 5.180       FileNotFoundError: [Errno 2] No such file or directory: 'requirements.txt'

I did not encounter this issue.
the following command worked fine
pip install ariadne-jwt


No instructions how to uninstall Docker-EE on Windows Server 2016

Problem description
For Troubleshooting it was necessary to uninstall and reinstall the Docker-EE von Windows Server 2016.
Problem location
There is no documented way how you can uninstall Docker-EE on a Windows Server 2016. Due to a misconfiguration of the Engine I was forced to reinstall the engine. At the current documentation we got a section for install, for update but not how to uninstall it on a Windows Server System. I did use the following documentation:
https://docs.docker.com/install/windows/docker-ee/
We need to have a section describes the uninstall of Docker EE as well. I used the following commands to uninstall it:
UnInstall-Package Docker -ProviderName DockerProvider -Force
UnInstall-Module DockerProvider -Force
Project version(s) affected
Every Docker EE Version for Windows Server 2016
Suggestions for a fix
Simply create a new section that describes how to uninstall Docker EE in the documentation at https://docs.docker.com/install/windows/docker-ee/ with a section Uninstall Docker EE.

Actually, Microsoft has it as EE Basic is technically their product.  I don't suppose they mind if we borrow it.
https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-docker/configure-docker-daemon

Closing this ticket due to its age, and the impending refactor. If you think this is in error, feel free to reopen. Thanks!


Search Bar in FAQ section

Is your feature request related to a problem? Please describe.
In FAQ section suppose i there are huge amount of lists, i want to find answer of some specific question then i've to manuaaly scroll and find that answer.
Describe the solution you'd like
The solution of that problem will be a searchbar to that section.
Describe alternatives you've considered
Alternatively I can add filters based on different categories of FAQ asked.
Additional context

Hey @jahnvisahni31  ,can you look into this issue and let me assign me this issue under any Level between 2/3 of GSSOC-extd and HackOctoberFest. Thanks in Anticipation :)

go ahead......

@powerstone666 I've created the PR, Please review it.
Thanks!

hey just a remainder your pr has some issue please check that


Add Python 3 compatibility and some code fixes

Change docstring format to reST (PEP 0287) for better usability.

I need this.


Deppart: Partitioning by image/preimage of affine functions

Realm needs to natively support image and preimage partitioning with affine functions, as emulating them with the current API is not very nice (unless the source partition is a dense, equal partition) and wouldn't perform really well.

I suppose this would also need a modification to the Legion interface as well since you'll be projecting through an affine transformation function instead of a field right?

Yes - the initial thinking is to make it look as much like the structured scatter/gather API as possible.  As part of it, we'll probably teach structured scatter/gather how to do modulo arithmetic on points as well.

Note; this needs to support "clamping" affine maps, which aren't really affine.

I've got a piece of code in Regent that would benefit from this feature. I'll try to produce a small example in the next few days.

This is script is similar to the code that I am using right now to compute the partitions that support the stencil accesses in my code (in the case of the following script the stencil spans [0:+2]).
If I understand correctly, this feature should allow one to compute these partitions more efficiently.
import "regent"

local C = regentlib.c

local Nx = 20

local Np = 2

struct MainRegionColumns {
   t : int
   -- Other data needed in the calculation
   a : double
   b : double
   -- ...
}

struct AuxColumns {
   cp1 : int1d
   cp2 : int1d
}

local Std_node = 0
local BC1_node = 1
local BC2_node = 2

local off_Std = terralib.newlist({1,2})
local off_BC1 = terralib.newlist({1,1})
local off_BC2 = terralib.newlist({0,0})
local off = terralib.global(`array(array(off_Std),
                                   array(off_BC1),
                                   array(off_BC2)))

local function GetP1( t, c, b) return rexpr (c + int1d{off[t][0]})%b end end
local function GetP2( t, c, b) return rexpr (c + int1d{off[t][1]})%b end end

task partRegion(r : region(ispace(int1d), MainRegionColumns),
                tiles : ispace(int1d),
                p_r : partition(disjoint, r, tiles))
where reads(r.t) do
   var s = region(r.ispace, AuxColumns)
   var p_s = s & p_r
   for c in s do
      s[c].cp1 = [GetP1( rexpr r[c].t end, rexpr c end, rexpr r.bounds end)]
      s[c].cp2 = [GetP2( rexpr r[c].t end, rexpr c end, rexpr r.bounds end)]
   end
   var p_r_cp1 = image(r, p_s, s.cp1)
   var p_r_cp2 = image(r, p_s, s.cp2)

   -- Return a partition of data that need to be read for a [0:+2] stencil
   return (p_r | p_r_cp1 | p_r_cp2)
end

task main()
   C.printf("Start\n")
   var is = ispace(int1d, Nx)
   var r = region(is, MainRegionColumns)

   var tiles = ispace(int1d, Np)

   var p_r = partition(equal, r, tiles)

   -- Periodic case
   fill(r.t, Std_node)

   var p_r_wrap = partRegion(r, tiles, p_r)

   C.printf("Periodic:\n")
   for t in tiles do
      C.printf("Tile %d\n", t)
      for c in p_r_wrap[t] do
         C.printf("  %d\n", c)
      end
      C.printf("\n")
   end

   C.printf("\n\n\n")

   -- BC case
   fill(r.t, Std_node)
   r[Nx-1].t = BC2_node
   r[Nx-2].t = BC1_node

   var p_r_bc = partRegion(r, tiles, p_r)

   C.printf("BC:\n")
   for t in tiles do
      C.printf("Tile %d\n", t)
      for c in p_r_bc[t] do
         C.printf("  %d\n", c)
      end
      C.printf("\n")
   end

   C.printf("End\n")
end

regentlib.start(main)

In my recent work (using Legion for distributed sparse tensor computations) I have several operations done manually right now that could fit into an affine-partitioning based API. I'll update this as I create more.

A literal affine projection. I have a function that remaps the upper and lower bounds of index space domains i.e. Point<N> -> Point<M> according to a constructed projection and fills in the remaining M - N points with some default value (in my case the bounds of that dimension of the index space). I can do this without a temporary region right now as my partitions are all dense, but if they were sparse I would need a temporary region.

A projection that "copies" a partition. This use case takes a partition of a N-dimensional space and projects into M-dimensional space by copying all N dimensions of the partition, and then filling in the remaining M-N dimensions with the bounds of the target index space.

A projection that does a gather-style operation. I have a region full of coordinates that point into a N-dimensional dense index space. I map the coordinates to a dimension of the index space, and fill the remaining N dimensions with the bounds of the index space, and then do a partition_by_image_range to construct the final output partition.


[operator][google marketplace] update image to 1.9.0 and latest helm chart

What does this PR do?
Update Google Marketplace manifests, schema, and chart for Operator 1.9.0
Motivation
Describe how to test/QA your changes
Possible Drawbacks / Trade-offs
Additional Notes

/merge


Allow Gateway API feature to be enabled in clusters that don't have GWAPI CRDs installed

cert-manager ExperimenatlGatewayAPISupport feature is currently in alpha, whilst the Gateway API itself is now beta and seems like it's gaining traction. It would be good if we could eventually graduate our feature through beta to GA. At the moment this would not be straightforward as users cert-manager installation would crash if the Gateway API feature was enabled, but the Gateway CRDs are not installed.
Context
cert-manager integrates with Gateway API by allowing users to auto-generate certs for Gateway API (gateway-shim) as well as by allowing GWAPI resources to be used to solve HTTP-01 challenge.
The gateway-shim integration is a controller that watches Gateway resources. If those are not found in cluster, cert-manager will crash. At the moment this controller is only enabled if the ExperimentalGatewayAPISupport feature is enabled. In the future we will want to remove the feature gate to signal the stability of this feature (see upstream docs for the intended purpose of feature gates). At the same time Gateway folks say that the Gateway CRDs will not be included in standard Kubernetes installation even when the Gateway API has graduated to beta. So we need to find a way how to graduate our support for Gateway API whilst not crashing users installations if they don't have the API installed.
To solve this, we could:

require users to explicitly enable gateway-shim controller via --controllers flag as well as enable the ExperimentalGatewayAPISupport alpha feature now and when the feature is on by default, a user who is already using this feature does not have to do anything
do nothing now, but when the feature become on by default, keep the controller disabled by default and require that users explicitly enable it via --controllers flag.

I'd prefer to the first approach as it'd be less work for existing users when the feature gets switched on by default and potentially less code change.
/kind feature

Out of curiosity. What if the gateway-shim controller checks for the existence of the Gateway API and stop itself when the Gateway API doesn't exist?

What if the gateway-shim controller checks for the existence of the Gateway API and stop itself when the Gateway API doesn't exist?

This isn't really possible, the controllers are created using Kubernetes libraries and there is a standard mechanism by which a particular resource type is being watched (i.e Gateways). There is no logic to start these controllers conditionally.

@irbekrm , I have made some rough changes to achieve this and they can be seen in this draft PR: https://github.com/cert-manager/cert-manager/pull/5624. As a pre-start check, we check if gateway CRD is present; if not present, we don't start the controller even if it is enabled. Let me know your thoughts!

Hi thanks @sathyanarays I had a brief look and this looks sane. I'd like to bring this up for a team discussion, there is some current work in design phase that might or might not change the controller structure.
I'll have a closer look at this early next week and might add it for discussion to our next week's dev meeting.

we don't start the controller even if it is enabled
Hi, thanks again for looking into this @sathyanarays . I've thought about this a bit more and I think that this will be problematic for cases where the Gateway CRDs are installed at a later point (after cert-manager) as it would just silently not work.


Add Dynamic nested router cannot match the basic api prefix endpoind and all the route path didnot catched by openapi.json

Environment
Nuxt project info:                                                                      18:20:04

Operating System: Windows_NT
Node Version:     v20.8.0
Nuxt Version:     3.11.2
CLI Version:      3.11.1
Nitro Version:    2.9.6
Package Manager<EMAIL_ADDRESS>Builder:          -
User Config:      devtools, extends, modules, colorMode, content, ui, i18n, site, sitemap, eslint, features, future, routeRules, nitro, runtimeConfig
Runtime Modules<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Build Modules:    -

Reproduction
https://stackblitz.com/edit/github-y9cdne?file=server%2Fplugins%2Fhooks.ts
Describe the bug
Add Dynamic nested router  in a nitro plugin and cannot match the basic api prefix endpoind and all the route path didnot catched in openapi.json
Additional context
My background:
Nuxt/Nitro's nested router need to be a '{dirName}/[...slug].ts', it always need to be a dir and a slug file, maybe a lot annoy. And if the router defined a nested like router.get('/level1/level2'), it only match the first level.
So I try to find a way to organize these files.
First, I try to make a middleware to list all the prefix and switch it use the right router handler with a useBase
but I need to make sure it runs on the last one, so I named it with '99999-controller.middleware.ts'
Now, I try to import all the router in a nitro plugin and register them manually like the code in reproduction.
Logs
No logs

Would you see if you can reproduce this in pure Nitro (reproduction sandbox), and if so, raise there? 🙏


ARMv5TE and ARMv6 use long sequences for loading immediates

Unlike ARMv7, ARMv5TE and ARMv6 lack the movw and movt instructions. Large immediates must therefore be loaded one byte at a time in a four-byte sequence. Using the object pool for large immediate loads would reduce code size and could potentially also improve performance on low-end devices.

I came back to this issue and I got it working without passing an extra argument controlling constant pool usage.
Lots of asserts should make it reliable. I'll send a cl for review soon.


Notification to reclaim from escrow appears inconsistently

Expected Behavior
If it has been over an hour and the recipient has not already claimed their payment, the user should have the option to reclaim from escrow.
Current Behavior
Some users are reporting not seeing the reclaim notification when they are supposed to.
Context: https://celo-org.slack.com/archives/C01QQPXC5U6/p1619042282022400

Should be fixed with #351


[FEATURE] Addition of BackToTop Button

Is your feature request related to a problem? Please describe.
Addition of BackToTop Button should be there in website for better navigation from bottom to top in just single click
Please assign this to me.

Hey @Shariq2003, Please proceed, I’m assigning this issue to you. Thank you for taking the initiative.

Hey @Shariq2003 , are you still working on this issue?

Yes sir, upon review repo i have found that this repo uses a tech stack of ejs, and during setup of project I got stuck because of env requirements

have you sorted out all the requirements and are you currently working on this issue? Let me know if you need any help!

@Vignesh025 sir yes I need help in setting up the project

.env required
DB_URL="your_mongo_atlas_url"
CLOUDINARY_CLOUD_NAME="your_cloudinary_cloud_name"
CLOUDINARY_KEY="your_cloudinary_api_key"
CLOUDINARY_SECRET="your_cloudinary_api_secret"
MAPBOX_TOKEN="your_mapbox_token"

That's why I have to create accounts on cloudinary, and mapbox

Please read the INSTALLATION.md file, you'll know how to get those tokens.

That's why I have to create accounts on cloudinary, and mapbox

Yes, you need to create accounts and get the token.

Please read the INSTALLATION.md file, you'll know how to get those tokens.

That's why I have to create accounts on cloudinary, and mapbox

Yes, you need to create accounts and get the token.

Sure sir, could u please add the label of level for gssoc ext for this

Sir level 1 ?

Sir level 1?

Make your pull request, depending upon the code and discussion with gssoc team, we'll decide the level.
the level in issue doesn't matter. the level in the pr matters

Make your pull request, depending upon the code and discussion with gssoc team, we'll decide the level. the level in issue doesn't matter. the level in the pr matters

okay sir

Hey @Shariq2003 , why did you remove your assignment?

Actually the mapbox account creation requires debit card, that's why I can't setup the project initialisation

do you have a college mail id?

Yes sir

if you use that, it won't ask for any card details, it is free to use when you signup using college mail id

@Shariq2003  Can you please! assign me to work on this issue.

Hey @Vignesh025 ,
I want to work on this issue. May you please assign it to me?

Hey @Vignesh025 , I want to work on this issue. May you please assign it to me?

Hey @Sneharaj7645653 , it's already done.

Oh, I see. Thank you.


Indicate that trigger-toggles worked

This fixes #606, by including a modal overlay of a "spinner" to indicate that the updated results are loaded.
Technically, this is a bit smoke & mirrors, as it's completely unrelated to the actual loading, instead there's a quarter second period in which the overlay is shown before then reloading the website with the new results.
While definitely not an elegant solution, I think for now this is a pragmatic approach, as the more dynamic implementation would mean a quite big chunk of work to refactor how the loading of the results happens!

Thanks @helendduncan for looking over it!


Question: UDPA implemented using eBPF

As eBPF enjoys increasing uptake, I was wondering if anyone has been looking into implementing UDPA using eBPF? I came across the Dropbox engineering report on their migration story to Envoy and hence would be interested to learn what others see or think about this.

Closing this issue out as https://github.com/cncf/xds is now the active xDS working group list. FWIW I think there are efforts around xDS and eBPF.


Provide a better deep checking middleware in Reducer

Need a deep checking middleware to make sure the state is not updated if current state equals dispatchValue

Created shallow comparison. Will not do a deep check because of performance issues. If someone wants a deep check they can create a middleware for it.


about deploy

when I reproduce your basic results, I used four EC2 with Ubuntu 18.04. But when I run ./run.sh new myrun1, there is a mistake said  # start the hotstuff replica **********************************************************************************************

replica2                   - FAILED!!! ------------------------------------------------------
Unable to import hotstuff_app due to invalid syntax
replica1                   - FAILED!!! ------------------------------------------------------
Unable to import hotstuff_app due to invalid syntax
replica0                   - FAILED!!! ------------------------------------------------------
Unable to import hotstuff_app due to invalid syntax
.

start the hotstuff replica **********************************************************************************************

replica3                   - FAILED!!! ------------------------------------------------------
Unable to import hotstuff_app due to invalid syntax

STATS *******************************************************************************************************************
localhost    : ok=7	changed=0	failed=0	unreachable=0	rescued=0	ignored=0
replica0    : ok=5	changed=4	failed=1	unreachable=0	rescued=0	ignored=0
replica1    : ok=5	changed=4	failed=1	unreachable=0	rescued=0	ignored=0
replica2    : ok=5	changed=4	failed=1	unreachable=0	rescued=0	ignored=0
replica3    : ok=5	changed=4	failed=1	unreachable=0	rescued=0	ignored=0
So what's going wrong?

Seems like a python import error: probably you didn't have Python3?

Have you solved this problem? I encountered it too. :(

Close because there is no response. Reopen it if there is any progress.

Ran into the same problem and solved it by changing the installation of ansible that works with python2.7 to python3.


Handle JSONP returns that begin with a comment

Many services, including Google, Facebook, and GitHub return JSONP that begins with a js comment in order to prevent attacks from Rosetta Flash (reference).
Example URL:
https://graph.facebook.com/SOME_ID?method=get&pretty=0&sdk=joey&callback=FB.__globalCallbacks.f1c77f051c
Unfortunately, it looks like json-viewer doesn't detect that this type of return is JSONP, and therefore doesn't style it.

I saw this issue at night yesterday and you already fixed it :tada:
Thanks!


[BUG][FLINK-24668] Pulsar consumption data repeated output

flink-connector-pulsar uses the RecordsBySplits of flink-connector-base to process data, But recordsInCurrentSplit and splitsIterator are not empty when nextSplit splitSiterator.hasNext () = false.

I think the behavior for RecordsBySplits is ok. But I don't know why the data is duplicated. I'm waiting for the response from the reporter.


CI: support the timer CI to verify the stability of the ingress controller

Create a timer CI (executed once a week) to verify the latest APISIX release in different k8s versions.

good idea, support from me.

May add this in Jun.


 This results in a slow runtime given the size of the cluster.
We didn't see the same behaviour when loading 1 day, or 1 month of the dataset (the full load is the equivalent of 450 days of data).
Environment Description
EMR 6.4.0

Hudi version: 0.9.0
Spark version :

3.1.2

Hive version :

Hive 3.1.2

Hadoop version :

Amazon 3.2.1

Storage (HDFS/S3/GCS..) :

S3

Running on Docker? (yes/no) :

no
Additional context
We're using the following EMR cluster:

Master: m6g.xlarge, 64 GB EBS storage
Core: 120 x r6g.4xlarge, 512GB EBS storage per node

with this configuration the job eventually succeeds after 1 hour 18 minutes.
If we run with less storage, the job inevitably fails as we experience huge amounts of disk spill (30TB written to HDFS after a replication factor of 2)
Settings
EMR:
[
	{
		"Classification": "spark",
		"Properties": {
			"maximizeResourceAllocation": "false"
		}
	},
	{
		"Classification": "spark-defaults",
		"Properties": {
			"spark.dynamicAllocation.enabled": "false",

			"spark.executor.instances": "599",
			"spark.executor.cores": "3",
			"spark.executor.memory": "19G",
			"spark.executor.memoryOverhead": "3072",
			"spark.driver.cores": "3",
			"spark.driver.memory": "19G",
			"spark.driver.memoryOverhead": "3072",
			"spark.default.parallelism": "10036",
			"spark.sql.shuffle.partitions": "10036",

			"spark.yarn.max.executor.failures": "100",
			"spark.yarn.maxAppAttempts": "1",
			"spark.sql.hive.convertMetastoreParquet": "false",
			"spark.scheduler.mode": "FAIR",
			"spark.task.maxFailures": "10",
			"spark.rdd.compress": "true",
			"spark.shuffle.service.enabled": "true",
			"spark.serializer": "org.apache.spark.serializer.KryoSerializer",
			"spark.kryoserializer.buffer.max": "256m",

			"spark.driver.extraJavaOptions": "-XX:+UseG1GC -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark -XX:InitiatingHeapOccupancyPercent=35",
			"spark.executor.extraJavaOptions": "-XX:+UseG1GC -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark -XX:InitiatingHeapOccupancyPercent=35",

			"spark.ui.prometheus.enabled": "true",
			"spark.metrics.namespace": "spark",
			"spark.executor.processTreeMetrics.enabled": "true"
		}
	},
	{
		"Classification": "spark-metrics",
		"Properties": {
			"*.sink.prometheusServlet.class": "org.apache.spark.metrics.sink.PrometheusServlet",
			"*.sink.prometheusServlet.path": "/metrics/prometheus",
			"master.sink.prometheusServlet.path": "/metrics/master/prometheus",
			"applications.sink.prometheusServlet.path": "/metrics/applications/prometheus",

			"master.source.jvm.class": "org.apache.spark.metrics.source.JvmSource",
			"worker.source.jvm.class": "org.apache.spark.metrics.source.JvmSource",
			"driver.source.jvm.class": "org.apache.spark.metrics.source.JvmSource",
			"executor.source.jvm.class": "org.apache.spark.metrics.source.JvmSource"
		}
	},
	{
		"Classification": "capacity-scheduler",
		"Properties": {
			"yarn.scheduler.capacity.resource-calculator": "org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator "
		}
	},
	{
		"Classification": "yarn-site",
		"Properties": {
			"yarn.nodemanager.vmem-check-enabled": "false",
			"yarn.nodemanager.pmem-check-enabled": "false",
			"yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage": "99.0"
		}
	},
	{
		"Classification": "hive-site",
		"Properties": {
			"hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"
		}
	},
	{
		"Classification": "hdfs-site",
		"Properties": {
			"dfs.replication": "2"
		}
	},
	{
		"Classification": "presto-connector-hive",
		"Properties": {
			"hive.metastore.glue.datacatalog.enabled": "true",
			"hive.parquet.use-column-names": "true"
		}
	},
	{
		"Classification": "spark-hive-site",
		"Properties": {
			"hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"
		}
	}
]

Hudi:
{
    "hoodie.table.name": hudi_table_name,
    "hoodie.datasource.write.table.type": "COPY_ON_WRITE",
    "hoodie.datasource.write.operation": "bulk_insert",
    "hoodie.datasource.write.recordkey.field": "id",
    "hoodie.datasource.write.precombine.field": "processed_date",
    "hoodie.datasource.write.partitionpath.field": "story_published_partition_date",
    "hoodie.datasource.write.keygenerator.class": "org.apache.hudi.keygen.TimestampBasedKeyGenerator",
    "hoodie.deltastreamer.keygen.timebased.timestamp.type": "DATE_STRING",
    "hoodie.deltastreamer.keygen.timebased.input.dateformat": "yyyy-MM-dd'T'HH:mm:ssZ,yyyy-MM-dd'T'HH:mm:ss.SSSZ",
    "hoodie.deltastreamer.keygen.timebased.input.dateformat.list.delimiter.regex": ",",
    "hoodie.deltastreamer.keygen.timebased.input.timezone": "",
    "hoodie.deltastreamer.keygen.timebased.output.dateformat": "yyyy-MM-dd",
    "hoodie.deltastreamer.keygen.timebased.output.timezone": "UTC",
    "hoodie.datasource.hive_sync.partition_extractor_class": "org.apache.hudi.hive.MultiPartKeysValueExtractor",
    "hoodie.datasource.write.hive_style_partitioning": "true",
    "hoodie.datasource.hive_sync.enable": "true",
    "hoodie.datasource.hive_sync.database": hudi_database_name,
    "hoodie.datasource.hive_sync.table": hudi_table_name,
    "hoodie.datasource.hive_sync.partition_fields": "story_published_partition_date",
    "hoodie.datasource.hive_sync.jdbcurl": f"jdbc:hive2://{master_hostname}:10000",
    "hoodie.cleaner.commits.retained": "1",
    "hoodie.write.markers.type": "TIMELINE_SERVER_BASED",
    "hoodie.datasource.hive_sync.support_timestamp": "true",
    "hoodie.metrics.on": "true",
    "hoodie.metrics.reporter.type": "PROMETHEUS",
    "hoodie.datasource.write.drop.partition.columns": "true",
    "hoodie.bloom.index.prune.by.ranges": "false",
    "hoodie.combine.before.insert": "true",
    "hoodie.upsert.shuffle.parallelism": "10036",
    "hoodie.insert.shuffle.parallelism": "10036",
    "hoodie.bulkinsert.shuffle.parallelism": "10036",
}

We're setting parallelism based on the Tuning Guide which states to set it such that its atleast input_data_size/500MB.
Script (somewhat truncated):
def clean_data_frame(spark: SparkSession, df: DataFrame) -> DataFrame:
    """
    Sanitised field names, removing invalid characters.
    """
    # apache avro does not support - in column names
    def sanitizeFieldName(s: str) -> str:
        return re.sub("[^0-9a-zA-Z_]+", "_", s)

    # We call this on all fields to create a copy and to perform any changes we might
    # want to do to the field.
    def sanitizeField(field: StructField) -> StructField:
        field = copy(field)
        field.name = sanitizeFieldName(field.name)
        # We recursively call cleanSchema on all types
        field.dataType = cleanSchema(field.dataType)
        return field

    def cleanSchema(dataType: DataType) -> DataType:
        dataType = copy(dataType)
        # If the type is a StructType we need to recurse otherwise we can return since
        # we've reached the leaf node
        if isinstance(dataType, StructType):
            # We call our sanitizer for all top level fields
            dataType.fields = [sanitizeField(f) for f in dataType.fields]
            dataType.names = [f.name for f in dataType.fields]
        elif isinstance(dataType, ArrayType):
            dataType.elementType = cleanSchema(dataType.elementType)
        return dataType

    # Now since we have the new schema we can create a new DataFrame by using the old Frame's RDD as data and the new schema as the schema for the data
    return spark.createDataFrame(df.rdd, cleanSchema(df.schema))

def read_parquet(spark: SparkSession, input_data_path: str) -> DataFrame:
    return spark.read.parquet(input_data_path)

def read_json(spark: SparkSession, input_data_path: str) -> DataFrame:
    return clean_data_frame(
        spark,
        spark.read.json(path=input_data_path).select(
            f.col("_id"),
            f.col("_version"),
            f.col("_source.*"),
        ),
    )

def map_column(
    schema_datatype: DataType, column_datatype: DataType, c: Column
) -> Column:
    try:
        # converts JSON string columns to their StructType equivalent
        if isinstance(column_datatype, StringType) and isinstance(
            schema_datatype, StructType
        ):
            return f.from_json(c, schema_datatype)

        # for each nested column, set any missing fields to null and drop any fields not in the target
        if isinstance(schema_datatype, StructType):
            target_schema_struct_columns = []

            for nested_field in schema_datatype.fields:
                nested_datatype = nested_field.dataType

                if isinstance(nested_datatype, (StructType, ArrayType)):
                    new_column = map_column(
                        nested_datatype,
                        column_datatype[nested_field.name].dataType,
                        c.getField(nested_field.name),
                    )
                elif nested_field.name in column_datatype.fieldNames():
                    new_column = c.getField(nested_field.name).cast(nested_datatype)
                else:
                    new_column = f.lit(None).cast(nested_datatype)

                target_schema_struct_columns.append(new_column.alias(nested_field.name))

            return f.struct(*target_schema_struct_columns)

        # TODO: support recursive conversion of nested columns with mismatched schema
        if isinstance(schema_datatype, ArrayType):
            schema_json = schema_datatype.jsonValue()
            schema_json["containsNull"] = True
            schema_datatype = ArrayType.fromJson(schema_json)

        return c.cast(schema_datatype)
    except Exception as ex:
        logger.exception(
            "Failed to cast column: %s with type %s to type %s",
            c.name,
            column_datatype.json(),
            schema_datatype.json(),
        )
        raise ex

def cast_target_columns_to_schema(input_df: DataFrame, schema: StructType) -> DataFrame:
    """
    Casts all columns in the input_df to the same column type as the column with a matching name in the target schema.
    Drops columns which do not exist in the target schema.
    Order is preserved in the target schema - this is necessary as Hudi does not support re-ordering of columns or dropping existing columns, new columns must be added at the end.
    i.e. "Add a new nullable column and change the ordering of fields" fails on on all table types
    see https://hudi.apache.org/docs/schema_evolution/
    """
    input_schema_field_names = input_df.schema.fieldNames()
    schema_field_names = schema.fieldNames()
    logger.debug("input data field names %s", input_schema_field_names)
    logger.debug("schema field names %s", schema_field_names)
    columns = []
    for field in schema.fields:
        if field.name.startswith("_hoodie"):
            continue
        if field.name not in input_schema_field_names:
            logger.info(
                "Column %s does not exist in source dataframe, added as null",
                field.name,
            )
            mapped_column = f.lit(None).cast(field.dataType)
        else:
            input_df_field_datatype = input_df.schema[field.name].dataType
            mapped_column = map_column(
                field.dataType, input_df_field_datatype, f.col(field.name)
            )

            logger.info(
                "Column %s existed in source dataframe, casted from schema %s to schema %s",
                field.name,
                field.dataType.json(),
                input_df_field_datatype.json(),
            )
        columns.append(mapped_column.alias(field.name))

    for field_name in input_schema_field_names:
        if field_name not in schema_field_names:
            logger.info("Column %s did not exist in target schema, dropped", field_name)

    return input_df.select(*columns)

def set_df_columns_nullable(
    spark: SparkSession, df: DataFrame, nullable: bool = True
) -> DataFrame:
    """
    Set all dataframe columns as nullable.
    This is primarily to match the output of Firehose/Hudi deltastreamer, as otherwise schema validation fails when writing into a table using Deltastreamer and syncing schema,
    as columns are set to nullable at this point.
    """

    def set_nullable(st: StructType) -> StructType:
        for struct_field in st.fields:
            if isinstance(struct_field.dataType, StructType):
                struct_field.dataType = set_nullable(struct_field.dataType)
            elif isinstance(struct_field.dataType, ArrayType) and isinstance(
                struct_field.dataType.elementType, StructType
            ):
                struct_field.dataType.elementType = set_nullable(
                    struct_field.dataType.elementType
                )
            struct_field.nullable = nullable
        return st

    return spark.createDataFrame(df.rdd, set_nullable(df.schema))

def main():
    spark = (
        SparkSession.builder.appName("hudi_full_load")
        .enableHiveSupport()
        .getOrCreate()
    )

    input_df = (
        (
            read_parquet(spark, input_data_path)
            if source_format == "parquet"
            else (read_json(spark, input_data_path))
        )
        .withColumnRenamed("_id", "id")
        # drop the existing version column which existed under _source.version in some old chunked datalake data
        # to avoid naming conflicts
        .drop("version")
        .withColumnRenamed("_version", "version")
    )
    input_df.printSchema()

    if target_table_schema is not None:
        logger.info("Using target table %s", target_table_schema)
        existing_table_df = spark.table(target_table_schema)
        logger.info(
            "Target schema for %s: %s", target_table_schema, existing_table_df.schema
        )
        existing_schema = existing_table_df.schema
        input_df = cast_target_columns_to_schema(input_df, existing_schema)
        input_df.printSchema()

    input_df = input_df.withColumn(
        "version", f.coalesce(f.col("version"), f.lit(0))
    ).withColumn("story_published_partition_date", f.col("story_published_date"))

    input_df = set_df_columns_nullable(spark, input_df, nullable=True)

    parallelism = spark.conf.get("spark.default.parallelism")
    logger.info("spark.default.parallelism: %s", parallelism)
    hudi_parallelism_options = {
        "hoodie.upsert.shuffle.parallelism": parallelism,
        "hoodie.insert.shuffle.parallelism": parallelism,
        "hoodie.bulkinsert.shuffle.parallelism": parallelism,
    }

    partitioned_df = input_df.repartition(int(parallelism)).persist(
        StorageLevel.MEMORY_AND_DISK
    )

    (
        partitioned_df.write.format("org.apache.hudi")
        .options(**hudi_options)
        .options(**hudi_parallelism_options)
        .mode(spark_write_mode)
        .save(hudi_table_path)
    )

if __name__ == "__main__":
    main()

Screenshots
I've included screenshots of the stages with shuffle spill. This is using bulk insert as the write mode into an empty table:

This is the disk usage on the 65.0 TB HDFS for the cluster during the jobs runtime (the low disk usage at the end is after the job completes and we run a second job on the same cluster).

@Limess for bulkinsert can you try setting https://hudi.apache.org/docs/configurations#hoodiebulkinsertsortmode to NONE. From your configs, it looks like it defaults to GLOBAL_SORT which could resulted in more disk spill. Please let us know if this makes any difference.

Would that not have a performance impact at query time on the end table?
This currently costs us a couple hundred dollars so we're not in the position to re-run regularly.

@Limess a few questions

for this dataset do you want to run bulkinsert or upsert? if it's append only dataset, then bulkinsert should be the mode
does smaller parquet files matter a lot in this case? GLOBAL_SORT is expected to use more diskspace as it does a shuffling sort to line up records for bulk writing. If you change to NONE sort mode, you'd get more more small files but faster write and less disk spill. Small files can be mitigated by clustering. There is some trade-off to consider based on your needs.
10036 parallelism may be a bit too high; some number around 3600 = # executors x # cores x 2 ?

@nsivabalan @yihua any other tuning tips could help here?

Thanks!
We're using bulk insert for this job and are happy with the performance vs regular upsert.
Re: parallelism, we bumped this up after:

Reading https://cwiki.apache.org/confluence/display/HUDI/Tuning+Guide, I guess this is now out of date
Observing the disk spill, we found increasing parallelism reduced it

Smaller parquet files don't matter too much, if clustering can later fix the small files/sorting problems that sounds like a good thing to look at down the line (we haven't investigated clustering at all yet)

thanks.
in general, try to ensure that each executor has enough memory to accommodate the parallel tasks that it can run based on your configs. for eg, if you let 8 tasks to run concurrently per executor and if executor has 32 GB for eg, each task may get only ~4 GB. Disk spill could happen only when there isn't sufficient memory.
btw, will close the issue out if you are good w/ perf now.

@Limess thanks for the info provided so far on this issue. closing this now assuming no further action needed here.


Allow changing the reserved keys

Allow changing the reserved keys in the confliction, call init_reserved_keys to change it

LGTM

/blossom-ci

/blossom-ci


Redis does not bind to all interfaces on Ubuntu machines

In tasks/Debian/redis.yml, the following block appears:

name: Ensure redis binds to accessible IP
lineinfile:
dest: /etc/redis/redis.conf
regexp: '^bind'
line: 'bind <IP_ADDRESS>'
notify: restart redis service

This is not present in tasks/Ubuntu/redis.yml, and as such redis does not bind to all interfaces on Ubuntu machines

PR Merged! Thank you!


Dev

Track the performance of each target separately using MSE metric.

Track the performance of each target separately.


[haskell-updates] haskellPackages.ghcide: 0.1.0 -> 0.2.0 (fix build)

I decided to leave haskell-lsp and haskell-lsp-types as extra packages,
because they will get bumped before the next ghcide release, and then
the version pinned by ghcide would be missing again.
Also I have run hackage2nix to automatically generate the changes.
If desired I can split the second part into its own commit, would that be better?

Motivation for this change
Fix new ghcide version.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

@maralorn In general we don't modify the hackage-packages.nix file manually, but wait for the automatic daily update thing to run.  (Which is slightly annoying, because it is hard to test that the PR will actually work.)
Could you remove the changes to hackage-packages.nix from this PR?

@cdepillabout Last friday @peti said, that he is totally open for others to also run the hackage-packages.nix update. So I thought this is fine.
I saw him rebasing all updates together before merging into master so maybe I should do it into it's own commit with the standard commit-message that ./update-nixpkgs.sh generates?
I think doing PRs like this is better for testing. But we can stick with the other way if that‘s preferred.

@maralorn Ah, okay, if @peti is okay with it, then it works for me!
I'll let him comment here before I merge it though.

In general we don't modify the hackage-packages.nix file manually, but wait for the automatic daily update thing to run.

I believe that no harm can come from people submitting changes to hackage-packages.nix that were generated by running hackage2nix properly. Those changes should not cause any merge conflicts when the generator commits the next update since it generates the exact same change. What we don't is people committing random manual changes to the file, obviously, but changes that come from hackage2nix should be safe.

@peti That makes sense.
In the future, I'll be sure to merge PRs even if they contain changes to hackage-packages.nix, as long as those changes have been generated by hackage2nix properly.


[Info request] Addic7ed provider 304 responses

I notice that I get a lot of 304 responses from the Addic7ed provider.
The last develop version of subliminal, already takes care of this for not discarding the provider.
However, a 304 always gives an empty response, so no content is returned.
Since I'm using subliminal in api mode, I want to get the response anyway, even with a 304.
I propose to implement a caching on the requets for Addic7ed, which should cache the responses as long as the server would return a 304.
Does anyone have an idea how long it takes until Addic7ed returns a 200 again instead of a 304?
If yes, we can use requests_cache for this, which makes sure that we always get a response (also for a 304 because the 200 will have been cached before)
Changes in addic7ed.py provider
from requests_cache import CachedSession
...
def initialize(self):
    self.session = CachedSession(cache_name='subliminal.addic7ed.cache', expire_after=600)
    self.session.headers['User-Agent'] = 'Subliminal/%s' % __short_version__

What do you think of this proposal? If you agree, I can make the PR.
I already tested it in my tool, and I get now the cached response in case of a 304.
The only thing left is the timeout (I took now 10 minutes)...Does anyone has an idea?
PS: I only checked it on Addic7ed because that's the one I'm using. I don't know if this also happens on other providers. If yes, perhaps we can make it global for all providers?

The 304 with no content is addic7ed blocking the subliminal

Ok, so wouldn't the cache be a solution for that?

How cache something you never got a valid response? The block can be in any request, include de sub download

I've just changed the 'User-Agent' and I'm still getting the 304... So how is Addic7ed knowing it's still subliminal?

Only changing the user agent seems to fix the 304 responses temporarily.


Implement download within viewer

Who it's for
Viewer users - to enable download directly from the viewer
Problem to be solved
Without a download button in the viewer, users must go back to the Work page to do so. This doesn't necessarily work well for users who have come to the viewer from a search engine, as they have no previous context for the work page, and may not know that download is an option.
Along with making it obvious that they can download these materials, users also need to know the licence conditions so they can be confident in their rights to use this material.
Acceptance Criteria

[ ] Download button visible in viewer
[ ] All current existing download options provided (full size, small size, PDF output)
[ ] Licensing information visible or accessible.

Success
TBD

#4325


Incorrect style for nested links in BitNav component

in Fluent UI the nested links show as bellow:

but in BitNav we have redundant space between icon and link text, the icon must be beside the text  with a few spaces from icon of the parent item

This issue is fixed in #830


oterm: 0.2.9 -> 0.4.2

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for oterm is: Text-based terminal client for Ollama
meta.homepage for oterm is: https://github.com/ggozad/oterm
meta.changelog for oterm is: https://github.com/ggozad/oterm/releases/tag/0.4.2
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 0.4.2 with grep in /nix/store/rz86g6ffvkhihz5pjxhsl4wxqs9n23c3-oterm-0.4.2
found 0.4.2 in filename of file in /nix/store/rz86g6ffvkhihz5pjxhsl4wxqs9n23c3-oterm-0.4.2

Rebuild report (if merged into master) (click to expand)

2 total rebuild path(s)

2 package rebuild(s)

First fifty rebuilds by attrpath

oterm

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/rz86g6ffvkhihz5pjxhsl4wxqs9n23c3-oterm-0.4.2 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A oterm https://github.com/r-ryantm/nixpkgs/archive/6815fd88e71a63f9b6aced293089f961adf7a3db.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/6815fd88e71a63f9b6aced293089f961adf7a3db#oterm

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/rz86g6ffvkhihz5pjxhsl4wxqs9n23c3-oterm-0.4.2
ls -la /nix/store/rz86g6ffvkhihz5pjxhsl4wxqs9n23c3-oterm-0.4.2/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

2 packages built:

oterm
oterm.dist

Maintainer pings
cc @suhr for testing.

[!TIP]
As a maintainer, if your package is located under pkgs/by-name/*, you can comment @NixOS/nixpkgs-merge-bot merge to automatically merge this update using the nixpkgs-merge-bot.

Add a :+1: reaction to pull requests you find important.

@NixOS/nixpkgs-merge-bot merge


Update upterm to 0.4.3

After making all changes to the cask:

[x] brew cask audit --download {{cask_file}} is error-free.
[x] brew cask style --fix {{cask_file}} left no offenses.
[x] The commit message includes the cask’s name and version.

Additionally, if updating a cask:

[ ] sha256 changed but version stayed the same (what is this?).
I’m providing public confirmation below.

@shoichiaizawa just wanted to thank you for regularly updating brew cask, it's been very helpful!


Run iCub telemetry server and openMCT visualizer in one line command: Iteration 1/2

Implements #25 .

Could it make sense to document its use somewhere? Furthermore, if we want to stop the program launched with this script, what would be the best way?

I was planning to add all the missing updates and the how to of this script in #22 , but indeed it's better to do it right away here.
With the current implementation, after the servers are launched by this script, we can stop them with a single command pkill -3 node. This is acceptable if these servers are the only running Node.js applications.
Another way would be to run from the terminal only the iCubTelemetryServer, and then have the server itself run the openmctStaticServer as child process using the spawn command.
Actually this should be a better solution since installing and running the openmctStaticServer server separately was only due to the difference in the Node.js versions.

I'll push the changes here.

Further analysis ongoing ... (https://github.com/ami-iit/yarp-openmct/issues/25#issuecomment-914900844)

A class OpenMctServerHandler object has been created for handling the child process meant to run the openMCT visualizer. For now:

The object is instanciated (only a few properties initialized).
The nvm use v14.17.0 command is run (unsuccessful, with no effect on other ongoing processes, to be fixed).

@traversaro , although it is not the best way to proceed in general, I would like to freeze the work of this PR as is, merge it, and implement the 2nd iteration (2/2: ) in a new PR on top of al the others. This would simplify the merge of the remaining PRs #20  and #30 . Do you agree?

@traversaro , although it is not the best way to proceed in general, I would like to freeze the work of this PR as is, merge it, and implement the 2nd iteration (2/2: ) in a new PR on top of al the others. This would simplify the merge of the remaining PRs #20 and #30 . Do you agree?

Ok!

Second iteration implemented in #32 .


403 error returned on all requests to atlas in AzureMapsWebApiToken sample code

I am trying to use the AzureMapsWebApiToken sample code. I have setup Azure Maps and Azure AD according to the README.

I created the App Registration in Azure Active Directory
I generated a Client Secret in Azure AD
I created the environment variable AzureServicesAuthConnectionString including the IDs and made AppKey the same as the Client Secret
I granted the Role Assignment in Azure AD to my Maps app registration

All requests to the atlas API return 403.
My requests do include an Authorization token (Bearer) in the header.
This is the error I get:
{
  "error": {
    "code": "403 Forbidden",
    "message": "Permission, capacity, or authentication issues."
  }
}

What am I doing wrong? Thank you.

I tried re-generating the Client Secret and now it is working. Thank you.


SocketException: Failed host lookup

Copied from flutter issue #132912
Steps to reproduce
Just make a network request.
For example, using the basic http package.
Also relevant when using other packages, such as Dio.
The same error, and in the same versions of Flutter.
Therefore, I think they have the same root.
The bug is very critical, as already many communities of various applications complain about it.
I have given the simplest code example below.
The main essence is in the line:
await http.get(Uri.parse('https://flutter.dev'));

This request throws an exception:
SocketException (SocketException: Failed host lookup: 'flutter.dev' (OS Error: ... , errno = 11004))
This error appears on ANY host of the many that I have tried.
That is, the problem is not in a particular host.
This problem is relevant for Windows (which discards many other similar problems that can be found on the Internet and which are associated with missing permissions on the Android system). It's not that problem.
I've reproduced this bug in the latest master as well as in the latest stable release.
On the Internet people advise to downgrade Flutter to version 3.10.5, where this problem does not reproduce. I personally haven't tried it yet, but I suspect that all versions after 3.10.5 have this bug.
But downgrading to that version is not an option for me anyway.
Please fix this bug as soon as possible.
Yes, this exception is skippable, but it creates debugging inconvenience in many cases.
Moreover, this problem is relevant for many.
Thank you!
Expected results
No exception
Actual results
This request throws an exception:
SocketException (SocketException: Failed host lookup: 'flutter.dev' (OS Error: ... , errno = 11004))
Code sample
Code sample
import 'package:flutter/material.dart';
import 'package:http/http.dart' as http;

void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
        colorScheme: ColorScheme.fromSeed(seedColor: Colors.deepPurple),
        useMaterial3: true,
      ),
      home: const MyHomePage(),
    );
  }
}

class MyHomePage extends StatefulWidget {
  const MyHomePage({super.key});

  @override
  State<MyHomePage> createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  Future<void> _test() async {
    try {
      await http.get(Uri.parse('https://flutter.dev'));
    } catch (ex) {
      // ignore: use_build_context_synchronously
      showDialog(
        context: context,
        builder: (context) {
          return AlertDialog(
            title: const Text('Result'),
            content: Text(
              ex.toString(),
            ),
            actions: <Widget>[
              TextButton(
                child: const Text('Ok'),
                onPressed: () {
                  Navigator.of(context).pop();
                },
              ),
            ],
          );
        },
      );
    }
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      floatingActionButton: FloatingActionButton(
        onPressed: _test,
        child: const Icon(Icons.error),
      ),
    );
  }
}

Screenshots or Video
No response
Logs
No response
Flutter Doctor output
Doctor output
Is there an existing issue for this?

    I have searched the [existing issues](https://github.com/flutter/flutter/issues)
    I have read the [guide to filing a bug](https://flutter.dev/docs/resources/bug-reports)

Steps to reproduce

Just make a network request.

For example, using the basic http package.

Also relevant when using other packages, such as Dio.
The same error, and in the same versions of Flutter.

Therefore, I think they have the same root.

The bug is very critical, as already many communities of various applications complain about it.

I have given the simplest code example below.
The main essence is in the line:

await http.get(Uri.parse('https://flutter.dev'));

This request throws an exception:
SocketException (SocketException: Failed host lookup: 'flutter.dev' (OS Error: ... , errno = 11004))

This error appears on ANY host of the many that I have tried.
That is, the problem is not in a particular host.

This problem is relevant for Windows (which discards many other similar problems that can be found on the Internet and which are associated with missing permissions on the Android system). It's not that problem.

I've reproduced this bug in the latest master as well as in the latest stable release.
On the Internet people advise to downgrade Flutter to version 3.10.5, where this problem does not reproduce. I personally haven't tried it yet, but I suspect that all versions after 3.10.5 have this bug.
But downgrading to that version is not an option for me anyway.

Please fix this bug as soon as possible.
Yes, this exception is skippable, but it creates debugging inconvenience in many cases.
Moreover, this problem is relevant for many.

Thank you!
Expected results

No exception
Actual results

This request throws an exception:
SocketException (SocketException: Failed host lookup: 'flutter.dev' (OS Error: ... , errno = 11004))
Code sample
Code sample

import 'package:flutter/material.dart';
import 'package:http/http.dart' as http;

void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
        colorScheme: ColorScheme.fromSeed(seedColor: Colors.deepPurple),
        useMaterial3: true,
      ),
      home: const MyHomePage(),
    );
  }
}

class MyHomePage extends StatefulWidget {
  const MyHomePage({super.key});

  @override
  State<MyHomePage> createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  Future<void> _test() async {
    try {
      await http.get(Uri.parse('https://flutter.dev'));
    } catch (ex) {
      // ignore: use_build_context_synchronously
      showDialog(
        context: context,
        builder: (context) {
          return AlertDialog(
            title: const Text('Result'),
            content: Text(
              ex.toString(),
            ),
            actions: <Widget>[
              TextButton(
                child: const Text('Ok'),
                onPressed: () {
                  Navigator.of(context).pop();
                },
              ),
            ],
          );
        },
      );
    }
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      floatingActionButton: FloatingActionButton(
        onPressed: _test,
        child: const Icon(Icons.error),
      ),
    );
  }
}

Screenshots or Video

No response
Logs

No response
Flutter Doctor output
Doctor output

[✓] Flutter (Channel master, 3.14.0-8.0.pre.17, on Microsoft Windows [Version 10.0.22621.2134], locale ru-UA)
    • Flutter version 3.14.0-8.0.pre.17 on channel master at C:\SDK\Flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 6f227c0784 (2 days ago), 2023-08-18 19:04:15 -0700
    • Engine revision f4bffdcf85
    • Dart version 3.2.0 (build 3.2.0-87.0.dev)
    • DevTools version 2.26.1

[✓] Windows Version (Installed version of Windows is version 10 or higher)

[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.2)
    • Android SDK at C:\SDK\Android
    • Platform android-33-ext5, build-tools 33.0.2
    • Java binary at: C:\Program Files\Android\Android Studio\jbr\bin\java
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-b2043.56-9586694)
    • All Android licenses accepted.

[✓] Chrome - develop for the web
    • CHROME_EXECUTABLE = C:\Program Files\Chromium\chrome

[✓] Visual Studio - develop Windows apps (Visual Studio Community 2022 17.7.1)
    • Visual Studio at C:\Program Files\Microsoft Visual Studio\2022\Community
    • Visual Studio Community 2022 version 17.7.34009.444
    • Windows 10 SDK version 10.0.22621.0

[✓] Android Studio (version 2022.2)
    • Android Studio at C:\Program Files\Android\Android Studio
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-b2043.56-9586694)

[✓] VS Code (version 1.81.1)
    • VS Code at C:\Users\Yegor\AppData\Local\Programs\Microsoft VS Code
    • Flutter extension version 3.70.0

[✓] Connected device (3 available)
    • Windows (desktop) • windows • windows-x64    • Microsoft Windows [Version 10.0.22621.2134]
    • Chrome (web)      • chrome  • web-javascript • unknown
    • Edge (web)        • edge    • web-javascript • Microsoft Edge 115.0.1901.203

[✓] Network resources
    • All expected network resources are available.

Possible reason: flutter/flutter#132912 (comment)

The behavior changed between 3.10.x (3.10.5 precisely) and 3.13.0 without an announcement about breaking changes so it shouldn't be a migration on the developer side to fix it. Especially since you can't change the configuration of certain servers when you don't have control over them.
I've found this comment which details in a more precise way the problem: issue #4695

This seems related to https://github.com/dart-lang/sdk/issues/53334

@NeroThroN are you seeing this only on Windows?
I tried running your simple example on Linux and do not see the exception.

@NeroThroN do you observe this exception only when running under the debugger or do you see it even on a regular run outside the debugger ?

This seems related to #53334

Yes indeed, I think it is indeed related

@NeroThroN are you seeing this only on Windows? I tried running your simple example on Linux and do not see the exception.

Based on my scenarios, I have the issue in multiple scenarios:
(Development OS > Target Device => result)

Linux (Fedora 38) > Linux (Fedora 38) => Problem occurs
Linux (Fedora 38) > Old Android (Android 9) => Problem does not occur
Windows 11 > New Android (Android 13) => Problem occurs

@NeroThroN do you observe this exception only when running under the debugger or do you see it even on a regular run outside the debugger ?

I only had the problem while in debug mode. Without debugger mode or even in release mode, the application works fine.
I also tested the first of the above scenarios (Linux > Linux) using Android Studio instead of vscode, and the problem did not appear.

I have same issue on window + android studio in debug. Running Android 13 at emulator.
Did not try in release mode. Downgraded to flutter 3.10.6

I can repro this issue with

Flutter 3.13.1,
VS code 1.81 on Windows. 'All exceptions' is not selected, 'Uncaught exceptions' is selected. Debug mode is 'Just my code'.
targetting Android 7.1.1

The exception is thrown and the debugger stops when making an https (secure) request to a IPv4-only internet (public) host, not supporting IPv6.
The debugger does not stop (meaning the exception is not thrown?) when making an http request to an IPv4-only LAN (not internet) host.
I understand that the http package tries IPv4 connections first, based on this reply from Brian Quinlan @brianquinlan :
https://github.com/dart-lang/sdk/issues/50868#issuecomment-1487709590
"I modified the socket connection logic to favor IPv4 connections over IPv6 connections, as we already did on iOS."

Because of this, it is a bit surprising that the debug stops when dealing with IPv4-only internet addresses.

I only had the problem while in debug mode. Without debugger mode or even in release mode, the application works fine. I also tested the first of the above scenarios (Linux > Linux) using Android Studio instead of vscode, and the problem did not appear.

Based on this I am marking this as a duplicate of https://github.com/dart-lang/sdk/issues/53334

any solution yet?

@cetindogu this issue is closed, if you are encountering an issue please open a new issue with instructions on how to reproduce the problem you are encountering.

i have faced this problem in all my projects for a long time, which work with the API. I can't  uncheck "all exceptions" because I need to see app exceptions inside try catch
I am very much looking forward to the decision

@astheras a fix for the above problem was landed a while back which problem are you still facing ?

I am still facing the same issue, flutter version : 3.10.6.
It is not easy to reproduce, but the way I was able to do it was, by minimizing the application for a while, re-opening it, and trying to perform any 'http calls' inside of the app for some time.
I am using Dio Package, and whenever this problem occurs, my interceptors are not getting hit and all HTTP calls automatically get 401 without going into the interceptors where I inject the auth token. it only works again If I stop the app and restart it.

I think I am seeing an issue related to this.
I have a Global external application load balancer exposing a HTTP ip v4 url. My Flutter app connects to this using NetworkImage to load images. This is working well for most users most of the time. I have some users somehow getting into a state where all images stop loading. These are very small images (~10KB) which cannot be loaded on a very fast cell signal. When the user switches over to WIFI the images start loading again.

I think I am seeing an issue related to this.
I have a Global external application load balancer exposing a HTTP ip v4 url. My Flutter app connects to this using NetworkImage to load images. This is working well for most users most of the time. I have some users somehow getting into a state where all images stop loading. These are very small images (~10KB) which cannot be loaded on a very fast cell signal. When the user switches over to WIFI the images start loading again.

I have users too who got failed lookup


List Formatting - Replace function just replace first match

Target SharePoint environment
SharePoint Online
What SharePoint development model, framework, SDK or API is this about?
Declarative list formatting
Developer environment
Windows
What browser(s) / client(s) have you tested

[ ] 💥 Internet Explorer
[X] 💥 Microsoft Edge
[X] 💥 Google Chrome
[X] 💥 FireFox
[X] 💥 Safari
[ ] mobile (iOS/iPadOS)
[ ] mobile (Android)
[ ] not applicable
[ ] other (enter in the "Additional environment details" area below)

Additional environment details
JSON functions from microsoft documentation:
https://docs.microsoft.com/en-us/sharepoint/dev/declarative-customization/formatting-syntax-reference
Describe the bug / error
The following error appear when Power user uses JSON function "replace" to replace repetitive word using format column.
Add following text in "single line of text or multiple line of text" Column:
"test1 test2 test3 test4 test5"
When function below is execute, only the first Match is replaced:

Steps to reproduce

Create column of type "single line of text or multiple line of text"
Access to format column
include following command "txtContent": "=replace(@currentField,'test','')"

Expected behavior
Or "replace" function should replace all match words or should exist "replaceAll" function to replace all match words

Is there any update on this issue? It's really annoying to have no function to replace all occurrences in the SP formatting.

Hi @PieterVandendriessche
Try:
"txtContent": "=replaceAll('H-e-l-l-o W-o-r-l-d', '-', '')"

Hi @aaclage
Thanks for your answer. The replaceAll indeed works for the txtContent field. In my use case I am trying to get it working on the href attrinbute, see my use case below (with workaround):
{   "$schema": "https://developer.microsoft.com/json-schemas/sp/v2/column-formatting.schema.json",   "elmType": "a",   "txtContent": "@currentField",   "attributes": {     "href": "=@currentWeb + '/Lists/MEMBRES/AllItems.aspx?FilterField1=EquipeFR&FilterValue1=' + replace(replace(replace(replace([$Title],'&','%26'),'&','%26'),'&','%26'),'&','%26') + '&FilterType1=Lookup'",     "target": "_blank"   } }

Tested your code.
Unless did something wrong the replaceAll did the job correctly.
{
  "$schema": "https://developer.microsoft.com/json-schemas/sp/v2/column-formatting.schema.json",
  "elmType": "a",
  "txtContent": "@currentField",
  "attributes": {
    "href": "=@currentWeb + '/Lists/MEMBRES/AllItems.aspx?FilterField1=EquipeFR&FilterValue1=' + replaceAll([$Title],'&','%26') + '&FilterType1=Lookup'",
    "target": "_blank"
  }
}

@aaclage Ok thanks for that. Indeed it works for me now as well. Must have mixed up something in the syntax.


Bidirectional streaming with SSL starts crashing after some time

What version of gRPC and what language are you using?
grpcio versions 1.51 to 1.54
Python 3.9.16
What operating system (Linux, Windows,...) and version?
Debian 10 with updates for April 27th, 2023
Kernel 4.19.0-23
OpenSSL 1.1.1n-0+deb11u4
What runtime / compiler are you using (e.g. python version or version of gcc)
Python 3.9.16
What did you do?
The client code is (simplified):
def _chunks(data: io.BytesIO):
    while content := data.read(4096):
        yield SvcRequest(data=content)

credentials = grpc.ssl_channel_credentials(...)
with grpc.secure_channel(target='server:33001', credentials=credentials) as channel:
    stub = SvcStub(channel)
    res = stub.Call(itertools.chain([SvcRequest(smth='string')], _chunks(data)))
    for chunk in res:   #  <---- Python stacktrace points on this line
        f.write(chunk.data)

This code is called in a multithreaded environment inside another grpc call. The file sent in chunks is not big, it's a relatively small HTML page (3-6kb), so usually the client makes 2 or 3 calls only. The request is consumed like this (also simplified, it doesn't keep things in memory):
for part in request_iterator:
    part_type = part.WhichOneof('msg')
    if part_type == 'smth':
        description = part.smth
    else:
        data += part.data

target: Path = do_smth(description, data)

with target.open(mode='rb') as f:
    while chunk := f.read(20480):
        yield SvcResponse(data=chunk)

Plz note that I was unable to reproduce this issue in any synthetic environment with this code alone, so I guess there's something related to multithreading and not cleaning up/freeing memory properly.
What did you expect to see?
Stream working w/o crashing
What did you see instead?
After some time it starts crashing with grpc_message:"Stream removed" and with additional SSL error (this is for 1.51, for 1.54 the file is different but the error is the same):
E0427 08:02:22.591486946    1145 ssl_transport_security.cc:556]        Corruption detected.
E0427 08:02:22.591548313    1145 ssl_transport_security.cc:532]        error:1e000065:Cipher functions:OPENSSL_internal:BAD_DECRYPT
E0427 08:02:22.591556670    1145 ssl_transport_security.cc:532]        error:1000008b:SSL routines:OPENSSL_internal:DECRYPTION_FAILED_OR_BAD_RECORD_MAC
E0427 08:02:22.591563513    1145 secure_endpoint.cc:304]               Decryption error: TSI_DATA_CORRUPTED
E0427 08:02:22.591684789    1145 ssl_transport_security.cc:583]        SSL_write failed with error SSL_ERROR_SSL.

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/example.py", line 210, in generate
    for chunk in res:
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 426, in __next__
    return self._next()
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 826, in _next
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Stream removed"
        debug_error_string = "UNKNOWN:Error received from peer ipv4:<IP_ADDRESS>:33001 {created_time:"2023-04-27T08:02:22.591645594+00:00", grpc_status:2, grpc_message:"Stream removed"}"
>

Anything else we should know about your project / environment?
All other call types incl Stream-Unary and Unary-Stream work fine, the only issue is with Stream-Stream. I was unable to reproduce this error in any non-production environment (with the same docker image, same libraries, same OS config etc).
After the request crashes once, all subsequent requests will also crash.

Hi, it's hard to tell what's wrong with existing information, can you help provide more logs with those environment vars added?
GRPC_VERBOSITY=DEBUG GRPC_TRACE=all,-timer,-timer_check

We do have lots of test for multi-threading, so it will be surprising if this issue was caused by multi-threading.
Can you share more information about your network setup? And also more information about the difference between the production environment and testing environment? This could also happen in case of network middleware doing incorrect deep packet inspection.