[Feature Request] crawl through the onion network to prevent abuse reports

Have you checked the roadmap on the website, and existing issues, before opening a dupllcate issue?
Yes
Is your feature request related to a problem? Please describe.
If you run this program you will start to show up in iknowwhatyoudownload.com and could potentially start receiving abuse reports.
Describe the solution you'd like
Let us route everything through tor if we want to, disabled by default. Make sure dns is proxied to prevent rdns leakage. Either a docker-compose tor container or system level tor service and have this container route through it either via the socks5 proxy on localhost, or just let us have a enviroment variable that is respected, like
SOCKS_PROXY=<IP_ADDRESS>:9050
services:
  bitmagnet:
    build:
      context: .
    container_name: bitmagnet
    #     image: ghcr.io/bitmagnet-io/bitmagnet:latest
    volumes:
      - ./data/bitmagnet:/tmp
    #ports:
      #- "3333:3333"
      # BitTorrent ports, wont work
      #- "3334:3334/tcp"
      #- "3334:3334/udp"
    restart: unless-stopped
    environment:
      - LOG_LEVEL=debug
      - POSTGRES_HOST=postgres
      - POSTGRES_PASSWORD=postgres
      - REDIS_ADDR=redis:6379
      - SOCKS_PROXY=<IP_ADDRESS>:9050
    command:
      - worker
      - run
      - --keys=http_server
      - --keys=queue_server
      - --keys=dht_crawler
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy

Describe alternatives you've considered
I figured out how to route this over a vpn but not over tor. What I did was

create a wireguard container with mullvad
set this container as "network mode container" and the container the wireguard mullvad container
However you cannot expose ports.

Additional context
Im not sure open ports are needed other than performance as the other end would need an open port to serve metainfo to us.

Hi, I'm not sure Tor integration is feasible, and is likely to be prohibitively slow. See https://blog.torproject.org/bittorrent-over-tor-isnt-good-idea/ or Google "BitTorrent over Tor" to find a ton of articles like this. In any case DHT uses UDP which isn't possible over Tor so this would just be for the peer protocol bit that downloads the meta info.
I DO want to make it easier to route through something like GlueTun, this might just be a case of providing an example docker-compose people can use.

Ah, sorry for the miscommunication, when I said "route everything over tor" I meant this container's public connections, not the entire download stack from bitmagnet -> arr -> client. I want to crawl dht files over tor only, however I would use a seedbox for the downloads. Crawling the network isnt that much bandwidth and the onion network is better setup for this. I think maybe 10-20KB per file, so even a million files would only be a few GBs.
A vpn / seedbox will usually ignore a few dmcas before banning you, but after running this only a hour I was listed for over 100 files on iknowwhatyoudownload.com and I think a overzelous copy protection agencies would send a letter on a metainfo req instead of waiting for you to download from them. However, they might actually wait to get data back from you before firing off a love letter. I dont know.
However, do you get abuse reports from running this? I may be overly concerned.
Can I contact you over email? There is something private I want to say.

I'm not sure I understand what kind of integration you have in mind, but for any part of the crawler to run over Tor would likely incur a significant overhead and might well be viewed as abuse of the Tor network. It's more about number of requests than size - the crawler might be making thousands of requests per minute. That said, I'm no Tor expert and if someone can come up with a workable PoC I'd be happy to be proven wrong.
Sure, feel free to get in touch at<EMAIL_ADDRESS>

There's a gluetun example here now @Motophan https://github.com/bitmagnet-io/bitmagnet/blob/main/docker-compose.yml


Langchain SQLDatabaseToolkit providing incorrect results. it's faking the results using  the top k rows

System Info
Here chain example:
Thought:I can query the 'information_enquiry' table to find out who is assigned to a job.
Action: SQL Query
Action Input: SELECT  name FROM  nformation_enquiry WHERE job_id = '123'
Observation: SQL Query is not a valid tool, try one of [sql_db_query, sql_db_schema, sql_db_list_tables, sql_db_query_checker].
Thought:I made a mistake in the action. I should use the sql_db_query tool to execute the SQL query.
Action: sql_db_query
Action Input: SELECT name FROM customer_information_enquiry WHERE job_no = '123'
Observation:
Thought:I have the information about the person assigned to the job.
Final Answer: The job was assigned to Helena.     --> this is wrong , it faked the first row
I have instructed prompt :
Begin!
Question: Who is assigned to 123?
Thought: I need to find the status of a specific job.
Action: SQL Query
Action Input: SELECT name FROM information_enquiry WHERE job_no ='123'
Observation:
Thought: I don't have the information about the job status.
Final Answer: information not found
    but it's not working

Who can help?
No response
Information

[ ] The official example notebooks/scripts
[ ] My own modified scripts

Related Components

[ ] LLMs/Chat Models
[ ] Embedding Models
[ ] Prompts / Prompt Templates / Prompt Selectors
[ ] Output Parsers
[ ] Document Loaders
[ ] Vector Stores / Retrievers
[ ] Memory
[ ] Agents / Agent Executors
[ ] Tools / Toolkits
[ ] Chains
[ ] Callbacks/Tracing
[ ] Async

Reproduction
llm = ChatOpenAI(model=os.getenv("OPENAI_CHAT_MODEL"), temperature=0)
sql_toolkit = SQLDatabaseToolkit(db=db, llm=llm)
sql_toolkit.get_tools()
sqldb_agent = create_sql_agent(
llm=llm,
toolkit=sql_toolkit,
agent_type=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
verbose=True,
handle_parsing_errors=True
)
formatted_prompt = updated_prompt.format(question=query)
result = sqldb_agent.run(formatted_prompt)
sqldb_agent.run(updated_prompt.format(
question="Who is assigned to the job 123"))
Expected behavior
it should return 'information not found'

What is the recommended approach to update the 'run' method in the following code snippet:
class CustomSQLDatabase(SQLDatabase):
def run(self, command: str, fetch: Union[Literal["all"], Literal["one"]] = "all") -> str:
result = self._execute(command, fetch)
res = [
tuple(truncate_word(c, length=self._max_string_length) for c in r.values())
for r in result
]
if not res:
return "information not found"
else:
return str(res)
This current implementation relies on external functions and produces errors.

This implementation can at times produce parsing errors, which can be influenced by the given prompt and the specific instructions provided. To create a more effective approach for crafting a prompt that aligns with predefined guidelines and includes tailored examples to minimize parsing errors, consider using the 'handle_parsing_errors=True' option, even though I am currently experiencing parsing errors."


chore: new release

Fixed

#633
#640
#649
#653

Chore

#652
#654

Hey team, I was checking the ticket regarding the fonts for mmi and the claim app and I think there is an issue with mmi (if not let me know if this is expected)
Checking the comments on the ticket by Franco and Lili I see that the expected fonts are the ones seen on the right MMI PR whereas on the release PR I see only the ones on the left MMI PR

Also, something minor but this copy to clipboard button does not seem to work

Un abrazo

Release looks good to me.
Cheers!


Add Material theme

https://github.com/Emiller88/emacs-doom-community-themes/pull/2/files

@ema2159 what else needs to be done to get Material Theme supported?

@tam5 I think the colors of that specific PR were not right


fix(build): Fix new nightly clippy lints and cargo feature resolution

Motivation
This PR fixes a cargo resolver version warning, and a bunch of new nightly clippy lints.
The new cargo resolver detects:

a missing tokio feature in zebra-state and zebra-network, and
missing dev dependencies in zebra-node-services.

This PR fixes those as well.
Review
This is a low priority cleanup.
This PR currently contains a copy of PR #6813, that will disappear from the diff once it merges.
Reviewer Checklist

[ ] Will the PR name make sense to users?

[ ] Does it need extra CHANGELOG info? (new features, breaking changes, large changes)

[ ] Are the PR labels correct?
[ ] Does the code do what the ticket and PR says?

[ ] Does it change concurrent code, unsafe code, or consensus rules?

[ ] How do you know it works? Does it have tests?

@mergifyio update

This PR failed when merged with PR #6813 due to a missing feature in that PR.
I put the whole of PR #6813 in this PR (it's about to merge to main anyway), then added the missing feature.

@mergifyio refresh

@mergifyio update


Does not work correctly with Chrome 41

When Chrome 41 is set to open with a new tab page, newtab-reloaded does not show.  It does show if you then open a new tab though.
Also, when Chrome 41 is set to open several pages, the first click on new tab does not show newtab-reloaded, but the 2nd click on new tab does show newtab-reloaded (sometimes...this behavior might be inconsistent).

What operating system and version of this extension are you using? Can you try to update to the new 0.1.3 version, or attach a screenshot or console log when it does not work?

I reverted the recent fixes and published 0.1.4 to the web store.

I was using 0.1.3.  0.1.4 seems to fix the issue.


fix(serve): port regex for url without trailing slash

If the serve url doesn't include a trailing slash the command fails.
New regex matches only the port digits without leading : and trailing /.
Also changed possible length of port to 2 - 4 digits to also enable non 4 digit ports.

Thanks for the PR!

Fix included in 2.0.1.


Module Displayed, Sheet Not

Hi, the module seems to be loading okay no errors to be seen in the dev console but the sheet is not loading as can be seen on the following image

@jonnyscott10 do you mind passing what your configuration looks like for the module? If your sheet data is private you can leave out the link otherwise I can try it out to see what's going wrong.

{
module: "MMM-GoogleSheets",
header: "Schedule",
position: "top_left",
config: {
url:"https://script.google.com/macros/s/AKfycbyOLx0LB8ro-g8RiW4LAf7fsCbWeKgUUHfOSAp5wCz_kkVElimZ/exec",
sheet: "Sheet1",
range: "A1:B10",
updateInterval: 1, // minutes
},

@jonnyscott10 it looks like you have a permission issue. When I click on your link, it requires a login (which it should not). If it requires a login when you click on it, that means it would also require a login when it is hit by magic mirror.
You need to remove the permissions of your script to get it to work. My best guess without seeing the way you set up the file is you missed this step in the installation procedures:
In the menu that pops up, change "Who has access to the app:" to Anyone, even anonymous
Without this step, it would require login anytime you hit the link. With this step, anyone can run it just by hitting the link.
Let me know if that works / doesn't work for you.

Hi, yes it seems it was set to public but not as me as a user, the module now shows undefined and I am currently researching this issue, if you want to have a look now you should be able to look at the script without logging in

@jonnyscott10 when I run magic mirror with your config it works fine. Are you still having issues?

@ryan-d-williams it currently is still showing undefined on the module

@ryan-d-williams it currently is still showing undefined on the module

I'm also having this issue.

@kalecream can you please post the config you are using?

@jonnyscott10 I have tried that exact config again and it is working fine. Can you try making sure you config matches what you have above? If that exact config is working for me and not for you it is likely an issue with your setup.

{
			module: "MMM-GoogleSheets",
			header: "Google Sheets",
			position: "top_center",
			config: {
				url: "[redacted]",
				sheet: "Dashboard",
				range: "C8",
			}
},

However, I'm sure the problem isn't our configuration, since we are getting this error (click on image below) from the google script library. It keeps pointing back to line 11 of the code from my last message.

@kalecream clicking on the link it provides should provide an error. This is because you are not providing the sheet and range to the url, so it errors. If you want to have the link not produce an error, you can try the link with this format:
url?sheet={sheetname}&range={range} so for example, the url from @jonnyscott10 above would be
https://script.google.com/macros/s/AKfycbyOLx0LB8ro-g8RiW4LAf7fsCbWeKgUUHfOSAp5wCz_kkVElimZ/exec?sheet=Sheet1&range=A1:B10
Notice that if you click that link, you will get data as expected.
If you try to reformat your url with the correct format, what does it return? I cannot try it for you as you have not provided your link (totally find if it has sensitive data, you will just have to do the debugging steps yourself)

@kalecream also note that the module does this url formatting for you automatically based on the config file. What I said above is just for you to see if the url itself is working or not, you do not need to change your module at all.

@ryan-d-williams Firstly excellent work on this, or at least so far and potentially.
I to had the line 11 problem but it turned out im dumb and put the book name not the sheet name.
It all works if i type the url in browser with the range and sheet parameters but not in the module.
I see the message
missing required parameter:url
missing required parameter:sheet
missing required parameter:range

i have reloaded module and copied and pasted a few times to check for errors, what am i missing?
Thanks in advance.


Bump Sarama to v1.41.0 and handle consumer state

What changed?
Bump Sarama to v1.41.0 and handle consumer state. When either the messages channel is closed or the session context is done, the ConsumeClaim func should break out of the for loop and return.
Note: the module path has changed to github.com/IBM/sarama since ownership transitioned away from Shopify

Why?
Pulling in latest updates and new module path. Also correcting consumer group implementation to handle rebalance states.

How did you test it?
go test --failfast ./...

Potential risks
If user has not previously configured a kafkaVersion setting that is equal to or older than their target server then they might see incompatible protocol attempted, as Sarama now supports a more complete set of versions

Release notes
N/A

Documentation Changes
N/A

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Closing this one as part of old PR cleanup effort. @neil-xie will evaluate the risks of this upgrade and apply in a separate PR with proper tests if we see fit.


Cache table sort options

When a user selects a specific sort column for any table with sorting enabled, cache that selection so subsequent page visits the selections are the same.

@Jonesy @elisafw-ux
Say for the Products page, is the intention that the sort order be applied to all products when the sort is selected in one product?  Looks like the sort order is preserved just for that particular product.. not sure if that was the intention?

@ikethecoder the sort order applies only to that particular product/table


(Info) Works with Fujifilm X-S10

Thanks for the nice Code !
I ordered at "banggood" for 25$ my m5Stick C-Plus and connected it with my Fujifilm X-S10. Everythings works YEAH
Thank you!

@dimitrij2k
Thanks for the report, much appreciated!
I've had additional success with other Fujifilm cameras, you have motivated me to update the README to include them.


Copying text from editor should not include formatting

VS Code Version: all
OS Version: windows all

Steps to Reproduce:

copy some text from vscode
paste in outlook email editor

Does this issue occur when all extensions are disabled?: Yes/No
yes

/question
If you prefer to copy without formatting you can turn it off with "editor.copyWithSyntaxHighlighting": false.


Check if file is already open?

Hi,
I've been going trough the docs but couldn't find an answer to this:
what happens if I call spiffs_open() twice with the same path+filename ? Do I get two different filehandles to the same file?
Is there any way to prevent or detect this? I have a situation where a file might or might not be open. If it is not open, it should be opened, if it is open, the existing filehandle should be used.
If course, I can write some code to check the names of open files, but if there is an easy way to do this with SPIFFS, that would have my preference..
Thanks!

You get two different filehandles to the same file, yes. There's nothing implemented to detect or prevent this - in fact, this is the way it is meant to work.

Thanks!
I'll see if I find a way to implement something. Perhaps add a flag to spiffs_open that tells it to return an existing filehandle if the file is already open,  and do the regular stuff it it's not open yet.  I'll see.   Need to find some time first :-)


Registration error: 403: Forbidden: Other Error

Describe Your Problem:
I keep receiving the message "Registration error: 403: Forbidden: Other Error" while going through the process under the Homebridge Terminal.
Any idea here?
Appreciate your help. Thanks!
Logs:
pi@homebridge:/var/lib/homebridge $ homebridge-playstation-login
Discovered device: {
  address: { address: '<IP_ADDRESS>', family: 'IPv4', port: 9302, size: 162 },
  hostRequestPort: 997,
  extras: {
    statusLine: '200 Ok',
    statusCode: '200',
    statusMessage: 'Ok',
    status: 'AWAKE'
  },
  discoveryVersion: '00030010',
  systemVersion: '06020004',
  id: '70662AA0B52F',
  name: 'Playstation',
  status: 'AWAKE',
  type: 'PS5'
}
Authenticate to Playstation? (y/n) y
Connecting to <70662AA0B52F>...
Registration error: 403: Forbidden: Other Error

Discovering next device...

Plugin Config:

    "pollInterval": 5000,
    "platform": "playstation"
}

Plugin Version: v1.4.2
Homebridge Version: v1.5.1
Node.js Version:
NPM Version:
Operating System:

I have the same error:
`
Discovering next device...
Discovered device: {
address: { address: '1xx.xx.xx.1', family: 'IPv4', port: 9302, size: 158 },
hostRequestPort: 997,
extras: {
statusLine: '200 Ok',
statusCode: '200',
statusMessage: 'Ok',
status: 'AWAKE'
},
discoveryVersion: '00030010',
systemVersion: '06020004',
id: 'XXXXXXXXX',
name: 'PS5-0XX',
status: 'AWAKE',
type: 'PS5'
}
Authenticate to PS5-0XX? (y/n) y
Connecting to ...
Registration error: 403: Forbidden: Other Error
Discovering next device...`
Homebridge v1.6.0
Plugin version: 1.4.2

Getting same error?

same issue

I have the same error. "Registration error: 403: Forbidden: Remote is already in use."

Hi, everyone. I also encountered this error today (403: Forbidden: Other Error), but after digging further into it, I think it's not real. I don't have a clear set of steps to get past it, so I'll just tell you what I experienced and let you try on your own.
I followed the steps in the README to add the PS5 to HomeBridge and hit the error. I turned Remote Play off and left it alone while I ate dinner. When I came back, I was able to add the PS5 to the Home app, but it said that the code was invalid. I tried again and told it to use the camera, and it scanned the QR code off the HomeBridge main screen and added the PS5.  At that point I turned Remote Play back on and ran homebridge-playstation-login and it successfully logged into the PS5. I was in the Remote Play screen and saw the notification pop up that my PSN ID had connected via Remote Play. From then on I was able to turn the PS5 on through the Home app.
I don't know that leaving it alone for a while really did anything, but you may want to try adding it to the Home app, then reconnecting with homebridge-playstation-login and seeing if it works. At worst it should be a no-op. At best you'll get past this error.
I wish there was a way to put it into sleep mode instead of powering it off, but 🤷 at least now I can turn on the TV, the Govee lights, set to HDMI 3 and turn on the PS5 all with a single voice command.


Is it possible to resolve Redis Cluster nodes from the config CNAME given by AWS Elasticache?

AWS provides a CNAME with all nodes in cluster via round-robin DNS, is it possible to add DNS based cluster creation adapter?

I have the same problem, when creating cluster, it breaks down to :
RedisServer(foo-bar-0001-001.baz.cache.amazonaws.com,6379,None,None)
RedisServer(foo-bar-0002-001.baz.cache.amazonaws.com,6379,None,None)

ClusterSlot(0,8191,ClusterNode(172.x.y.z,6379,sj5s7fsa),List())
ClusterSlot(8192,16383,ClusterNode(172.x.y.z,6379,owr83r3210),List())

And it doesn't connect because of :
https://github.com/etaty/rediscala/blob/master/src/main/scala/redis/RedisCluster.scala#L65
Any idea?

Realized this could be solved with following in code for now. Actual RedisCluster should likely do internally though
val addr = InetAddress.getAllByName("AWS-CNAME").map( x => RedisServer(x.getHostAddress, 6379))
  val redis = RedisCluster(addr)

I'm getting the same error, though InetAddress doesn't solve it.
I'm running Redis in a docker container and trying to connect to it from another container. In that container I specify the redis container (called redis) and the port.
However the find call that @l15k4 mentioned above fails because the master node is being resolved with an ip address of <IP_ADDRESS> while the connections are pointing to redis.
Could not find `ClusterNode(<IP_ADDRESS>,7001,f6485677229bb9e36df71dfac22efa1c451f6cdd)` in `Map(RedisServer(redis,7002,None,None) -> RedisConnection(Actor[akka://application/user/RedisClientPool-$a#464875078],scala.concurrent.stm.ccstm.CCSTMRefs$BooleanRef@da95dd21))`
Could not find `ClusterNode(<IP_ADDRESS>,7002,b54769ab763dd8a5288d48bfc1497fcaf55e5965)` in `Map(RedisServer(redis,7002,None,None) -> RedisConnection(Actor[akka://application/user/RedisClientPool-$a#464875078],scala.concurrent.stm.ccstm.CCSTMRefs$BooleanRef@da95dd21))`
Could not find `ClusterNode(<IP_ADDRESS>,7000,ae1ede2bf527ce23b286f7c2a8e1b430e4fa6d2e)` in `Map(RedisServer(redis,7002,None,None) -> RedisConnection(Actor[akka://application/user/RedisClientPool-$a#464875078],scala.concurrent.stm.ccstm.CCSTMRefs$BooleanRef@da95dd21))`

(had to fork the lib to add those logs statements)
Resolving the hostname (like @jreichhold did) did not solve the problem since redis gets resolve to the container's IP address in the network (which is not <IP_ADDRESS>).


HTML Snippets Combine

VSCode Version: Code 1.14.2 (cb82febafda0c8c199b9201ad274e25d9a76874e, 2017-07-19T23:34:09.706Z)
OS Version: Windows_NT ia32 10.0.15063
Extensions:

Extension
Author (truncated)
Version

Bookmarks
ale
0.15.2

vscode-color
ans
0.4.5

theme-verdandi
be5
1.5.0

color-info
bie
0.5.0

path-intellisense
chr
1.4.2

vscode-eslint
dba
1.2.11

vsc-material-theme
Equ
1.0.2

beautify
Hoo
1.1.1

vscode-gutter-preview
kis
0.10.0

HTMLHint
mka
0.3.3

color-highlight
nau
2.3.0

material-icon-theme
PKi
2.1.0

vscode-icons
rob
7.12.0

code-settings-sync
Sha
2.8.2

stylelint
shi
0.28.0

ayu
tea
0.4.0

(16 theme extensions excluded)

Steps to Reproduce:

Create an HTML document.
Use the '!' emmet snippet to create a boilerplate.
In the body section create a form element and enter key.
Insert an input snippet.
Tab to edit the type attribute.
Edits the attribute and outer element.

Expected behaviour: edit only the attribute and keep using tab inside the snippet.

Reproduces without extensions: Yes

Also repros without the form element

Type ! and select the suggested emmet expansion
Tab till you reach the body
type input and select the suggested emmet expansion (this expands to <input type="${2:text}">"
Start typing to edit the type attribute, typed text appears at the end as well

@jrieken An extra cursor is being placed at the end causing the edits to the placeholder appear in 2 places. Must be due to nested snippets?
Doesn't happen if you escape snippet mode after the first snippet gets expanded

@jrieken An extra cursor is being placed at the end causing the edits to the placeholder appear in 2 places. Must be due to nested snippets?

Yeah, must be the merging of snippets... I'll take a look

Ok, I can reproduce, both placeholders (of the nested snippet) get the same id assigned. We insert <input type="${2:text}"> at placeholder 8 which is implemented as merging the second placeholder into the original snippet as 8.2. However, the final placeholder is getting the same index because it assumes the indices of placeholder start counting at 1 and don't skip a number...


Why is the Game Version 1.18.2 when the mod works perfectly on 1.19.2?

I noticed that you could download 1.19 versions. So I tried it and it worked. Only thing is config crashed my game but I just edited it manually. Can you fix this crash and set Game Version to 1.19.2 cus it totally works on there

The crash in 1.19.2 has now been fixed by updating MidnightConfig to the latest version.


Document full text indexes for PostgreSQL

This is still at the design stage, just creating an issue to track it.
Migrations issue for this is here.
Current Concepts page for full text indexes is here (MySQL and MongoDB).

Closing this old issue. Will use #3350 to track the state of full text search and open new issues as needed.


Autoregressive flow instead of WaveGlow?

Hi, thank you for this amazing idea. It is really nice :).
I was wondering if it would be possible to replace the WaveGlow model by some more expressive flow, for example by autoregressive WaveNetor WaveFlow? In WaveGlow, the audio/spectrograms are directly encoded into samples from some gaussian distributions and an external encoder can be used to evaluate likelihood of the sample. WaveNet instead does a scale + shift transformation of a standard normal distribution based on previous audio timesteps. The use of a model such as WaveFlow/WaveNet could boost expressivenes of the system and lower the number of necessary params, but I was not yet able to figure out how it could be integrated in your framework. Did you consider similar options when you wrote the paper?

You can use autoregressive flows such as WaveFlow or WaveNet as vocoders of Glow-TTS.
WaveFlow and WaveNets are both conditioned by input mel-spectrograms. Therefore, you can predict a mel-spectrogram through Glow-TTS, and then feed it into those models.
Actually there was a discussion about how to integrate Tacotron 2 and WaveNet (https://github.com/NVIDIA/tacotron2/issues/52). The point was the preprocessing for mel-spectrograms of both models should be same.
My concern is that fast sampling speed of Glow-TTS would become useless because autoregressive vocoders are significantly slow at inference compared to our model.
Despite of the slow speed, you still could get good quality mel-spectrograms from Glow-TTS.

@jaywalnut310 Thanks for response :). It is clear to me that it i possible to use different vocoders. But what I mean is a bit different. Could the Glow model that transforms the latent variables to spectrograms be replaced by eg. WaveFlow or a Gaussian WaveNet? The process would still be very similar, but the latent variables from text encoder would be transformed into spectrograms with an autoregressive flow instead. I was also wondering, did you by any chance experiment with transforming the latent space directly to waveforms instead to spectrograms? The WaveGlow model could be capable of this. It would make the model truly end to end (if we do not count the grapheme-to-phoneme transformation), because no vocoder would be needed.

TLDR:  That is up to you: 1) learned prior distribution without conditioning on vocoders as Glow-TTS or 2) standard normal prior distribution with conditioning on vocoders as Flow-TTS.
@LordOfLuck  Oh sorry, now I understand what you meant.
Although I haven't tried text-to-wave (is that right?) experiments, I think it sounds reasonable!
Conceptually, any audio representation such as raw waveforms and mel-spectrograms can be generated from latent variables from text encoder.
So my answers are:

Yes, I think it can be trained in end-to-end with powerful vocoders as you mentioned.
But you cannot use local conditioning layers in those vocoder; you can only train the prior distribution from text encoder with monotonic alignment search.

So, I hope you make progress on Glow-TTS though :), I think the Flow-TTS work (https://ieeexplore.ieee.org/abstract/document/9054484) would be more familiar with you for that end-to-end learning case. It also uses standard normal distribution as prior distribution.

TLDR:  That is up to you: 1) learned prior distribution without conditioning on vocoders as Glow-TTS or 2) standard normal prior distribution with conditioning on vocoders as Flow-TTS.
@LordOfLuck  Oh sorry, now I understand what you meant.
Although I haven't tried text-to-wave (is that right?) experiments, I think it sounds reasonable!
Conceptually, any audio representation such as raw waveforms and mel-spectrograms can be generated from latent variables from text encoder.
So my answers are:

Yes, I think it can be trained in end-to-end with powerful vocoders as you mentioned.
But you cannot use local conditioning layers in those vocoder; you can only train the prior distribution from text encoder with monotonic alignment search.

So, I hope you make progress on Glow-TTS though :), I think the Flowtron work (https://github.com/NVIDIA/flowtron) would be more familiar with you for that end-to-end learning case. It also uses standard normal distribution as a prior distribution.

Thanks! Especially for the Flowtron reference. They are pretty much doing what I meant except that they do the local conditioning on text via attention, while with GlowTTS the text conditioning would be done as you said with the trainable priors. :) I like the trainiable prior idea, it is similar to how the original Glow model can be conditioned on predicted classes.

Yes, I love your idea, and also love to see how future studies will tackle into "real" end-to-end training!
Closing the issue at this point.


[Bug]  query values with the same primary key

Search before asking

[X] I searched in the issues and found nothing similar.

Paimon version
0.8-snapshot
Compute Engine
spark 3.1
flink-1.17.2
Minimal reproduce step
A large online table with frequent data writing and updates, with a data volume of approximately 600 million. CDC synchronizes to the payment table, and after running for a period of time, it is found that the data in the table has duplicate primary keys (using SparkSQL and FlinkSQL queries will result in duplicate data). After executing compact, the same primary key will be merged out

What doesn't meet your expectations?
The query result does not have duplicate primary keys
Anything else?
No response
Are you willing to submit a PR?

[X] I'm willing to submit a PR!

Thanks @zhourui999


arm (32-bit) release contains a lib/client and lib/server directory for Java 17. While all other platforms only contain a lib/server.

I don't know if this is the right place to report this. But is it expected that the arm (32-bit) release contains a lib/client and lib/server directory. While all other platforms only contain a lib/server.
I noticed it because both directories contain a libjvm.so and this confused bitbake.
Originally posted by @bachp in https://github.com/adoptium/adoptium/issues/73#issuecomment-930274451

I just checked and also the 32 bit arm build of OpenJDK 11 contains the lib/client directory. I'm currently checking if deleting the client folder has any negative effects.

Ref reference, it was added via this issue and the associated PR:


Wait for CI before assigning a reviewer

We had a discussion about the review queue on Zulip, and a simple actionable point that came out of it is: rustbot should wait for CI to turn green before assigning a reviewer, to avoid the reviewer needing to remember to check back on the PR later.
As a corollary, it would be nice if @rustbot ready also waited for CI to turn green. In both cases rustbot could say "waiting for CI to turn green" (or set a waiting-on-CI label) to inform the author of what's happening.

Link to the suggestion on Zulip.
Just to understand how that could work:
a) Apply a waiting-on-ci (or ci-running) on a new PR and when the CI completes, remove that label. Triagebot should react to the event "removed waiting-on-ci label" by throwing a dice and find a reviewer.
or
b) the triagebot should keep an internal state of the PR and choose the reviewer when the CI becomes green
Option (a) a bit easier to implement, but in both cases it should be clarified if there a hook of any kind to be notified that "CI for PR #000 is now green".

It is possible to listen for webhooks related to check suites assigned to a branch/PR, and find out that it all the workflows of the check suite were green, but that is a bit involved (new bors works like this). I'm not sure if there's a simpler way.

Maybe this is simpler.


WebChat plugin location misaligned

Screenshots
In NLU

In Dialogue

Hi, I think it's because your window is smaller than the minimum resolution we support, either display the app in a larger window, or use ctrl (cmd on Mac)  and - key to zoom out in the window. Let me know if this corrects your issue


python312: 3.12.0b4 -> 3.12.0rc2; python38: 3.8.17 -> 3.7.18; python39: 3.9.17 -> 3.9.18

Description of changes

python312
https://docs.python.org/3.12/whatsnew/changelog.html#python-3-12-0rc2
Fixes: https://github.com/advisories/GHSA-4j9r-82g6-9mj3

python38
https://docs.python.org/release/3.8.18/whatsnew/changelog.html
Fixes: https://github.com/advisories/GHSA-4j9r-82g6-9mj3

python39
https://docs.python.org/release/3.9.18/whatsnew/changelog.html
Fixes: https://github.com/advisories/GHSA-4j9r-82g6-9mj3

Things done

Built on platform(s)

[x] x86_64-linux
[x] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

nixpkgs-review reveals no new failures, macos 10.15

Result of nixpkgs-review pr 254221 run on aarch64-linux 1

3 packages marked as broken and skipped:

cjdns
ntfy
ntfy.dist

8 packages failed to build:

easyabc
friture
friture.dist
google-cloud-sdk-gce
klipper-firmware
klipper-flash
transifex-client
transifex-client.dist

31 packages built:

blackmagic
crossfire-server
edgetx
gcc-arm-embedded
gcc-arm-embedded-11
google-compute-engine
google-compute-engine.dist
libsForQt5.kdev-python
monoid
opentx
osmscout-server
proxmark3
python312
python312.debug
python38
python38.debug
python38Full
python38Full.debug
python39
python39.debug
python39Full
python39Full.debug
qmk
qmk.dist
spidermonkey_78
spidermonkey_78.dev
taler-exchange
taler-merchant
valhalla
zeroad
zeroadPackages.zeroad-unwrapped

@mweinelt I'm sorry to bother you, but could you please review this PR?

@ofborg build python38.tests
@ofborg build python39.tests


For Android Platform : Update Tts.java file

Hi ,
i found an issue while taking a build after added a plugin & fixed the issue using below changes !
Please update a speech to text files .
For your reference,
Add two methods in Tts.java file
@Override
public void onTTSStopped() {
Log.d(TAG, "### onTTSStopped ###");
}
@Override
public void onTTSError(int statusCode) {
    Log.d(TAG, "### onTTSError: "+statusCode+" ###");
    this.synthesizeContext.error("Synthesize Failed");
}

TextToSpeech.sharedInstance().initWithConfig(tConfig,this);

Hi @prameshkumar , thanks for reporting this, there are some out-of-date code, and I fixed it. And these two methods are already there in the Tts.java: public void onTTSStopped() and public void onTTSError(int statusCode), and TextToSpeech.sharedInstance().initWithConfig(tConfig,this); is updated in the latest commit.


how about  the schedule of adding FIPS_drbg_health_check&FIPS_drbg_set_check_interval?

I See there:https://www.openssl.org/docs/OpenSSL300Design.html
will these two API to de added?

I don't think we're planning on adding those two API calls. #11752 adds an ability to run the self-test KATs for the whole module (i.e. not just the DRBG) on demand via the function OSSL_PROVIDER_self_test.

I don't think we're planning on adding those two API calls. #11752 adds an ability to run the self-test KATs for the whole module (i.e. not just the DRBG) on demand via the function OSSL_PROVIDER_self_test.
SO Can the 3.0 design document be modified?and SELF_TEST_post() can use to test on demand?
i see some Comments on the above of this function:
/* This API is triggered either on loading of the FIPS module or on demand */

#11752 adds an ability to run the self-test KATs for the whole module (i.e. not just the DRBG) on demand via the function OSSL_PROVIDER_self_test.
SELF_TEST_post() can use to test on demand?
i see some Comments on the above of this function:
/* This API is triggered either on loading of the FIPS module or on demand */

@mattcaswell is correct.  We're not supporting these two calls.  In what way should the design documentation be updated.  I don't remember these two being mentioned.
The global self test on demand is runnable and, due to changes in the FIPS policies, they should be far faster than perviously.  They always run the DRBG KATs.  A pull request that adds selectable granularity to the FIPS on demand self tests would likely be considered for inclusion.
FIPS doesn't mandate rerunning the DRBG KATs regularity, so this won't be supported.

In what way should the design documentation be updated. I don't remember these two being mentioned.

you can see the URL i methioned above,it's from the OpenSSL 3.0 Alpha4 Release's design document.

Marking as inactive, to be closed at the end of 3.4 dev barring further input

Closing, this isn't relevant anymore.


utilize util.promisify?

Will bluebird start using util.promisify?

No, since it's slower, less versatile and wouldn't reduce allocations.

Full disclosure - I worked on util.promisify

Chad Scira liked your email
Spark by Readdle

Will bluebird start using util.promisify?

It would be just as good an idea as for Bluebird to become based off ES6 Promise :smile:


wttr.in is down

Curl requests for some cities (Bremen, Berlin) are working, but not for others (Salt+Lake+City)
x@y:~$ curl wttr.in/Salt+Lake+City

Sorry, we are running out of queries to the weather service at the moment.
Here is the weather report for the default city (just to show you what it looks like).
We will get new queries as soon as possible.
You can follow https://twitter.com/igor_chubin for the updates.
======================================================================================

Any ETA on when we might get a fix?

It seems to be a problem with the data source. Let's wait until tomorrow morning

As of 8 PM PDT, it appears to be back up.


False positive in Rails/FilePath

On master branch (6c3ea4753bbb9f1ffb0f2ee60400fd0b432ede3f) I’m seeing Rails/FilePath complain about this code:
<<-FOO
  1
  /
FOO

test.rb:1:1: C: Rails/FilePath: Please use Rails.root.join('path', 'to') instead.
<<-FOO
^^^^^^

With git bisect I found that the false positive was introduced in 707c9a85c1d99bfee8a10ac404abf6a14d6a88f0 – cc @koic

Expected behavior
No offense should be found.
Actual behavior
An offense found for Rails/FilePath.
Steps to reproduce the problem
See above.
RuboCop version
$ rubocop -V
0.52.1 (using Parser <IP_ADDRESS>, running on ruby 2.4.2 x86_64-darwin15)
$ git rev-parse @
6c3ea4753bbb9f1ffb0f2ee60400fd0b432ede3f

Thanks for the feedback.
I reproduced this false positive. I will investigate.

I just found another false positive which may be related.
'' \
'/'

test.rb:1:1: C: Rails/FilePath: Please use Rails.root.join('path', 'to') instead.
'' \
^^^

One more:
"#{42}/"

test.rb:1:1: C: Rails/FilePath: Please use Rails.root.join('path', 'to') instead.
"#{42}/"
^^^^^^^^


Update protocol 17 to implement room customization

Add these modifier inside 17 protocol:

Player number allows in a room
Player velocity
Player Shoot frequency
Ennemies velocity
Ennemies shoot frequency
Time acceleration
Update RFC documentation for protocol 17

Add missing documentation


[Enhancement] Unified Catalog support Paimon

Why I'm doing:
Make unified catalog support paimon data sources.
example:
CREATE EXTERNAL CATALOG unified
PROPERTIES
(
    "type" = "unified",
    "unified.metastore.type" = "hive",
    "hive.metastore.uris" = "thrift://localhost:9083",
    "paimon.catalog.warehouse" = "hdfs://<IP_ADDRESS>:9000/paimon",
);

What I'm doing:
Fixes #issue
What type of PR is this:

[ ] BugFix
[ ] Feature
[x] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Does this PR entail a change in behavior?

[ ] Yes, this PR will result in a change in behavior.
[x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

[ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
[ ] Parameter changes: default values, similar parameters but with different default values
[ ] Policy changes: use new policy to replace old one, functionality automatically enabled
[ ] Feature removed
[ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

[x] I have added test cases for my bug fix or my new feature
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

[ ] This is a backport pr

Bugfix cherry-pick branch check:

[x] I have checked the version labels which the pr will be auto-backported to the target branch

[x] 3.3
[x] 3.2
[ ] 3.1
[ ] 3.0
[ ] 2.5

please rebase main

ignore backport check: 3.3.6


How do you only record requests that try to access a certain port?

I tried to use --proxy-all="http://localhost:3000" to capture anything my angular app tries to get from my rails app but instead it proxied stuff from port 3000 to port 8080 which wasn't what I was looking for.
I just want my api calls which are on port 3000 to be recorded.
Hope that makes sense.

As per the README, please can you post this (with some more details and code) on the mailing list.


Export/extract validation error with ISJSON

SqlPackage or DacFx Version: 162.1.167
.NET Framework (Windows-only) or .NET Core: both
Environment (local platform and source/target platforms): SQL 2022

Steps to Reproduce:

Create a database with following schema in SQL 2022:

CREATE TABLE tb_json([htmlMetadata] NVARCHAR(MAX))

ALTER TABLE tb_json
ADD CONSTRAINT json_constraint
CHECK ([htmlMetadata] IS NULL OR isjson([htmlMetadata],OBJECT)>0);

Export will fail with this error

*** Error extracting database:Validation of the schema model for data package failed.
Error SQL71501: Error validating element [dbo].[json_constraint]: SqlCheckConstraint: [dbo].[json_constraint] has an unresolved reference to object [dbo].[tb_json].[OBJECT].

DacFx is not understanding the OBJECT keyword in the ISJSON function, thinking it's a column reference.
(DacFx/SqlPackage/SSMS/Azure Data Studio)

This bug is still present when using sqlpackage.exe version <IP_ADDRESS>.   Is there a newer version coming out that this fix is included in?   I tested this using SSDT projects with compatibility level set to SQL 2022. Additionally, I attempted using Target Platforms SQL Server 2022 and then Microsoft Azure SQL Database to determine if either target platform setting worked correctly.  Visual Studio Enterprise 2022 was updated to version 17.11.5 running on Windows 11.

@mattshanaman, When you mention testing with an SSDT project, are you utilizing its dacpac/bacpac to test in the sqlpackage.exe application? This fix is exclusive to SqlPackage; you can test directly by extracting/exporting your database from the command prompt or by using the DacFx NuGets. The SSDT 17.12 Preview 4 version includes this fix, whereas earlier versions still employ DacFx 162.3.xxx.

Thank for you the additional information @ssreerama!  I will test that...


If my esp32 reboot I only received the first retained message

if you subscribe the a topic you only get the FIRST message send from the broker. Others got wiped for whatever reason.
I have been trying to change the buffer size without effect.
Anyone knows how to fix that?
thank you!

There is only ever one retained message on a topic.
If you publish A then B, both as retained messages, B will replace A as the retained message on the topic.

Should I subscribe to multiple topics if I have more than one retained messages?

It depends what you are trying to do. If you are trying to using retained messages to queue up messages whilst the client is disconnected, that is not the right thing to do.
To do that properly you need to:

set the cleanSession flag to false in the call to client.connect.
subscribe at QoS 1 (this library doesn't support qos 2)
make sure the messages published to that topic are published at QoS 1 as well.

If that's all done correctly, the broker will queue up messages for those topics for the client to receive when it next connects and subscribes.

Ok! Thank you!!


[Question]Dummy leaf node

Can dummy node use high qc instead of nil qc?
func (p *RoundRobin) startNewViewTimeout(stopContext context.Context) {
	for {
		select {
		case <-p.resetTimer:
		case <-stopContext.Done():
			return
		case <-time.After(p.timeout):
			// add a dummy block to the tree representing this round which failed
			logger.Println("NewViewTimeout triggered")
			p.SetLeaf(hotstuff.CreateLeaf(p.GetLeaf(), nil, nil, p.GetHeight()+1))
			p.SendNewView(p.getLeader(p.GetHeight() + 1))
		}
	}
}

Closing this issue since the code in question has been rewritten.


allow mouse buttons to retain their native function if used as pie key

adding here for reference. user lingjianjun in discord says hes using right click to trigger the pie menu, and obviously this supresses the normal right click
algo could be similar to https://github.com/mmikeww/DragToScroll where a delay is done before the pie menu is shown. if the right click is released within that delay, then normal right click is sent, if that delay times out and right click is still held, then show the pie menu
i searched for this issue already but didnt find it. issue #64 seems similar but not exactly

dumbeau suggests:

Keep in mind, the center slice is also a function!  You can set the center of the pie menu to "Mouse Click", set it to Right, and disable "Drag".
This way if you right-click and release without moving the mouse out of the ring, you'll perform the original RMB click

which seems good enough for now


PropertySortOrder: use separate_groups without order

The documentation hints, that

separate_groups is only enforced if a custom order is specified via the order option.

It would be nice, if separate_groups used without order would enforce a capsulated order of groups. Every group would be sorted independently (alphabetically by default).
For example this:
video {
      height: auto;
      left: 50%;
      min-height: 100%;
      min-width: 100%;
      position: absolute;
      top: 50%;
      transform: translate(-50%, -50%);
      width: auto;
}

would become this:
video {
      height: auto;
      min-height: 100%;
      min-width: 100%;
      width: auto;

      position: absolute;
      left: 50%;
      top: 50%;
      transform: translate(-50%, -50%);
}

or this:
video {
      min-height: 100%;
      min-width: 100%;

      height: auto;
      width: auto;

      position: absolute;
      left: 50%;
      top: 50%;
      transform: translate(-50%, -50%);
}

It would be nice, if separate_groups used without order would enforce a capsulated order of groups.

I'm not sure I understand this suggestion. How would we know what to put in what group without specifying it through the order?

Okay, I'll try to improve the explanation:
Per default ungrouped properties are sorted alphabetically. If you now insert a blank line the properties are splitted in two groups (as from the view of the editor) and the order has to remain alphabetically for all properties.
The suggestion is, that both property groups are now sorted indepent alphabetically, if the separate_groups option is used without the order option.
But maybe this suggestion should use a new option...

Ah, I see what you mean now. Arbitrary groups with alphabetic ordering enforced within the groups. I don't think this is possible with the current implementation. I'm happy to accept a PR though.


Rewrite node tracking

This should be considered as a from scratch rewrite of the node tracking stuff, and not a change.  Nothing uses it, and the whole approach has changed.

Pants. There's a race condition.

It wants go 1.19.  I have a bunch of changes for that, I'll throw it on this PR too.  I don't like overloading it, but it is what it is.


How do you actually use heatmap.

I would like to use heatmap however there is no proper documentation

@bfayers Also just a heads up, threshold isn't used on the gmaps api, and maxIntensity isn't implemented in a released version yet (you're fine if you've installed from the master branch, but not if you've installed release 1.2).
The current work around I have is a script that edits the maxIntensity settings at the end of the html file after the map is drawn.

@aSipiere link ain't working mate

@sriharsha0806 Sorry the site changed the link on me. Here.

Also this repo hasn't aged well and isn't really supported anymore. The new google maps api changes also mean that you will almost always get maps with "for development purposes only" watermarks on them.
I've recently moved back to matplotlib for geospatial plots.

The API key requirements should have been solved in #23 2 years ago. I noted somewhat recently that from_geocode didn't get API key support and should have.

I've just added (in #162) documentation and examples for each of the public functions to our Wiki, which includes heatmap().
Note that the documentation currently reflects the latest commit, so there might be a few differences until we release the next version, but this should good enough to begin with.


Crashing on message.member if the author is not a guildMember (i.e. webhook bots or DMs)

Please describe the problem you are having in as much detail as possible:
A TypeError crash occurs when checking the message author's roles through message.member.roles or message.guild.member(message.author).roles in case the author is a bot created by a discord webhook. The exact type error (logged from code snippet below) is as follows:
TypeError: Cannot read property 'roles' of null
    at Client.client.on.msg (D:\UserFiles\DevProjects\testbot\app.js:11:19)
    at emitOne (events.js:96:13)
    at Client.emit (events.js:191:7)
    at MessageCreateHandler.handle (D:\UserFiles\DevProjects\testbot\node_modules\discord.js\src\client\websocket\packets\handlers\MessageCreate.js:9:34)
    at WebSocketPacketManager.handle (D:\UserFiles\DevProjects\testbot\node_modules\discord.js\src\client\websocket\packets\WebSocketPacketManager.js:102:65)
    at WebSocketConnection.onPacket (D:\UserFiles\DevProjects\testbot\node_modules\discord.js\src\client\websocket\WebSocketConnection.js:329:35)
    at WebSocketConnection.onMessage (D:\UserFiles\DevProjects\testbot\node_modules\discord.js\src\client\websocket\WebSocketConnection.js:292:17)
    at WebSocket.onMessage (D:\UserFiles\DevProjects\testbot\node_modules\ws\lib\EventTarget.js:103:16)
    at emitTwo (events.js:106:13)
    at WebSocket.emit (events.js:194:7)

The issue is caused by message.member (or message.guild.member(msg.author)) being null for webhook bots or if the bot runs in DM.
Lastly, here is as an attachment the NPM Debug log: 2017-05-09T22_28_32_710Z-debug.txt
A temporary workaround to this issue is limiting webhooks to 1 or more specific channels and excluding them from the entire client.on("Message") chain by starting with either if(message.author.bot) return; or if(!message.member) return;

Include a reproducible code sample here, if possible:
First of all create a webhook in Discord. I recommend using the website Zapier to generate webhook messages as it has a test feature that you can use to quickly generate them without having to wait for an automated trigger. Then use the following code snippet:
const Discord = require("discord.js");
const client = new Discord.Client();
const botToken = 'Some_Token_Here';
const roleToCheck = 'Some_Role_ID_Here';
client.login(botToken);

client.on("ready", () => {
    console.log("Bleep Bloop! This is Test HQ!");
});

client.on("message", message => {
    console.log(message.member);
    console.log(message.member.roles);
    try {
        message.member.roles.map(r => r.id).includes(roleToCheck)
        console.log('Role check passed')
    } catch (e) {
        console.error(e);
    }
})

Upon triggering the webhook test message in Zapier you should see the error as shown above being thrown. I have been able to reproduce this crash at a rate of 100% using only this code as well as using a fully developed bot so I am confident that it will be reproducible.

Further details:

discord.js version: 11.1.0
node.js version: 7.9.0
Operating system: Windows 10 Creators Update as well as Raspbian (Raspberry Pi 3)
Priority this issue should have – please be realistic and elaborate if possible: low

Initially I set this issue on high, however after talking with Crawl and others in #general on the server we can only draw the conclusion that taking this into account in the library can be done, but all of them kinda lead to a bad solution in the long run. Therefore I have edited the priority to low as it is something that still remains to be an issue, but requires a good amount of thought for a potential fix.

[X] I have also tested the issue on latest master, commit hash: f95f18b586069bb40ebe027c4a014a0263190641

p.s. naming issues is hard. But so is naming software in general. I think that's an excuse ;-)

I'm unsure why this is an issue with the lib.  Webhook messages are still messages and you cannot expect a webhook to have a non-null .member property as they are not guild members.

Webhooks are not GuildMembers, so member will always be null for any of their messages. It's stupidly simple to skip over them:
if(msg.webhookID) return;

I don't see any reason we should work around this in the lib. If anyone else has any opinions on the matter, continue discussing, by all means. Until then, I'm closing this.


Bump Kubernetes to v1.30.3

Description
This PR bumps Kubernetes libraries to v1.30.3

Close as replaced by #2705


[Question] How to use ferdi by itself ?

I wanted to use Franz in a self-contained (not self-hosted!) fashion. The only communication should be with the actual services. Does Ferdi require a server to work? Can it be setup as to not  require a server? ("require" as in "I have an airgapped laptop that can only connect to whatsapp servers")
If not, I would like to suggest that as a feature (maybe bundle recipes?).

We added the option to use Ferdi without an external server in v5.4.0. You should see the option to "Use Ferdi without an account" when you are on Ferdi's welcome, login or sign up screen. This will save Ferdi's options completely locally on your computer. Ferdi v5.4.0 also ships with Ferdi's complete recipe repository (see https://github.com/getferdi/recipes) build-in.
Please keep in mind that this is a relatively new feature and they might still be some bugs. Currently known bugs:

The internal database will sometimes be created in read-only mode on Windows (file at "%appdata%/Ferdi/server.sqlite"). This can be fixed by disabling write protection in the Windows Explorer.
The internal server has an invalid response on some Ferdi versions. Update to 5.4.1-beta.5 should fix this.

We advice you to enable beta updates in Ferdi's settings when using this feature.

Thanks! I got a couple of glitches that make ferdi unusable to me, tough.
Service button not working. Probably the server issues

Other than that there is heavy update check trolling...


⚠ Bofrost.de has degraded performance

In 90b883d, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1260 ms

Resolved: Bofrost.de performance has improved in 7c65370 after 26 minutes.


Should linter consider .yml incorrect in favor of .yaml?

http://www.yaml.org/faq.html says "please use .yaml whenever possible". Would a PR to make ansible-lint warn when files have .yml rather than .yaml extensions be welcome?

possibly relevant: https://github.com/willthames/ansible-lint/pull/69

most ansible docs use .yml. This is the first time I have heard .yaml is proffered by anyone

I won't be making (or including) a rule for this - I agree with @smiller171, even though I'm happy to incorporate #69.
But it should be a simple enough task to create (and share if you like) your own rules though - if there's anything in ansible-lint that is preventing that working as a custom rule, let me know.


Mailman errors if address already exists for another user

If a user is being sync'd and their address is listed as belonging to another user, the api sync errors out. Since we know the person owns the address, we should go ahead and merge the address into the sync'd user. And maybe delete the old user. Would need to do some research to figure out the implications of this.

Duplicate of #789


Flaky test failure in preview2_udp_bind

This failure has now been seen here and here.
Currently what I've seen is:

Both failures happen on Windows, one on MinGW and one on MSVC
Both CI failures were trigger by failures of both the sync and async versions of the test
All failures are this line which is acquiring the local address of a UDP socket after it's bound to a fixed port.

This all seems fishy to me, I'm not sure what's going on! cc @badeend as you're likely interested.

Failure 1
failures:

---- async_::preview2_udp_bind stdout ----
preopen: TempDir { path: "C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\wasi_components_preview2_udp_bind.component_uiQC80" }
[guest] stderr:
thread 'main' panicked at crates\test-programs\src\bin\preview2_udp_bind.rs:35:43:
called `Result::unwrap()` on an `Err` value: ErrorCode { code: 9, name: "invalid-state", message: "The operation is not valid in the socket's current state." }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

===
thread 'async_::preview2_udp_bind' panicked at crates\wasi\tests\all\async_.rs:335:51:
called `Result::unwrap()` on an `Err` value: error while executing at wasm backtrace:
    0: 0xd3bc - <unknown>!__rust_start_panic
    1: 0xd200 - <unknown>!rust_panic
    2: 0xd133 - <unknown>!std::panicking::rust_panic_with_hook::h639672e96e0f2421
    3: 0xc545 - <unknown>!std::panicking::begin_panic_handler::{{closure}}::he2feed3b05783a82
    4: 0xc468 - <unknown>!std::sys_common::backtrace::__rust_end_short_backtrace::h2453fabad70b604e
    5: 0xcaea - <unknown>!rust_begin_unwind
    6: 0x11a8a - <unknown>!core::panicking::panic_fmt::h8d812bd208f205dc
    7: 0x13655 - <unknown>!core::result::unwrap_failed::h0c9b525dd5a26387
    8: 0x1ae8 - <unknown>!core::result::Result<T,E>::unwrap::hf03bf2ffbe46122a
    9: 0x2411 - <unknown>!preview2_udp_bind::test_udp_bind_specific_port::h457a5e69c52cafb1
   10: 0x3b5f - <unknown>!preview2_udp_bind::main::hc4fa6ed34db99c90
   11: 0x15d2 - <unknown>!core::ops::function::FnOnce::call_once::hacc676ed954b86d3
   12: 0x15a9 - <unknown>!std::sys_common::backtrace::__rust_begin_short_backtrace::h5d7b67e9a5e5af12
   13: 0x168a - <unknown>!std::rt::lang_start::{{closure}}::h5a806a5629e6b99f
   14: 0xacce - <unknown>!std::rt::lang_start_internal::h409072ad2c29d9a2
   15: 0x1635 - <unknown>!std::rt::lang_start::h776dc8db019efe49
   16: 0x410a - <unknown>!__main_void
   17: 0x1584 - <unknown>!_start
   18: 0x2e76a7 -<EMAIL_ADDRESS>note: using the `WASMTIME_BACKTRACE_DETAILS=1` environment variable may show more debugging information

Caused by:
    wasm trap: wasm `unreachable` instruction executed
stack backtrace:
   0: std::panicking::begin_panic_handler
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library\std\src\panicking.rs:597
   1: core::panicking::panic_fmt
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library\core\src\panicking.rs:72
   2: core::result::unwrap_failed
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library\core\src\result.rs:1652
   3: core::result::Result<T,E>::unwrap
   4: all::async_::preview2_udp_bind::{{closure}}
   5: all::async_::preview2_udp_bind::{{closure}}
   6: <core::pin::Pin<P> as core::future::future::Future>::poll
   7: <F as core::future::into_future::IntoFuture>::into_future
   8: <F as core::future::into_future::IntoFuture>::into_future
   9: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
  10: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
  11: tokio::runtime::context::runtime::enter_runtime
  12: tokio::runtime::runtime::Runtime::block_on
  13: tokio::runtime::runtime::Runtime::block_on
  14: all::async_::preview2_udp_bind::{{closure}}
  15: all::async_::preview2_udp_bind::{{closure}}
  16: core::ops::function::FnOnce::call_once
  17: core::ops::function::FnOnce::call_once
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library\core\src\ops\function.rs:250
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

---- sync::preview2_udp_bind stdout ----
preopen: TempDir { path: "C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\wasi_components_preview2_udp_bind.component_oQBE1O" }
[guest] stderr:
thread 'main' panicked at crates\test-programs\src\bin\preview2_udp_bind.rs:35:43:
called `Result::unwrap()` on an `Err` value: ErrorCode { code: 9, name: "invalid-state", message: "The operation is not valid in the socket's current state." }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

===
thread 'sync::preview2_udp_bind' panicked at crates\wasi\tests\all\sync.rs:278:45:
called `Result::unwrap()` on an `Err` value: error while executing at wasm backtrace:
    0: 0xd3bc - <unknown>!__rust_start_panic
    1: 0xd200 - <unknown>!rust_panic
    2: 0xd133 - <unknown>!std::panicking::rust_panic_with_hook::h639672e96e0f2421
    3: 0xc545 - <unknown>!std::panicking::begin_panic_handler::{{closure}}::he2feed3b05783a82
    4: 0xc468 - <unknown>!std::sys_common::backtrace::__rust_end_short_backtrace::h2453fabad70b604e
    5: 0xcaea - <unknown>!rust_begin_unwind
    6: 0x11a8a - <unknown>!core::panicking::panic_fmt::h8d812bd208f205dc
    7: 0x13655 - <unknown>!core::result::unwrap_failed::h0c9b525dd5a26387
    8: 0x1ae8 - <unknown>!core::result::Result<T,E>::unwrap::hf03bf2ffbe46122a
    9: 0x2411 - <unknown>!preview2_udp_bind::test_udp_bind_specific_port::h457a5e69c52cafb1
   10: 0x3b5f - <unknown>!preview2_udp_bind::main::hc4fa6ed34db99c90
   11: 0x15d2 - <unknown>!core::ops::function::FnOnce::call_once::hacc676ed954b86d3
   12: 0x15a9 - <unknown>!std::sys_common::backtrace::__rust_begin_short_backtrace::h5d7b67e9a5e5af12
   13: 0x168a - <unknown>!std::rt::lang_start::{{closure}}::h5a806a5629e6b99f
   14: 0xacce - <unknown>!std::rt::lang_start_internal::h409072ad2c29d9a2
   15: 0x1635 - <unknown>!std::rt::lang_start::h776dc8db019efe49
   16: 0x410a - <unknown>!__main_void
   17: 0x1584 - <unknown>!_start
   18: 0x2e76a7 -<EMAIL_ADDRESS>note: using the `WASMTIME_BACKTRACE_DETAILS=1` environment variable may show more debugging information

Caused by:
    wasm trap: wasm `unreachable` instruction executed
stack backtrace:
   0: std::panicking::begin_panic_handler
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library\std\src\panicking.rs:597
   1: core::panicking::panic_fmt
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library\core\src\panicking.rs:72
   2: core::result::unwrap_failed
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library\core\src\result.rs:1652
   3: core::result::Result<T,E>::unwrap
   4: all::sync::preview2_udp_bind::{{closure}}
   5: all::sync::preview2_udp_bind::{{closure}}
   6: all::sync::preview2_udp_bind::{{closure}}
   7: core::ops::function::FnOnce::call_once
   8: core::ops::function::FnOnce::call_once
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library\core\src\ops\function.rs:250
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

failures:
    async_::preview2_udp_bind
    sync::preview2_udp_bind

Failure 2
failures:

---- async_::preview2_udp_bind stdout ----
preopen: TempDir { path: "C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\wasi_components_preview2_udp_bind.component_C7RHTg" }
[guest] stderr:
thread 'main' panicked at crates\test-programs\src\bin\preview2_udp_bind.rs:35:43:
called `Result::unwrap()` on an `Err` value: ErrorCode { code: 9, name: "invalid-state", message: "The operation is not valid in the socket's current state." }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

===
thread 'async_::preview2_udp_bind' panicked at crates\wasi\tests\all\async_.rs:335:51:
called `Result::unwrap()` on an `Err` value: error while executing at wasm backtrace:
    0: 0xc363 - <unknown>!__rust_start_panic
    1: 0xc1a7 - <unknown>!rust_panic
    2: 0xc0da - <unknown>!std::panicking::rust_panic_with_hook::h639672e96e0f2421
    3: 0xb4ec - <unknown>!std::panicking::begin_panic_handler::{{closure}}::he2feed3b05783a82
    4: 0xb40f - <unknown>!std::sys_common::backtrace::__rust_end_short_backtrace::h2453fabad70b604e
    5: 0xba91 - <unknown>!rust_begin_unwind
    6: 0x10a31 - <unknown>!core::panicking::panic_fmt::h8d812bd208f205dc
    7: 0x125fc - <unknown>!core::result::unwrap_failed::h0c9b525dd5a26387
    8: 0x1b45 - <unknown>!core::result::Result<T,E>::unwrap::hba05ad71650d65eb
    9: 0x246e - <unknown>!preview2_udp_bind::test_udp_bind_specific_port::h7df0afeb18519e70
   10: 0x3bbc - <unknown>!preview2_udp_bind::main::h3b5389890cf6c15d
   11: 0x162f - <unknown>!core::ops::function::FnOnce::call_once::he547435657136ad8
   12: 0x1606 - <unknown>!std::sys_common::backtrace::__rust_begin_short_backtrace::h99201b78ada4a42e
   13: 0x16e7 - <unknown>!std::rt::lang_start::{{closure}}::h856cf0990f321ac7
   14: 0x9c75 - <unknown>!std::rt::lang_start_internal::h409072ad2c29d9a2
   15: 0x1692 - <unknown>!std::rt::lang_start::h9ee8d16b8e164ff4
   16: 0x4167 - <unknown>!__main_void
   17: 0x15e1 - <unknown>!_start
   18: 0x2e7076 -<EMAIL_ADDRESS>note: using the `WASMTIME_BACKTRACE_DETAILS=1` environment variable may show more debugging information

Caused by:
    wasm trap: wasm `unreachable` instruction executed
stack backtrace:
   0: rust_begin_unwind
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library\std\src/panicking.rs:597:5
   1: core::panicking::panic_fmt
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library\core\src/panicking.rs:72:14
   2: core::result::unwrap_failed
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library\core\src/result.rs:1652:5
   3: core::result::Result<T,E>::unwrap
   4: all::async_::preview2_udp_bind::{{closure}}::test_impl::{{closure}}
   5: all::async_::preview2_udp_bind::{{closure}}
   6: <core::pin::Pin<P> as core::future::future::Future>::poll
   7: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
   8: tokio::runtime::park::CachedParkThread::block_on
   9: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
  10: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
  11: tokio::runtime::context::runtime::enter_runtime
  12: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
  13: tokio::runtime::runtime::Runtime::block_on
  14: all::async_::preview2_udp_bind
  15: all::async_::preview2_udp_bind::{{closure}}
  16: core::ops::function::FnOnce::call_once
  17: core::ops::function::FnOnce::call_once
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library\core\src\ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

---- sync::preview2_udp_bind stdout ----
preopen: TempDir { path: "C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\wasi_components_preview2_udp_bind.component_3DHjto" }
[guest] stderr:
thread 'main' panicked at crates\test-programs\src\bin\preview2_udp_bind.rs:35:43:
called `Result::unwrap()` on an `Err` value: ErrorCode { code: 9, name: "invalid-state", message: "The operation is not valid in the socket's current state." }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

===
thread 'sync::preview2_udp_bind' panicked at crates\wasi\tests\all\sync.rs:278:45:
called `Result::unwrap()` on an `Err` value: error while executing at wasm backtrace:
    0: 0xc363 - <unknown>!__rust_start_panic
    1: 0xc1a7 - <unknown>!rust_panic
    2: 0xc0da - <unknown>!std::panicking::rust_panic_with_hook::h639672e96e0f2421
    3: 0xb4ec - <unknown>!std::panicking::begin_panic_handler::{{closure}}::he2feed3b05783a82
    4: 0xb40f - <unknown>!std::sys_common::backtrace::__rust_end_short_backtrace::h2453fabad70b604e
    5: 0xba91 - <unknown>!rust_begin_unwind
    6: 0x10a31 - <unknown>!core::panicking::panic_fmt::h8d812bd208f205dc
    7: 0x125fc - <unknown>!core::result::unwrap_failed::h0c9b525dd5a26387
    8: 0x1b45 - <unknown>!core::result::Result<T,E>::unwrap::hba05ad71650d65eb
    9: 0x246e - <unknown>!preview2_udp_bind::test_udp_bind_specific_port::h7df0afeb18519e70
   10: 0x3bbc - <unknown>!preview2_udp_bind::main::h3b5389890cf6c15d
   11: 0x162f - <unknown>!core::ops::function::FnOnce::call_once::he547435657136ad8
   12: 0x1606 - <unknown>!std::sys_common::backtrace::__rust_begin_short_backtrace::h99201b78ada4a42e
   13: 0x16e7 - <unknown>!std::rt::lang_start::{{closure}}::h856cf0990f321ac7
   14: 0x9c75 - <unknown>!std::rt::lang_start_internal::h409072ad2c29d9a2
   15: 0x1692 - <unknown>!std::rt::lang_start::h9ee8d16b8e164ff4
   16: 0x4167 - <unknown>!__main_void
   17: 0x15e1 - <unknown>!_start
   18: 0x2e7076 -<EMAIL_ADDRESS>note: using the `WASMTIME_BACKTRACE_DETAILS=1` environment variable may show more debugging information

Caused by:
    wasm trap: wasm `unreachable` instruction executed
stack backtrace:
   0: rust_begin_unwind
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library\std\src/panicking.rs:597:5
   1: core::panicking::panic_fmt
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library\core\src/panicking.rs:72:14
   2: core::result::unwrap_failed
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library\core\src/result.rs:1652:5
   3: core::result::Result<T,E>::unwrap
   4: all::sync::preview2_udp_bind::test_impl
   5: all::sync::preview2_udp_bind
   6: all::sync::preview2_udp_bind::{{closure}}
   7: core::ops::function::FnOnce::call_once
   8: core::ops::function::FnOnce::call_once
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library\core\src\ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

failures:
    async_::preview2_udp_bind
    sync::preview2_udp_bind

One possible clue, we don't know at this time which of these four functions triggered the failure:
    test_udp_bind_specific_port(&net, IpAddress::IPV4_LOOPBACK);
    test_udp_bind_specific_port(&net, IpAddress::IPV6_LOOPBACK);
    test_udp_bind_specific_port(&net, IpAddress::IPV4_UNSPECIFIED);
    test_udp_bind_specific_port(&net, IpAddress::IPV6_UNSPECIFIED);

This may perhaps be an ipv4-vs-ipv6 thing on a specific github actions builder, unsure!

I think this is (yet again :) ) the same issue as in https://github.com/bytecodealliance/wasmtime/issues/7429 and https://github.com/bytecodealliance/wasmtime/pull/7339.  If bind fails, we fall through instead of short circuiting the test case... :grimacing: Line 35 just happens to be the first thing to execute after bind failed, but other than that I don;t think it's actually related.

I'll submit a fix

Oh oops, and good catch! Here I was looking for a more complicated answer when the obvious one was staring me in the face...


How can I read data from the dataset I created?

Hi @jufeif,
Which Fondant version have you used?
After you have run a pipeline parquet files where created in your base path/working directory.
To export your dataset we recommend to use one of the reusable write components (e.g. write to file) or implement your own write component.
The export could look like this:
 dataset.write(
    YourWriteComponent,
    arguments={
        ...
    }
)

We are using write component to export your dataset to different data sinks.


Added restorecon to restore SELinux contexts to file default after mv

The mv command will take the SELinux contexts from the source file
when overwriting an existing one.
This was the primary way files were changed in the miqnet.sh script.
A new file, created by root, will most likely have different contexts
than the default for the file being edited.
This was causing NetworkManager to fail to write DNS servers retrieved
from the DHCP server to resolv.conf after we had previously edited it.
https://bugzilla.redhat.com/show_bug.cgi?id=1133554

Thanks @carbonin - great job.
Do you also think there is a bug around this for the imagefactory files that are originally put down?

It is definitely possible.  I was also looking at https://bugzilla.redhat.com/show_bug.cgi?id=1145752
In this situation it looks like the file has the incorrect permissions on first boot of the downstream appliance, but I don't think it is causing a problem with the DHCP (not with the latest appliance at least).  Something else around SELinux may have changed to allow dhclient to read that file since the bug was reported.

@kbrock
I would assume we are overwriting the permissions any time we replace a system file with one we create as root.  It does look like we wouldn't be running into the issue if we copied the files rather than moved them (http://wiki.centos.org/HowTos/SELinux#head-65817443e1aad7a1d1c87f6a4dc73a964b38224b), but I don't feel comfortable relying on that behavior.

Agreed. moving and then restorecon them sounds good.
:+1: I'm good with this - ship it.

@jrafanie @gtanzillo Please review

It's it's a common pattern to move and restorecon, does it make sense to create a method that does that pair of operations?

@jrafanie @carbonin I've mentioned this before, post imagefactory, pre ship, I'd like to restorecon -R /
@Fryguy Almost seems like we want a chef like method:
def build_file(filename, template = nil)
  f = TempFile.new
  yield f
  if !File.exist?(filename) || IO.read(f.path) != IO.read(filename)
    File.copy(f.path, filename)
    `restorecon #{filename}`
  end
ensure
  f && file.unlink && file.close
end

NOTE: 100% pseudo code. and including an erb would be nice too.
But when ever I see us copying a method from a configuration file management system, I always need to ask, "Can we just push this off to puppet / satellite and get out of configuration file buisness all together?"

:+1: I'm good with Nick's change.

The function for the file move and restorecon should be good to implement for this particular file.

I'd :+1: with this.  @carbonin if you want to create a function, it's up to you, otherwise we can merge it.
It feels dirty that you have to do this every single time you move a system file as root.  I guess my hope was that systemd provided CLI options to eliminate 90% of this miqnet.sh / selinux :hankey:


KEP-3668: promote ServiceNodePortStaticSubrange to beta

This PR promotes KEP-3668: ServiceNodePortStaticSubrange to beta

/assign @Rishit-dagli
/milestone 1.28

The upstream PR https://github.com/kubernetes/kubernetes/pull/117877 is merged now. However this PR is in conflict, so I have rebased it.
Would you mind reviewing this PR again? @tengqm

/lgtm


https://github.com/stratospheric-dev/cdk-constructs/issues/4

@BjoernKW It's hard to see the actual changes because there's a lot of whitespace changes included...

@BjoernKW It's hard to see the actual changes because there's a lot of whitespace changes included...

You're right. The code formatting according to the settings in .editorconfig has changed every source code file. I've created an intermediate branch with those formatting changes and a new pull request for merging into that branch: https://github.com/stratospheric-dev/cdk-constructs/pull/6
That way the actual changes become more obvious.


Support parquet in BigDiffy/BigSampler

this PR expands on the limited parquet-support in parquet-extras; moves all Parquet logic into ratatool-sampling and ratatool-diffy; and adds Parquet integration to BigDiffy/BigSampler/directSampler.
Note that Parquet can currently be written in Scio as case classes ("parquet-typed"), or as Specific/GenericRecord Avro ("parquet-avro"). The code I added supports inputs of either type by reading all Parquet inputs as GenericRecords. This is possible because all Parquet files have schema information in the file metadata header; even if you write Parquet from case classes, it can still be converted to an Avro schema and read into GenericRecords.
This includes unit tests and I also tested all three commands with parquet-typed and parquet-avro data.

@clairemcginty
I ran BigDiffy on my hack days project with the Scio pipeline writing parquet files, and it worked!


KAFKA-5350: Modify unstable annotations in Streams API

For all public interfaces that are being affected by any ongoing KIPs, I have left them as evolving.

I am also wondering why it's only 48 files -- I just had a look and is should be more than 70 ?

Some of the class files do not have this annotations at the first place: we were not very consistent when adding these classes before.

Thanks for the reviews. Merged to trunk and 0.11.0.


refactor(#125): Added attribute data to context for proactive fetching

This refactor proactively fetches data into the context. Done this as part of #10 because I'll be using that to store the Chart polling job and data (as it seemed better and to avoid this logic into the component)
This has some nice advantages:

Removes the need to use useEffect on the component themselves, being able to be "dumb" (they just operate on the given props/context)
Allows for proactive logic in table selection
Separates fetching and mapping logic from rendering. Easier unit testing

Having the Chart data in the context also allows for the Chart data to be maintained while on different tabs (as base HawtIO loses the data when changing to Attribute view). Could also be improved on the future to maintain the data on multiple visited nodes, or whatever we want.
We could move the actions page data to the context as well if that's ok, although that has a bit more logic so I left it out for now.
Putting this up to get some feedback on the design. I left out the types I'll be using for Chart in case you want to give some feedback on them. WDYT?

I should add also some tests. I'll be working on those

@tadayosi It's as we were talking about in the other PR, as this allows us to fetch the data proactively and let the components only focus on the rendering part, having the data already gathered.
Also allows for only one point of data gathering instead of having every page have its own job

If we remove scheduler registration in the useEffect, how can we get the realtime data of the attributes?

The idea is to have that Data service take care of the scheduler jobs, instead of every component, which would be the next step. That way every component can always ensure that they have the latest data only accessing the context.
Also helps avoids bugs because if we have () => unregister x in every component in the return function of the useEffect, which is the cleanup, it could happen that it cleans up something that it shouldn't and creates logic dependency between components.
For example with the chart component I'll be setting a Post job request like in base hawtio and store all the data there, but I'll remove all the logic from the component, and allowing us to do logic and store data between components, as base hawtio loses all the data on component switch, making really complicated for the user to have any kind of trackable data unless they don't switch components (in fact, while trying to record a gif with peek, I discovered it loses the data even in a window resize!)

We should at least keep the PluginNodeSelectionContext as only holding the selected node.

I can create a new context that only pertains to the current plugin. For example, in JMXContent.  Would that be better for you?

@joshiraez No, I won't accept the changes. I don't think you fully understand what you are going to change. Can you instead just create a Chart component that is self-content, reading the mbean attributes within itself?  I'd like to ask you to first understand what the old version of Hawtio does in the Chart view and replicate the logic.

@tadayosi I can do that but it'll be both slower, have more complexity, and it will have bugs like it happened with the tabular attribute table. Is that okay with you?
I mean, is just copying this same logic to the component itself. There is no work lost there. Plus I'm decoupling it a bit as the chart component on base HawtIO depended on Core for rendering which I though was one of the goals to achieve.


react-router v7

going from react-router-dom v6 to react-router v7, its supposed to be a non-breaking change, but it seems incompatible from my end. any ideas?

realized a mistake on my part when reattempting.


Sync custom select focus with form control focus

Fixes #27944.

Tests fail because Variable $custom-select-focus-width is not being used!

I deleted the variable. Don't think that would break anything since this variable was used to construct $custom-select-focus-box-shadow.

Hmm, don't think we can delete variables in v4 (backward compatibility). Maybe first deprecate them by reassigning their values? Unsure about the unused vars linter and how to whitelist something, if we can at all.

I've tweaked the variable so that it makes use of $custom-select-focus-width. @mdo, what do you think of this?


Stream function error: MPV can't access audio drivers (Streamlit web app)

Is there any way to use the stream function on the cloud-deployed web app?
I'm using it in a very simple Streamlit app. Although it works on my local device, the stream function doesn't play any sound on Streamlit cloud or within a Docker container. Specifically, this line just doesn't work: mpv_process.stdin.write(chunk)  # type: ignore. It seems like MPV cannot access the system's audio drivers in streamlit cloud or in a Docker container. Here's the stderror output for the mpv_process on Streamlit cloud:
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'\nALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory\nALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings\nALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory\nALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name\nALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory\nALSA lib conf.c:5233:(snd_config_expand) Evaluate error: No such file or directory\nALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM default\nCannot connect to server socket err = No such file or directory\nCannot connect to server request channel\njack server is not running or cannot be started\nJackShmReadWritePtr::~JackShmReadWritePtr - Init not done for -1, skipping unlock\nJackShmReadWritePtr::~JackShmReadWritePtr - Init not done for -1, skipping unlock\nALSA lib confmisc.c:767:(parse_card) cannot find card '0'\nALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory\nALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings\nALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory\nALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name\nALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory\nALSA lib conf.c:5233:(snd_config_expand) Evaluate error: No such file or directory\nALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM default\nALSA lib pcm_hw.c:1829:(_snd_pcm_hw_open) Invalid value for card\ncouldn't open play stream: No such file or directory\n

On streamlit cloud, when i list the audio devices with mpv --audio-device=help", this is what I receive:
List of detected audio devices:
'auto' (Autoselect device) '
alsa' (Default (alsa))
'jack' (Default (jack)) '
sdl' (Default (sdl))
'sndio' (Default (sndio))

Any advice here? I want to 'hear' the audio on a deployed webapp. It doesn't have to be on streamlit cloud but I want the stream function to work.

I tested locally on streamlit but only the source machine plays the audio...not sure if the 2 are connected from your issue.
I'm curious to this thread as I'm planning to spin up a cloud instance (debating providers now) I'm wondering if some providers have issues ... streamlit being one since I also am testing on streamlit.
I'll let you now what I find

Hey guys, did you find a fix to this? trying to do something similar


Some questions

Hi @ValWood, I have a question about the physical interactions in Canto for the documentation.

Does it ever matter the order in which the genes are listed? My intuition is that:
* For affinity captures, the first one should be the bait and the other
* For Protein-RNA (binds to RNA), the first one should be the protein gene, and the second one the RNA one.
* For Protein-Peptide, the first one should be the full protein, the second one the peptide.

Another question, that I am adding to a section More about expression level and edge cases:

PomBase does not capture the difference between strains expressing an allele from a plasmid or the native locus in phenotype annotations.  Similarly, we do not have a way to capture the fact that the control of an experiments is over-expressing the wild-type allele. Therefore, if both the wild-type (control) and mutant allele are over-expressed / knocked-down, ???

What do we say here?

Therefore, if both the wild-type (control) and mutant allele are over-expressed / knocked-down,

I'm not sure. Yesterday I made such and annotation
overexpressed mutant compared to OEX WT control. I
made the mutant overexpresses  the assumption being that the WT control is also Overexpressed.
This is never stated explicitly in the curation  though that we are not comparing to WT with normal expression.

The interactions are directional
https://wiki.thebiogrid.org/doku.php/experimental_systems
but the direction should be explicit in the text in parentheses (i.e "affinity captures"  was supposed imply that a captures b
Maybe this is not clear enough?

Ok, that should be enough for me to write it


The placeholder under "Complete FortiGate command-line configuration" is wrong

Under "Complete FortiGate command-line configuration" (https://docs.microsoft.com/en-us/azure/active-directory/saas-apps/fortigate-ssl-vpn-tutorial#complete-fortigate-command-line-configuration), Step 2. Run these comands [...]
There is a line that says:
set idp-single-sign-on-url  <Azure AD Identifier>
This should be:
set idp-single-sign-on-url <IdP Single Sign-On URL>
And now that I write this, the overview table just above says:
IdP Entity ID (idp-entity-id) 	Azure Login URL
IdP Single Sign-On URL (idp-single-sign-on-url) 	Azure AD Identifier
But this should be the other way around, like this:
IdP Entity ID (idp-entity-id) 	Azure AD Identifier
IdP Single Sign-On URL (idp-single-sign-on-url) 	Azure Login URL
With the commands as specified the VPN-SSL connection does not work, when changing it as I suggest, it works.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 675b6583-259f-1135-6443-b9aa40258163
Version Independent ID: 28e01c75-b130-260c-4aa6-75a1cba20558
Content: Tutorial: Azure AD SSO integration with FortiGate SSL VPN - Microsoft Entra
Content Source: articles/active-directory/saas-apps/fortigate-ssl-vpn-tutorial.md
Service: active-directory
Sub-service: saas-app-tutorial
GitHub Login: @jeevansd
Microsoft Alias: jeedes

@datavinc Thanks for your feedback. We will investigate and update the thread.

@datavinc Thanks for your contribution to enhance the documentation. I have created the PR to update the changes. Changes will be reflected after author's approval.


PTS rollover handling incorrectly applied to DASH and fMP4 HLS

Have you read the FAQ and checked for duplicate open issues?
Yes
What version of Shaka Player are you using?
3.0.3 (shaka 2.5.x release is working without the issue)
Can you reproduce the issue with our latest release version?
Yes
Can you reproduce the issue with the latest code from master?
Yes
Are you using the demo app or your own custom app?
I have used the demo app
If custom app, can you reproduce the issue using our demo app?
Yes
What browser and OS are you using?
Chrome/MacOS
For embedded devices (smart TVs, etc.), what model and firmware version are you using?
What are the manifest and license server URIs?
Have been sent by email
What did you do?
Start playing DASH stream with embedded WebVTT subtitles
What did you expect to happen?
subtitles should be displayed
What actually happened?
With DASH or HLS with MPEG-TS, the maximum PTS is 2^33 =<PHONE_NUMBER> . There are packager on the market (eg. Harmonic), which are messing up this rollover within the vtt files. There are always several .vtt files arriving with values above<PHONE_NUMBER>, before the rollover is working. This is a bug on a packager side, and I guess you are trying to work around this bug. We are doing the same in our packager code, but only for MPEGTS input.
With DASH or HLS with fMP4 (that’s what we are using), this limit does not exist, but the rollover code is still applied.
Commit that brings this bug
W3 docs

The problem is that TS segments will always have a small timestamp due to the rollover problem.  To fix this, we adjust the WebVTT timestamps backward so they appear at the same time relative to the TS segment.  You can also have both TS and MP4 in the same manifest (though we won't switch between them).  This can complicate where the segments and text samples should be put.
What we should be doing is adjusting the TS segments forward so they appear at the correct time even though their timestamps aren't big enough to hold the correct time.  So if a TS segment says it should be at 4 seconds, we may need to add the rollover so it appears after the previous rollover.  We can use the WebVTT timestamps to find where to adjust the first segments.
I also wonder if we handle mid-stream rollovers correctly.  So if we have a segment with a large timestamp and the next segment has a small one, will it work.  The browser is supposed to handle this while playing, but this can complicate seeking.

I had intended to fix this for v3.1, thinking I could just revert that change and it would work out.  But unfortunately, we no longer parse segment times for WebVTT, so reverting that change wouldn't do what I wanted.  This means that there is no simple fix for this issue and we'd need to do the "full" fix for handling TS timestamps.
Since this change would be a large change and might introduce subtle bugs, we're going to wait until v3.2 to make this change.  We may be able to put this in a v3.1.x bugfix release, depending on how clean the fix is.

I don't know if this is resolved with the latest changes in HLS, @joeyparrish , can you check it?

I believe this is resolved now since #4217.


Fix #29 add jobid checks

@dylan-chong Please test it.

@roxma Can you review/merge it?
Note: I have removed tabs.

It looks like the errors in  #29 are triggered when exit_cb is called before err_cb or out_cb.
"exit_cb": handler	Callback for when the job ends.  The arguments are the
                        ...
                        ...
			Note that data can be buffered, callbacks may still be
			called after the process ends.

vim-hug-neovim-rpc cleanup the job entry when exit_cb is called. It doesn't look like a good place to do the cleanup.
I'm wondering, is it better to cleanup the data in close_cb?

If this PR is merged, we might lose data when exit_cb is called before data callbacks.

@Shougo I actually get a different error but I think this is because Macvim has some annoying problems being compiled with python.
In normal Vim, I don't get any problems on the current master or this branch.
Sorry if trying to fix this wasted some of your time

Hm.  It is related error?
https://github.com/Shougo/deoplete.nvim/issues/782

Closed.  Because #31 is better.


Add a clean time series method "average"

When I am doing reconciliation between top down and bottom up data and get a series of "factors" over time, I may want to to take an average of these factors to serve as reconciliation factors in future years.
To do this, I need a clean timeseries method called "average" that fills missing years with an average of the given years.
AC

the TimeSeries.clean method accepts an interpolation_method or extrapolation_method called "average" and uses the mean of the given data to fill in missing data over a new index

Ben, assigned to you to review and merge with master.


Support user specified OS name and version

supported os: rhel, ubuntu, sles
add logic in machine detection code
add logic in TKG makeGen to support user specified os
add tkg tests

fixes #417

run tkg-test


Format Code

Add/Create a js folder and move javascript files in js folder.
Inline styling in .js files should be in style.scss.
Make sure that your are making styling changes in style.scss file not in style.css file.

Please assign this to me. I would like to try.

@keivalya Are you working on this?

Hey @jatiinyadav !
Can you elaborate a little bit on how to accomplish 2 and 3. I am not sure what it actually means.

You have to remove inline css from javascript files and create class/id in style.scss file.
Use Live Sass Compiler if you're using Visual Studio Code (VSCode) and enable the live sass compiling. Live Sass Compiler

Do styling changes in style.scss  file.
Click on Watch Sass at the bottom of your VSCode.

Your Sass Folder will look like this:-

If you are facing any difficulties you can refer to this documentation.

Thank you so much, I will get it done as soon as possible.


--stdin-filepath should affect application of rules

Currently if I have a config file like:
# taplo.toml

include = [
    "path/to/thing.toml",
    # ...
]

[formatting]
reorder_keys = false

[[rule]]
include = ["path/to/thing.toml"]
keys = ["dependencies", "target.*.dependencies"]
[rule.formatting]
reorder_keys = true

# path/to/thing.toml

[dependencies]
y = 1
x = 1

then, running taplo format --config ./taplo.toml path/to/thing.toml, it correctly applies reorder_keys to reorder x before y.
But if I run taplo format --config ./taplo.toml --stdin-filepath path/to/thing.toml - <path/to/thing.toml, x and y are not reordered, despite the applicable rule.
I would have expected rules to apply based on stdin-filepath.
Mentioning @lffg since --stdin-filepath was introduced by you in #250.

It's not an issue with --stdin-filepath, it doesn't reorder keys without it either

It's not an issue with --stdin-filepath, it doesn't reorder keys without it either

I assume what you mean is that if you run cat path/to/thing.toml | taplo format --config ./taplo.toml - then it does not reorder x and y.
Yes, obviously -- in this case there is no possibility that it could apply a rule which only applies to "path/to/thing.toml".
But after being informed that the stdin filepath is "path/to/thing.toml", then rules which apply to "path/to/thing.toml" should get applied. Not applying them is unexpected behavior.

Thanks for the bug report! I currently don't have access to a computer but could you try testing with #378 to see if it fixes the issue? This looks like a path mismatch.

The behavior is the same even with #378.