Rethink how we handle hashes

This is a breakout from #433, which we attempted to fix in #462.
The problem boils down to this:

When a user passes --require-hashes or a requirements file that contains hashes, we should disable the dependency resolution machinery: the presence of hashes implies a fully resolved file, so we shouldn't attempt to resolve further.
#433 observed that, when a correctly hashed input contains hashes from third-party indices, we fail to check them correctly (since we only check hashes from PyPI, which don't necessarily match)
#462 attempted to fix #433 by moving hash checking into the dependency resolution layer, which produced the correct results...
...but regressed overall, since --require-hashes no longer skips dependency resolution and instead produces conflicts (#511)

TL;DR: I think we need to entirely rethink how we handle hashes. Some ideas:

Move hash checking to its own layer, outside of the vulnerability service layer (pre #462) or dependency layer (post #462). This entails some possible work duplication: we'll probably have to iterate over the requirements again, and make some more (cached) network requests, but it properly preserves the guarantee that hashing disables dependency resolution without locking us into just PyPI's hashes.
Loosen our hash checking enforcement: instead of validating hashes, we can simply check for their presence and well-formedness. This is "okay" in the sense that pip-audit is used in workflows with pip, and the latter will actually prevent any hash mismatches from falling through in practice. On the other hand, users may find this confusing (even if it's arguably fine from within our security and use model).

cc @di for thoughts.

Thinking about this some more: approach (1) wouldn't be too bad if we architect it correctly: we probably want internal _pep503.py and _hashes.py modules that abstract the indices and hash checking respectively, while reusing our caching middleware. From there, we'd run the hash checking step as a pre-step before anything else.
OTOH, this would still produce a decent slowdown compared to (2) -- we'd still end up fetching a bunch of distributions only to hash and compare them, which would mean that --require-hashes would go from "nearly instantaneous" to "roughly as slow as pip install.

And another argument for (2) -- in --require-hashes mode we never actually install anything, so validating the hashes rather than merely enforcing their presence produces no security properties.

You touched on it in another comment but I'm assuming there must be some kind of dependency resolution bug lurking here. I would have thought that even if we fully resolve a hashed requirements file, we'd get back the same set of dependencies that are listed in the requirements file (rather than complain about impossible resolution).
For me, the slowdown is expected since we pushed the hashed requirements path into dependency resolution (although I underestimated how much of a difference that would make) but the fact that we can't even resolve the requirements is a surprise to me.

Yeah, I think you're right -- this should have just been slow, rather than producing impossible resolutions. So there's likely a separate bug here.

So there's likely a separate bug here.

Random guess that just occurred to me: what might be happening is that we resolve foo==1.2.3, which produces bar==1.2.4 as a subdependency. But bar==1.2.3 is already pinned at the top-level, so we end up with an impossible resolution.

Random guess that just occurred to me: what might be happening is that we resolve foo==1.2.3, which produces bar==1.2.4 as a subdependency. But bar==1.2.3 is already pinned at the top-level, so we end up with an impossible resolution.

Wouldn't that be impossible to pip install though? Or does pip just try its best even if it can't satisfy all requirements correctly.
Anyway, it should be easy to do an experiment and figure out what pip's behaviour is.

Wouldn't that be impossible to pip install though? Or does pip just try its best even if it can't satisfy all requirements correctly.

Sorry, this was ambiguous of me -- what I meant is that this is possibly our behavior, not pip's.
In particular, my SWAG is that our backtracking doesn't take top-level requirements into account, only their subdependencies. This is fine in most "normal" requirements inputs, but breaks fully pinned ones because of scenarios like this:

Top-level: foo==1.2.3, bar==1.2.3 (from foo)
Resolve: foo==1.2.3 -> bar~=1.2 -> bar==1.2.4
Backtrack: bar==1.2.3

That's probably what pip does, whereas we might be getting caught on (2). I'll investigate to confirm that.

As we discussed earlier, we're going to just remove the hash checking logic altogether since pip does this already. We'll check that all requirements have a hash but we won't be checking their actual values.

The dependency resolution bug that we were discussing is captured in #522.

Everything is handled by pip now so I don't think there's much to do here anymore.


Fixes bug where dates were automatically iso over anything else due to a prop error

In https://github.com/airbnb/react-dates/pull/229, I apparently did not check the SDP (or if I did, my eyes glazes over when I saw an ISO format date in the input and called it a day). Either way, @vklein pointed out that this was likely due to an error in prop naming in https://github.com/airbnb/react-dates/issues/281.
SO. This fixes that issue. :)
to: @airbnb/webinfra @vklein

Coverage remained the same at 86.926% when pulling 9ba49191084ab9c6b1002808b63b48fc665c8496 on maja-fix-display-format-bug into 4f9cfbca18a0583233f187cdc49e9509c6e83f53 on master.

Coverage increased (+0.02%) to 86.944% when pulling 3d8ea0bb36188b90a9e89b500a3e6e0513bf21a0 on maja-fix-display-format-bug into f08b3e63da0c3670c5f154810bf1c6e7d42c90a9 on master.


tree_type_checker(*NEURITES) does not filter out soma

since NEURITES is a tuple that includes NeuriteType.all, it will never filter anything out.
Here is a modified version of the existing unit test to test for soma filtration.
def test_tree_type_checker():
    tree_filter = tree_type_checker(NeuriteType.all)
    nt.ok_(tree_filter('fake_tree'))

    mock_tree = Mock()
    mock_tree.type = NeuriteType.axon

    tree_filter = tree_type_checker(*NEURITES)
    nt.ok_(tree_filter(mock_tree))

    mock_tree.type = NeuriteType.soma
    nt.ok_(not tree_filter(mock_tree))  # <- This assertion fails

This is a topic for discussion rather. NeuroM followed the logic of NEURON simulation, and considered soma as section hence NEURITES were defined right in that paradigm. Now, since NeuroM moved to MorphIO, soma is not considered as section. Hence .all can't be part of NEURITES. I can drop it from there but I need confirmation from other interesting parties.
@arnaudon can you on behalf of scientists discuss the drop of NeuriteType.all from NEURITES in neurom.core.types module?

I sent it around to discuss, I'll come back to you. I never use .all myself, but I've seen it in a few places. It may break some codes if we remove it.

I guess it's a bit of semantic issue. I wouldn't consider the soma as a neurite. But that doesn't mean it's not a section though.
So maybe the naming NEURITES was a bit unfortunate?

But that doesn't mean it's not a section though.

@wvangeit  In NeuroM v2 soma is not considered as section because of MorphIO.

I agree that soma does not make sense as a neurite. However, we need to compute measurements, such as total length, for all neurites (including for example axon + dendrites). In addition, a lot of publications require the features for axons + dendrites.

Removing of .all won't affect axon + dendrites. They stay. Only soma will be out of measurements. Currently when you use NEURITES, you get soma into your measurements as well.


Release new integrations for 7.54 Agent

What does this PR do?

Motivation

Additional Notes

Review checklist (to be filled by reviewers)

[ ] Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
[ ] Changelog entries must be created for modifications to shipped code
[ ] Add the qa/skip-qa label if the PR doesn't need to be tested during QA.
[ ] If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

Hadhemi approved on behalf agent-integrations, she just hasn't been added to the gh team yet.


Update links in design doc

This updates some old links regarding the scheduler design proposal. I also changed extender -> plugin in the same places since this seems to be the current terminology used. However, I'm sure there are many more places where we have "extender", not sure if it is worth updating all of them since it is still easy to understand I think.
The old link goes here: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/scheduler_extender.md
It links to this archived repo. But I think it makes more sense to change the link to the scheduling framework docs (which is what I did in this PR), or to this KEP (which is linked in the scheduler framework docs also).

@lentzi90 Thank you for your PR.
But, "Scheduler Extender" and "Scheduling Framework (plugin)" are different. It is a mistake to change "Scheduler Extender" to "Scheduling Framework (plugin)" since it is "Scheduler Extender" that is being referred to in the pointed out section.

@lentzi90 On the other hand, the documentation for Scheduler Extender is archived. This URL will be updated in the #411.

Ah thank you, now I understand!
I noticed also that the scheduler extender will probably be removed according to this so maybe not so important to update this.


Flip Cotton order to match LDtk layer order

Fix has been prepared. Close on next release.

Fixed in 0.6.1


Translate concepts/workloads/pods/ephemeral-containers into Japanese

Closes #23933.

/assign @inductor
Page preview: https://deploy-preview-23936--kubernetes-io-master-staging.netlify.app//ja/docs/concepts/workloads/pods/ephemeral-containers/

ご対応ありがとうございます
/lgtm

/approve


Support isinf/isnan for complex number using numpy semantic

Only for scalar mode, let's implement this for complex numbers at some point!

can we remove the function in utils namespace then
https://github.com/QuantStack/xsimd/blob/55996ddaea1e716548167639d5dd8a7d10202fd7/test/xsimd_test_utils.hpp#L34-L44
Also, is our SIMD isinf function doing the correct thing for xsimd batches?

I didn't implement the batch version ^^!


docs: highlight lack of joystick info on majority of platforms

Just thought I'd add a note to the Input class that explicitly describes the lack of platform-specific Joypad info on the majority of platforms. This is the silliest PR ever, but it actually tripped me up for a second.
I did check the source to verify that only Windows and Linux explicitly collect and pass system-specific Joypad information in before making this claim.

You should squash the two commits on your PR into one, see here.

Thanks! And congrats for your first merged Godot contribution :tada:


[Side project] Add price predictions to tweets with ticker

Make a seperate repo for this, since it is a project on its own.
This takes a lot of space, since we need to have a database to use for classifying certain tickers.
We could limit it to a certain amount in the beginning and slowly scale, for instance only classifying BTC, ETH, XRP on the 4h timeframe, using x amount of candles.

If we want to use less data we can use the sentiment + current day price change for our prediction. It will be less accurate, but might be worth something.

Could use combination of sentiment in tweet + simple strategy on the price to guess price direction
mathematics-10-02156-v2.pdf

Wait after finishing https://github.com/StephanAkkerman/Crypto_Forecasting

Results for predicting with only price data were not satisfactory. We would run into problems like:

What data do we use?
How often do we train?
What models to use?
One model per ticker or one model for all?


[Question]: Is it possible to append new nodes to an existing index without overriding the whole index?

Question Validation

[X] I have searched both the documentation and discord for an answer.

Question
I have a database to which I constantly add new data, then I made an index to query that database, but since that database is constantly being updated, every time the database changes, I need to overwrite the index with the new database nodes, which are the exact same except for the new ones.
Is there a way I could just add the new elements to the same index instead of overwriting it?

If you have a way of setting a consistent doc id for each input document, you could use the refresh function.
Basically, the same source document should have some consistent ID for this to work properly. For example, it could be a filename, or maybe the thread_id for a thread of discord messages. Just has to be static
https://gpt-index.readthedocs.io/en/stable/core_modules/data_modules/index/document_management.html#refresh


Fix jwt decoding

The algorithm was not passed to the decoding function.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.


Update --count option description

It shows 'number of matching lines', not number of matches.
This is related to #121. Original description is a little confusing.

Original description is a little confusing.

Oh...
LGTM. Thanks!!


Include cookie name

Support for an optional cookie name, since the default connect.sid from  express-session may not be recommended or loved by everyone

Ooh neato, good idea - thanks for this!
I'm hoping to give next-auth some attention this week before the holidays.


Get...Ref functions - export from TADA?

Is your feature request related to a problem? Please describe.
When working on the training materials, I thought it might be helpful to allow users to see the reference tables.
Describe the solution you'd like
I'd like to export the Get...Ref functions from TADA
Describe alternatives you've considered
The current alternative is that users must go to the WQX domain page to find these reference tables, and some of the ones within the package have additional columns added by TADA.
Additional context
Add any other context or screenshots about the feature request here.

Good idea! Yes, let's export these


Create a kubectl like example supporting core verbs

What problem are you trying to solve?
Would like to have an easy way to produce arbitrary api requests using kube to show (using tracing/logs) how we follow the api / deviate from kubectl.
Would also be nice to have a short example showcasing the basics of how kubectl works. The core verbs should not require a whole lot of work.
(We should not do the hard verbs, because then we'd have to merge in lots of complicated pod examples like port-forwarding/proxy/exec.)
Describe the solution you'd like
An example that starts with api discovery (like dynamic_api example but for extracting ApiResource) and creates a map of unambiguous kinds -> ApiResource as a BTreeMap<String, ApiResource> after iterating over discovered results. It's ok to ignore ambiguous kinds for this or take the first kind.
Then, arg parsing for supporting kubectl style cargo run --example kubectl -- get pods mycontainer, and using that in simple behaviour:

based on arg2 (resource) create an Api (namespaced if the apiresource is namespaced)
based on arg3 (name) perform the verb action (arg1) e.g. api.get(name) and print the result

i.e. main things to do in the example:

[ ] initialise easy to read logging (env logger / tracing fmt)
[ ] api discovery into dict of kinds
[ ] arg parsing (kubectl subcommands style: verb noun --flags)
[ ] sanity: fail early if invalid verbs, resources
[ ] support get name -> print yaml
[ ] support list -> print table
[ ] support delete name -> delete and print message
[ ] parse ListParams from flags for delete and list verbs
[ ] support delete with ListParams -> api.delete_collection
[ ] use await_condition is_deleted in the api.delete branch
[ ] support watch (create a watcher and print simplified events)
[ ] support apply (add a input file arg, serialize it from yaml, into json and pass to api.patch)

Partial PRs for this is also appreciated, but this might be a nice little exercise that gets to use quite a bit of the API without having to battle with dragons everywhere.
Target crate for feature

examples

Think this is actually a good exercise for us to do, and could provide good content as:

a walkthrough of using the api
showing how conceptually simple kubectl is
showing how simple it can be done with kube-rs

so will try to do this and possibly record a thing.


AfterFeatures is never called in child processes

I am registering an "AfterFeatures" callback in hooks.js using the following code
this.registerHandler('AfterFeatures', function (event, callback) {
global.driver.quit();
callback();
});
Which then closes the chrome window that it is using.
When running my tests using parallel-cucumber-js, this does not get called and my browsers stay open until I close them manually

Turns out AfterFeatures was being called, but I had been setting a reference to driver using the global scope.
As well as this, I had to add a timeout before calling back to give it enough time to close the browser.
I assigned driver to the world instance instead, here is the final code that works:
this.registerHandler('AfterFeatures', function (event, callback) {
    this.driver.close(callback);
    setTimeout(callback, 1000);
});


📝 Fix typo in link to typer-cli

This looks good to me.
I manually confirmed that the link is broken (404) in the docs, and that it's working in the docs preview.

Thanks! :mag:
I hadn't seen this PR and I discovered the same and fixed it in https://github.com/tiangolo/typer/pull/588. Given that, I'll close this one. But thanks for the effort! :bow:


Fork Tidecoin to dogecoin PQC

For the lulz and because the bitcoin bros didn't listen to our quantum concerns surrounding taproot, lets make a Quantum Resistant dogecoin instead. Let's start with merging the supply and block time modifications that are the foundation of the dogecoin economic model. much wow.

it won't happen that fast, but you can see that the problem is known to the creator of bitcoin
https://www.publish0x.com/crypto-snacks/bitcoin-taproot-upgrade-increases-quantum-vulnerability-xglypvg


UnhandledPromiseRejection when opening Telescope

Neovim version (nvim -v | head -n1)
nvim v0.7.0
Operating system/version
Arch Linux
How to reproduce the issue

Open Telescope

Expected behavior
No errors
Actual behavior
When I open Telescope (git-files in this case) I see at least sometimes:
Error executing vim.schedule lua callback: UnhandledPromiseRejection with the reason:
...vim/site/pack/packer/start/nvim-ufo/lua/ufo/provider.lua:48: ...site/pack/packer/start/nvim-ufo/lua/ufo/provider/lsp.lua:58: attempt to call method 'match' (a nil value)
stack traceback:
        ...re/nvim/site/pack/packer/start/nvim-ufo/lua/ufo/fold.lua:51: in function <Anonymous:40>

What's lsp client?

In this case it is gopls.
Also the error seems to go away after I opened a fold for the first time (closing folds does not make it go away).

Not a server, I mean coc.nvim or nvim lsp.
It's weird that throw a nil reason by error()

I'm sorry, I use nvim LSP client.

Update: I now also got it after closing and opening a fold again, so no reliable way to get it solved (that I know of).

I can swallow the nil reason in the fallback function for the nvimlsp client, but it's fragile, need to handle it inside nvimlsp, need time to explore the source code.

Please update the code, I have checked buffer valid before request server.

Updated, problem still present. I even deleted my session and created a new one.
When I open Telescope I still get the exact same error message as in the issue description.
UfoInspect now has a slightly different output for the telescope buffer though:
Buffer: 14
Fold Status: start
Main provider: lsp
Fallback provider: indent
Selected provider: indent

try try-fix-nvimlsp branch please, and post the message if has any errors.

Error also present and still exactly the same:
Error executing vim.schedule lua callback: UnhandledPromiseRejection with the reason:
...vim/site/pack/packer/start/nvim-ufo/lua/ufo/provider.lua:48: ...site/pack/packer/start/nvim-ufo/lua/ufo/provider/lsp.lua:58: attempt to call method 'match' (a nil value)
stack traceback:
        ...re/nvim/site/pack/packer/start/nvim-ufo/lua/ufo/fold.lua:51: in function <Anonymous:40>

I think I have known the reason, please update the code and post the error again, I want to confirm that.

Error message changed now:
Error executing vim.schedule lua callback: UnhandledPromiseRejection with the reason:
{ ["message"] = read /: is a directory, ["code"] =  }
stack traceback:
        ...re/nvim/site/pack/packer/start/nvim-ufo/lua/ufo/fold.lua:51: in function <Anonymous:40>

Sorry, I don't understand what you mean by that.
What I do is simply opening a Telescope window. It does not matter whether that is showing files (':Telescope git_files'), help (:Telescope help_tags) or git branches (:Telescope git_branches). I always get the same error message. I have no idea where the root directory / could come into play here.

I have a similar issue if I commit via neogit:
Error executing vim.schedule lua callback: UnhandledPromiseRejection with the reason:
...vim/site/pack/packer/start/nvim-ufo/lua/ufo/provider.lua:48: ...site/pack/packer/start/nvim-ufo/lua/ufo/provider/lsp.lua:58: attempt to call method 'match' (a nil value)
stack traceback:
        ...re/nvim/site/pack/packer/start/nvim-ufo/lua/ufo/fold.lua:51: in function <Anonymous:40>
Press ENTER or type command to continue

The filetype is: NeogitStatus and it does not happen every time.

Sorry, I don't understand what you mean by that.
What I do is simply opening a Telescope window. It does not matter whether that is showing files (:Telescope git_files), help (:Telescope help_tags) or git branches (:Telescope git_branches). I always get the same error message. I have no idea where the root directory / could come into play here.

The message looks like returning by the language server, we should avoid sending invalid buffers, need the condition to filter them out.

Telescope and Neogit buffers are all nofile buftype, I am not sure  whether should skip nofile at ufo,
    require('ufo').setup({
        provider_selector = function(bufnr, filetype)
            if vim.bo[bufnr].bt == 'nofile' then
                return ''
            end
        end,
    })

should work.
If someone master nvim lsp please tell me, need help.

 provider_selector = function(bufnr, filetype)
            if vim.bo[bufnr].bt == 'nofile' then
                return ''
            end
        end,

I still get the error for neogit:
Error executing vim.schedule lua callback: UnhandledPromiseRejection with the reason:
{ ["code"] = 0, ["message"] = stat /home/rz/repos/gitlab.pnet.ch/linux/ces/ces-operator/NeogitStatus: no such file or director }
stack traceback:
        ...re/nvim/site/pack/packer/start/nvim-ufo/lua/ufo/fold.lua:51: in function <Anonymous:40>

And I have the following ufo config:
local status_ok, ufo = pcall(require, "ufo")
if not status_ok then
	vim.notify("ufo (fold) plugin not found!")
	return
end

vim.wo.foldlevel = 99
vim.wo.foldenable = true

local capabilities = vim.lsp.protocol.make_client_capabilities()
capabilities.textDocument.foldingRange = {
	dynamicRegistration = false,
	lineFoldingOnly = true,
}

local handler = function(virtText, lnum, endLnum, width, truncate)
	local newVirtText = {}
	local suffix = ("  %d "):format(endLnum - lnum)
	local sufWidth = vim.fn.strdisplaywidth(suffix)
	local targetWidth = width - sufWidth
	local curWidth = 0
	for _, chunk in ipairs(virtText) do
		local chunkText = chunk[1]
		local chunkWidth = vim.fn.strdisplaywidth(chunkText)
		if targetWidth > curWidth + chunkWidth then
			table.insert(newVirtText, chunk)
		else
			chunkText = truncate(chunkText, targetWidth - curWidth)
			local hlGroup = chunk[2]
			table.insert(newVirtText, { chunkText, hlGroup })
			chunkWidth = vim.fn.strdisplaywidth(chunkText)
			-- str width returned from truncate() may less than 2nd argument, need padding
			if curWidth + chunkWidth < targetWidth then
				suffix = suffix .. (" "):rep(targetWidth - curWidth - chunkWidth)
			end
			break
		end
		curWidth = curWidth + chunkWidth
	end
	table.insert(newVirtText, { suffix, "MoreMsg" })
	return newVirtText
end

-- global handler
ufo.setup({
	fold_virt_text_handler = handler,
	provider_selector = function(bufnr, filetype)
		if vim.bo[bufnr].bt == "nofile" then
			return ""
		end
	end,
})

 provider_selector = function(bufnr, filetype)
            if vim.bo[bufnr].bt == 'nofile' then
                return ''
            end
        end,

I still get the error for neogit:
Error executing vim.schedule lua callback: UnhandledPromiseRejection with the reason:
{ ["code"] = 0, ["message"] = stat /home/rz/repos/gitlab.com/linux/ces/ces-operator/NeogitStatus: no such file or director }
stack traceback:
        ...re/nvim/site/pack/packer/start/nvim-ufo/lua/ufo/fold.lua:51: in function <Anonymous:40>

And I have the following ufo config:
local status_ok, ufo = pcall(require, "ufo")
if not status_ok then
	vim.notify("ufo (fold) plugin not found!")
	return
end

vim.wo.foldlevel = 99
vim.wo.foldenable = true

local capabilities = vim.lsp.protocol.make_client_capabilities()
capabilities.textDocument.foldingRange = {
	dynamicRegistration = false,
	lineFoldingOnly = true,
}

local handler = function(virtText, lnum, endLnum, width, truncate)
	local newVirtText = {}
	local suffix = ("  %d "):format(endLnum - lnum)
	local sufWidth = vim.fn.strdisplaywidth(suffix)
	local targetWidth = width - sufWidth
	local curWidth = 0
	for _, chunk in ipairs(virtText) do
		local chunkText = chunk[1]
		local chunkWidth = vim.fn.strdisplaywidth(chunkText)
		if targetWidth > curWidth + chunkWidth then
			table.insert(newVirtText, chunk)
		else
			chunkText = truncate(chunkText, targetWidth - curWidth)
			local hlGroup = chunk[2]
			table.insert(newVirtText, { chunkText, hlGroup })
			chunkWidth = vim.fn.strdisplaywidth(chunkText)
			-- str width returned from truncate() may less than 2nd argument, need padding
			if curWidth + chunkWidth < targetWidth then
				suffix = suffix .. (" "):rep(targetWidth - curWidth - chunkWidth)
			end
			break
		end
		curWidth = curWidth + chunkWidth
	end
	table.insert(newVirtText, { suffix, "MoreMsg" })
	return newVirtText
end

-- global handler
ufo.setup({
	fold_virt_text_handler = handler,
	provider_selector = function(bufnr, filetype)
		if vim.bo[bufnr].bt == "nofile" then
			return ""
		end
	end,
})

will work now, neogit init too late.

My issue is solved, so would be great to have it merged back into master branch. :+1:

sure

Have handled the buftype, no need to use provider_selector to handle it by yourself.


Always try to join threads in tests

Scope & Purpose
(Please describe the changes in this PR for reviewers, motivation, rationale - mandatory)

[ ] :hankey: Bugfix
[ ] :pizza: New feature
[ ] :fire: Performance improvement
[ ] :hammer: Refactoring/simplification

Checklist

[ ] Tests

[ ] Regression tests
[ ] C++ Unit tests
[ ] integration tests
[ ] resilience tests

[ ] :book: CHANGELOG entry made
[ ] :books: documentation written (release notes, API changes, ...)
[ ] Backports

[ ] Backport for 3.10: (Please link PR)
[ ] Backport for 3.9: (Please link PR)
[ ] Backport for 3.8: (Please link PR)

Related Information
(Please reference tickets / specification / other PRs etc)

[ ] Docs PR:
[ ] Enterprise PR:
[ ] GitHub issue / Jira ticket:
[ ] Design document:

@markuspf

Can you describe the problem you are solving?

Check slack message

Can this be done with less duplication?

Yes but I'm not interested.
Also we don't have std::jthread in libc++

@markuspf

Can you describe the problem you are solving?

Check slack message

Its not useful for this discussion to go across media; I also do not know (or understand) what you mean.

Can this be done with less duplication?

Yes but I'm not interested. Also we don't have std::jthread in libc++

I am not sure what I understand what you mean by "I am not interested"?
You asked for a review, I tried reviewing the code and I do not know what the problem is you're solving (and I do not understand how), but I see that there is a lot of code duplication.
I think that duplication should be avoided.
In particular I am opposed to merging this into the Replication2 code base as is.

@markuspf

I am not sure what I understand what you mean by "I am not interested"?

It's mean I'm not interested to beatiful fix replication2/etc test code.
It means authors of this code wrote it buggy: not-always join threads and I fix only this.
If you want avoid duplication you can create special wrapper, struct for create threads in test.

@markuspf

In particular I am opposed to merging this into the Replication2 code base as is.

But I am opposed to have bugs in code.
I agreed duplication not good but it already was (see for-loops for join threads).
Why it should be fixed by me?

@markuspf

I also do not know (or understand) what you mean.

If we have exception or early return already created threads in vector will be destroyed.
And in that case will be called std::terminate, because we skip to call join or detach

I agreed duplication not good but it already was (see for-loops for join threads). Why it should be fixed by me?
Your comment about duplication should be NIT in my opinion.

Code duplication is a code smell, especially if it occurs more than twice (like here). Yes, this is "just" test code, but test code should have the same quality standards as production code.
In general code that we check code should be cleaner than the code we checked out! So you should try to reduce the existing code duplication instead of adding yet more. Why you? Because you are already touching that code!

But I doesn't have a time to add new files and check everything builds locally or fix it and rerun jenkins few times.
If this fix with a little more duplication then was is worse than have bugs, ok, closed

Reopening;
The problem you spotted (that exceptions could be thrown while joining the threads created in a test) is an actual problem. That is a good spot!
I did not understand previously that you were trying to fix this problem, and it would have been beneficial if you had described it right away in the PR description, so that I can read it and review the PR with that in mind.
Then I spotted a lot of duplicated lines being introduced; that always looks odd to me, so I asked whether this fix can be done without so much duplication. This was also a genuine question: I did not understand the problem, or the solution yet, so I could not in a review tell whether it could be done (within reasonable time).
I was trying to help getting this PR merged by providing a review, that is all.
I will now try to make a suggestion how to improve this code.

I think this is now superseded by #16918 and I will close it again. Thanks for everyone for the contributions.


⚠ Bofrost.at has degraded performance

In 9ca4a87, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 2357 ms

Resolved: Bofrost.at performance has improved in ae56e6a.


Specify ToggleGroup

https://github.com/tudo-aqua/bgw-core/blob/c106f5cea48b6ae11efd346fb7a4342a3b2459a3/src/main/kotlin/tools/aqua/bgw/elements/uielements/RadioButton.kt#L17
For users that don't know the "RadioButton" concept the "ToggleGroup" is unknown aswell.
Explain, how the group and the individual buttons work together.

Same goes for "ToggleButton"


Update Copy-DbaDbTableData output & bugfix

Adding (Source and Destination) Schema to the output; helpful in cases where you have >1 schema with the same named table.
Also fixed -AutoCreateTable -> It was not properly replacing the schema name for the new table if it differed from the source schema.  It was retaining the original source schema due to bug in regex

Type of Change

[X ] Bug fix (non-breaking change, fixes <didn't log bug, found during feature 4786>)
[X ] New feature (non-breaking change, adds functionality #4786 )

Purpose

Fixes AutoCreateTable when the Source schema differs from the desired destination schema.  the current code just used the source due to typo in the regex.
Second change is to add SourceSchema and DestinationSchema to the pscustomobject output of the function so that you can clearly see what schema the tables are in
Approach

Added the values to the output.
Commands to test

$copyArgs = @{

   SqlInstance = $devInstance;
   Database = "Sandbox";
   Table = 'ref.YourTable';
   Destination = $devInstance;
   DestinationDatabase = "Sandbox";
   DestinationTable = 'new.NewName';
   AutoCreateTable = $true;
   KeepIdentity = $true;
   KeepNulls = $true;
   Truncate = $true;
   Verbose = $true;
}

Copy-DbaDbTableData @copyArgs;

Screenshots

excellent, thank you adam!


Update index.md.erb

Fixed redirection url

Hey there @lambasoft. I just realized I hadn't taken the time to close your Pull Request and thank you for fixing the links.
We had pushed your update a few weeks ago.
Thanks! Please don't hesitate to ping us if you don't see any activity on your future pull requests. Although we're pretty busy working on features, we still love our users' feedback!


js_vars doesn't work when units are used and function is style

I'm using Kirki 2.2.4.
js_vars doesn't work when units are used and function is style. It works okay when the function is css.
The following code won't work unless we change function in js_vars to css.
Kirki::add_field( 'config_config', array(
  'settings'          => 'field_test',
  'type'              => 'number',
  'label'             => esc_html__( 'Field Label', 'nighthawk' ),
  'section'           => 'section_test',
  'default'           => 20,
  'sanitize_callback' => 'absint',
  'transport'         => 'postMessage',
  'js_vars'           => array(
    array(
      'element'  => '.element',
      'function' => 'style',
      'property' => 'padding-right',
      'units'    => 'px',
    ),
  ),
  'output'            => array(
    array(
      'element'  => '.element',
      'property' => 'padding-right',
      'units'    => 'px',
    ),
  ),
) );

Thus currently there is no way using js_vars with units for psuedo elements. Hopefully this would be the next version. :)
EDIT: Updated version number to 2.2.4.

---
Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/30649968-js_vars-doesn-t-work-when-units-are-used-and-function-is-style?utm_campaign=plugin&utm_content=tracker%2F2030945&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F2030945&utm_medium=issues&utm_source=github).

That's weird....
$prefix is added on https://github.com/aristath/kirki/blob/2.0.7/includes/class-kirki-customizer-scripts-postmessage.php#L114 so it should be working fine!

@rahulnever2far I just noticed you said in the original post you're using v2.0.7...
Can you please try using the latest stable version (<IP_ADDRESS>) or even better the develop branch?

@aristath The issue persists in v<IP_ADDRESS> and develop branch.

Just wanted to confirm that this issue exists in 2.2.4 as well.

@rahulnever2far can you please try again using the develop branch?
I just refactored the way postMessage works in Kirki so we should be ok now

@aristath Just tried the develop branch. It works now. Thank you so much for fixing this.

Awesome, thanks for testing!


updated changelog for 0.9.17-SNAPSHOT

Pull Request Test Coverage Report for Build 621

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 88.163%

Totals

Change from base Build 618:
0.0%

Covered Lines:
1944

Relevant Lines:
2205

💛  - Coveralls


wasmer: 4.2.6 -> 4.2.7

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://repology.org/project/wasmer/versions.
meta.description for wasmer is: The Universal WebAssembly Runtime
meta.homepage for wasmer is: https://wasmer.io/
Updates performed

Rust version update

To inspect upstream changes
Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 4.2.7 with grep in /nix/store/7ahxl3amvi6mzrc34mx75bch9gqmd7dp-wasmer-4.2.7
found 4.2.7 in filename of file in /nix/store/7ahxl3amvi6mzrc34mx75bch9gqmd7dp-wasmer-4.2.7

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
wasmer

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/7ahxl3amvi6mzrc34mx75bch9gqmd7dp-wasmer-4.2.7 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A wasmer https://github.com/r-ryantm/nixpkgs/archive/72016cd6de69be575867bbd694ec52025646c6cc.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/72016cd6de69be575867bbd694ec52025646c6cc#wasmer

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/7ahxl3amvi6mzrc34mx75bch9gqmd7dp-wasmer-4.2.7
ls -la /nix/store/7ahxl3amvi6mzrc34mx75bch9gqmd7dp-wasmer-4.2.7/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review --extra-nixpkgs-config '{ allowInsecurePredicate = x: true; }' run on x86_64-linux 1

1 package built:

wasmer

Maintainer pings
cc @Br1ght0ne @SCOTT-HAMILTON @NickCao for testing.

$ /nix/store/7ahxl3amvi6mzrc34mx75bch9gqmd7dp-wasmer-4.2.7/bin/wasmer run cowsay test
 ______
< test >
 ------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
               ||----w |
                ||     ||


Support dotted fields in query object in conversion index

The conversion pageview will send the page query across in the shape:
{
  "query": {
    "query": "things I am searching for",
    "workType": ["a", "b"],
    "production.dates.from": "1908"
  }
}

Currently, due to the way that the path_match on dynamic templates, it won't deep match the productions.dates.from and thus not index the data.
I am still researching ☝ but here are some tip-offs:
https://github.com/elastic/elasticsearch/issues/32595
Related to https://github.com/wellcomecollection/wellcomecollection.org/issues/6130

This has been done. We now replace . with _ on the search logger so production.dates.from becomes production_dates_from.
A problem we have had with this is that we have lost data which would have come from those requests. Annoyingly we are quite lax with the data coming in, not needing it to have a specific structure. What happened was:

Segment sent through `{ "query": { "production.dates.from": "1908" } }
The lambda tried to index the above document
We got an class org.elasticsearch.index.mapper.TextFieldMapper cannot be cast to class org.elasticsearch.index.mapper.ObjectMapper error. So basically we were trying to index an object on a text field, so the documents were lost.

This means we should:

start alerting on error
have dashboards for simple things like "amount of times a filter is used"


Adding readonly file on different drive throws exception

Aider version: 0.55.0
Python version: 3.11.9
Platform: Windows-10-10.0.22631-SP0
Python implementation: CPython
Virtual environment: Yes
OS: Windows 10 (64bit)
Git version: git version 2.46.0.windows.1
An uncaught exception occurred:
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "__main__.py", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "main.py", line 694, in main
    coder.run()
  File "base_coder.py", line 729, in run
    user_message = self.get_input()
                   ^^^^^^^^^^^^^^^^
  File "base_coder.py", line 739, in get_input
    read_only_files = [self.get_rel_fname(fname) for fname in self.abs_read_only_fnames]
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "base_coder.py", line 739, in <listcomp>
    read_only_files = [self.get_rel_fname(fname) for fname in self.abs_read_only_fnames]
                       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "base_coder.py", line 1627, in get_rel_fname
    return os.path.relpath(fname, self.root)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen ntpath>", line 766, in relpath
ValueError: path is on mount 'C:', start on mount 'E:'

Thanks for trying aider and filing this issue.
The fix is available in the main branch. You can get it by installing the latest version from github:
aider --install-main-branch

# or...

python -m pip install --upgrade git+https://github.com/paul-gauthier/aider.git

If you have a chance to try it, let me know if it works better for you.

confirmed working on main


exec /opt/poetry-venv/bin/poetry: exec format error

While running CPU Dockerfile on AWS ECS Fargate, getting this poetry exec format error.  The Fargate task definition stays in the Pending state and  I cannot change it to the running state. Any idea why?

Hi @avsavani
It seems the issue is related with the build architecture. I checked previous GitHub action built image and they are on linux/amd64. What is your Fargate cluster architecture? Is it linux/arm64? If yes, the next release will support multi arch with docker buildx.
Greetings,
Ahmet

Hi, I am in the same boat and running it on a Fargate linux. Any idea when is the next release?  Thank you!

Hi @gautampambhar,
I plan to create a new version by tomorrow at the latest.
Greetings,
Ahmet

Hi Just checking in to see if the AWS fargate deployment issue got resolved with the previous release. I am still getting the same error on AWS fargate logs.

Hi @gautampambhar,
I am still implementing the multi architecture version. Sorry for being late. I will let you know when it is ready.
Greetings,
Ahmet


Question: would proposing a 'playgrounds' extension to LSP be sensible?

Hi there,
I'm interested in whether folks think it would be appropriate and worthwhile to propose an an extension to the LSP specifying 'playgrounds' functionality, à la the playgrounds feature in XCode or http://support.hfm.io/1.0/playgrounds.html (the 'Playground editor' and 'Playground Results and Types' components specifically).
The background is that I'd like to start working on 'playgrounds' functionality for Haskell & Elm in Emacs. To me it seems that it would be most constructive to build on LSP so that other languages and editors can more readily implement this set of features.
The best TL;DR I've found of what I mean by 'playgrounds' functionality is the right-hand side of the image on http://support.hfm.io/1.0/playgrounds.html. This image demonstrates adding code to the playground pane and having the results of evaluating that code (and type info if applicable) shown on the corresponding line in the results pane.
Would this, in theory, be a natural fit for the LSP? Or am I better off just focusing on tools specifically for Emacs?
Thanks,
Kieran

Playgrounds seem to be one form of execution, so they would probably fit better in the Debug Protocol which does understand some concepts about launching: https://github.com/Microsoft/vscode-debugadapter-node/tree/master/protocol

That's interesting @mickaelistria, I haven't heard of the Debug Protocol before. Is this something that is exclusive to VS Code at the moment?
You're correct that playgrounds require evaluation of code; in essence they serve a similar purpose as an automated REPL with results being displayed per line. It's possible that this is too far away from the other features specified by the LSP. I'm not sure how close playgrounds are to the debug features currently specified by the Debug Protocol either, so perhaps the more natural delineation would be to have a distinct Playgrounds Protocol.

That's interesting @mickaelistria https://github.com/mickaelistria, I
haven't heard of the Debug Protocol before. Is this something that is
exclusive to VS Code at the moment?

Nope, it's open to any IDE to support it. Eclipse IDE does support it for
instance through the LSP4E project (which also brings support for LSP)

You're correct that playgrounds require evaluation of code; in essence
they serve a similar purpose as an automated REPL with results being
displayed per line. It's possible that this is too far away from the other
features specified by the LSP. I'm not sure how close playgrounds are to
the debug features currently specified by the Debug Protocol either, so
perhaps the more natural delineation would be to have a distinct
Playgrounds Protocol.

Playgrounds are basically consoles with stdin/stdout. What would a
Playground Protocol do more/better than starting a console on the right
REPL applicaiton here?

Nope, it's open to any IDE to support it. Eclipse IDE does support it for instance through the LSP4E project (which also brings support for LSP)

Okay, I'll have a look at whether this functionality would be a natural fit in the Debug Protocol.

Playgrounds are basically consoles with stdin/stdout. What would a Playground Protocol do more/better than starting a console on the right REPL applicaiton here?

I'm unsure whether you're asking:

Why should we have a protocol rather than just implementing an Emacs extension for Haskell/Elm that does thing? The answer to this is the same as the rationale behind the LSP.
Why use a playground when you can just use a REPL? The answer to this is that a REPL is manual while a playground is automated. As soon as you type an expression it is evaluated and the results shown in the playground pane. Think of dot completion: we could accomplish the same end manually by loading a REPL and finding all of the methods that can be invoked on a given object, however it's much more convenient to have those methods shown automatically in your editor.

I agree with @mickaelistria comments.
@trezona-lecomte I will close the issue assuming that it will continue in the debugger protocol land. Please ping if you disagree.


Fix ANR due race accessing DAO in NTP

Task/Issue URL: https://app.asana.com/0/1202552961248957/1207980975183570/f
Description
Ensure only NTP constructs the new tab pixel avoding a race and eventual ANR
Steps to test this PR
Feature 1

[ ]
[ ]

#4863  👈
develop

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @aitorvs and the rest of your teammates on  Graphite


[SPARK-31405][SQL][3.0] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files

What changes were proposed in this pull request?

When reading/writing datetime values that before the rebase switch day, from/to Avro/Parquet files, fail by default and ask users to set a config to explicitly do rebase or not.
Why are the changes needed?

Rebase or not rebase have different behaviors and we should let users decide it explicitly. In most cases, users won't hit this exception as it only affects ancient datetime values.
Does this PR introduce any user-facing change?

Yes, now users will see an error when reading/writing dates before 1582-10-15 or timestamps before 1900-01-01 from/to Parquet/Avro files, with an error message to ask setting a config.
How was this patch tested?

updated tests

cc @HyukjinKwon @MaxGekk

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27269/
Test PASSed.

Test build #122607 has started for PR 28526 at commit 9f14144.

Test build #122607 has finished for PR 28526 at commit 9f14144.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AvroDeserializer(

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122607/
Test FAILed.

Many ..
diff --cc external/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala
index 27206edb287,1d18594fd34..00000000000
--- a/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala
+++ b/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala
@@@ -110,22 -116,13 +116,29 @@@ class AvroDeserializer
        case (LONG, TimestampType) => avroType.getLogicalType match {
          // For backward compatibility, if the Avro type is Long and it is not logical type
          // (the `null` case), the value is processed as timestamp type with millisecond precision.
++<<<<<<< HEAD
 +        case null | _: TimestampMillis if rebaseDateTime => (updater, ordinal, value) =>
 +          val millis = value.asInstanceOf[Long]
 +          val micros = DateTimeUtils.fromMillis(millis)
 +          val rebasedMicros = rebaseJulianToGregorianMicros(micros)
 +          updater.setLong(ordinal, rebasedMicros)
 +        case null | _: TimestampMillis => (updater, ordinal, value) =>
 +          val millis = value.asInstanceOf[Long]
 +          val micros = DateTimeUtils.fromMillis(millis)
 +          updater.setLong(ordinal, micros)
 +        case _: TimestampMicros if rebaseDateTime => (updater, ordinal, value) =>
 +          val micros = value.asInstanceOf[Long]
 +          val rebasedMicros = rebaseJulianToGregorianMicros(micros)
 +          updater.setLong(ordinal, rebasedMicros)
++=======
+         case null | _: TimestampMillis => (updater, ordinal, value) =>
+           val millis = value.asInstanceOf[Long]
+           val micros = DateTimeUtils.millisToMicros(millis)
+           updater.setLong(ordinal, timestampRebaseFunc(micros))
++>>>>>>> fd2d55c9919... [SPARK-31405][SQL] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files
          case _: TimestampMicros => (updater, ordinal, value) =>
            val micros = value.asInstanceOf[Long]
-           updater.setLong(ordinal, micros)
+           updater.setLong(ordinal, timestampRebaseFunc(micros))
          case other => throw new IncompatibleSchemaException(
            s"Cannot convert Avro logical type ${other} to Catalyst Timestamp type.")
        }
diff --cc external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala
index dc232168fd2,21c5dec6239..00000000000
--- a/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala
+++ b/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala
@@@ -155,15 -160,10 +160,22 @@@ class AvroSerializer
        case (TimestampType, LONG) => avroType.getLogicalType match {
            // For backward compatibility, if the Avro type is Long and it is not logical type
            // (the `null` case), output the timestamp value as with millisecond precision.
++<<<<<<< HEAD
 +          case null | _: TimestampMillis if rebaseDateTime => (getter, ordinal) =>
 +            val micros = getter.getLong(ordinal)
 +            val rebasedMicros = rebaseGregorianToJulianMicros(micros)
 +            DateTimeUtils.toMillis(rebasedMicros)
 +          case null | _: TimestampMillis => (getter, ordinal) =>
 +            DateTimeUtils.toMillis(getter.getLong(ordinal))
 +          case _: TimestampMicros if rebaseDateTime => (getter, ordinal) =>
 +            rebaseGregorianToJulianMicros(getter.getLong(ordinal))
 +          case _: TimestampMicros => (getter, ordinal) => getter.getLong(ordinal)
++=======
+           case null | _: TimestampMillis => (getter, ordinal) =>
+             DateTimeUtils.microsToMillis(timestampRebaseFunc(getter.getLong(ordinal)))
+           case _: TimestampMicros => (getter, ordinal) =>
+             timestampRebaseFunc(getter.getLong(ordinal))
++>>>>>>> fd2d55c9919... [SPARK-31405][SQL] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files
            case other => throw new IncompatibleSchemaException(
              s"Cannot convert Catalyst Timestamp type to Avro logical type ${other}")
          }
diff --cc sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 6c18280ce4d,aeaf884c7d1..00000000000
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@@ -2509,57 -2520,71 +2509,83 @@@ object SQLConf
      .booleanConf
      .createWithDefault(false)

++<<<<<<< HEAD
 +  val LEGACY_PARQUET_REBASE_DATETIME_IN_WRITE =
 +    buildConf("spark.sql.legacy.parquet.rebaseDateTimeInWrite.enabled")
 +      .internal()
 +      .doc("When true, rebase dates/timestamps from Proleptic Gregorian calendar " +
 +        "to the hybrid calendar (Julian + Gregorian) in write. " +
 +        "The rebasing is performed by converting micros/millis/days to " +
 +        "a local date/timestamp in the source calendar, interpreting the resulted date/" +
 +        "timestamp in the target calendar, and getting the number of micros/millis/days " +
 +        "since the epoch 1970-01-01 00:00:00Z.")
-       .version("3.0.0")
-       .booleanConf
-       .createWithDefault(false)
-
-   val LEGACY_PARQUET_REBASE_DATETIME_IN_READ =
-     buildConf("spark.sql.legacy.parquet.rebaseDateTimeInRead.enabled")
++=======
+    val LEGACY_INTEGER_GROUPING_ID =
+     buildConf("spark.sql.legacy.integerGroupingId")
        .internal()
-       .doc("When true, rebase dates/timestamps " +
-         "from the hybrid calendar to Proleptic Gregorian calendar in read. " +
-         "The rebasing is performed by converting micros/millis/days to " +
-         "a local date/timestamp in the source calendar, interpreting the resulted date/" +
-         "timestamp in the target calendar, and getting the number of micros/millis/days " +
-         "since the epoch 1970-01-01 00:00:00Z.")
-       .version("3.0.0")
+       .doc("When true, grouping_id() returns int values instead of long values.")
+       .version("3.1.0")
        .booleanConf
        .createWithDefault(false)

-   val LEGACY_AVRO_REBASE_DATETIME_IN_WRITE =
-     buildConf("spark.sql.legacy.avro.rebaseDateTimeInWrite.enabled")
+   val LEGACY_PARQUET_REBASE_MODE_IN_WRITE =
+     buildConf("spark.sql.legacy.parquet.datetimeRebaseModeInWrite")
        .internal()
-       .doc("When true, rebase dates/timestamps from Proleptic Gregorian calendar " +
-         "to the hybrid calendar (Julian + Gregorian) in write. " +
-         "The rebasing is performed by converting micros/millis/days to " +
-         "a local date/timestamp in the source calendar, interpreting the resulted date/" +
-         "timestamp in the target calendar, and getting the number of micros/millis/days " +
-         "since the epoch 1970-01-01 00:00:00Z.")
+       .doc("When LEGACY, Spark will rebase dates/timestamps from Proleptic Gregorian calendar " +
+         "to the legacy hybrid (Julian + Gregorian) calendar when writing Parquet files. " +
+         "When CORRECTED, Spark will not do rebase and write the dates/timestamps as it is. " +
+         "When EXCEPTION, which is the default, Spark will fail the writing if it sees " +
+         "ancient dates/timestamps that are ambiguous between the two calendars.")
++>>>>>>> fd2d55c9919... [SPARK-31405][SQL] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files
        .version("3.0.0")
-       .booleanConf
-       .createWithDefault(false)
+       .stringConf
+       .transform(_.toUpperCase(Locale.ROOT))
+       .checkValues(LegacyBehaviorPolicy.values.map(_.toString))
+       .createWithDefault(LegacyBehaviorPolicy.EXCEPTION.toString)
+
+   val LEGACY_PARQUET_REBASE_MODE_IN_READ =
+     buildConf("spark.sql.legacy.parquet.datetimeRebaseModeInRead")
+       .internal()
+       .doc("When LEGACY, Spark will rebase dates/timestamps from the legacy hybrid (Julian + " +
+         "Gregorian) calendar to Proleptic Gregorian calendar when reading Parquet files. " +
+         "When CORRECTED, Spark will not do rebase and read the dates/timestamps as it is. " +
+         "When EXCEPTION, which is the default, Spark will fail the reading if it sees " +
+         "ancient dates/timestamps that are ambiguous between the two calendars. This config is " +
+         "only effective if the writer info (like Spark, Hive) of the Parquet files is unknown.")
+       .version("3.0.0")
+       .stringConf
+       .transform(_.toUpperCase(Locale.ROOT))
+       .checkValues(LegacyBehaviorPolicy.values.map(_.toString))
+       .createWithDefault(LegacyBehaviorPolicy.EXCEPTION.toString)

-   val LEGACY_AVRO_REBASE_DATETIME_IN_READ =
-     buildConf("spark.sql.legacy.avro.rebaseDateTimeInRead.enabled")
+   val LEGACY_AVRO_REBASE_MODE_IN_WRITE =
+     buildConf("spark.sql.legacy.avro.datetimeRebaseModeInWrite")
        .internal()
-       .doc("When true, rebase dates/timestamps " +
-         "from the hybrid calendar to Proleptic Gregorian calendar in read. " +
-         "The rebasing is performed by converting micros/millis/days to " +
-         "a local date/timestamp in the source calendar, interpreting the resulted date/" +
-         "timestamp in the target calendar, and getting the number of micros/millis/days " +
-         "since the epoch 1970-01-01 00:00:00Z.")
+       .doc("When LEGACY, Spark will rebase dates/timestamps from Proleptic Gregorian calendar " +
+         "to the legacy hybrid (Julian + Gregorian) calendar when writing Avro files. " +
+         "When CORRECTED, Spark will not do rebase and write the dates/timestamps as it is. " +
+         "When EXCEPTION, which is the default, Spark will fail the writing if it sees " +
+         "ancient dates/timestamps that are ambiguous between the two calendars.")
        .version("3.0.0")
-       .booleanConf
-       .createWithDefault(false)
+       .stringConf
+       .transform(_.toUpperCase(Locale.ROOT))
+       .checkValues(LegacyBehaviorPolicy.values.map(_.toString))
+       .createWithDefault(LegacyBehaviorPolicy.EXCEPTION.toString)
+
+   val LEGACY_AVRO_REBASE_MODE_IN_READ =
+     buildConf("spark.sql.legacy.avro.datetimeRebaseModeInRead")
+       .internal()
+       .doc("When LEGACY, Spark will rebase dates/timestamps from the legacy hybrid (Julian + " +
+         "Gregorian) calendar to Proleptic Gregorian calendar when reading Avro files. " +
+         "When CORRECTED, Spark will not do rebase and read the dates/timestamps as it is. " +
+         "When EXCEPTION, which is the default, Spark will fail the reading if it sees " +
+         "ancient dates/timestamps that are ambiguous between the two calendars. This config is " +
+         "only effective if the writer info (like Spark, Hive) of the Avro files is unknown.")
+       .version("3.0.0")
+       .stringConf
+       .transform(_.toUpperCase(Locale.ROOT))
+       .checkValues(LegacyBehaviorPolicy.values.map(_.toString))
+       .createWithDefault(LegacyBehaviorPolicy.EXCEPTION.toString)

    /**
     * Holds information about keys that have been deprecated.
@@@ -3139,9 -3166,7 +3165,13 @@@ class SQLConf extends Serializable wit

    def csvFilterPushDown: Boolean = getConf(CSV_FILTER_PUSHDOWN_ENABLED)

++<<<<<<< HEAD
 +  def parquetRebaseDateTimeInReadEnabled: Boolean = {
 +    getConf(SQLConf.LEGACY_PARQUET_REBASE_DATETIME_IN_READ)
 +  }
++=======
+   def integerGroupingIdEnabled: Boolean = getConf(SQLConf.LEGACY_INTEGER_GROUPING_ID)
++>>>>>>> fd2d55c9919... [SPARK-31405][SQL] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files

    /** ********************** SQLConf functionality methods ************ */

diff --cc sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
index 11ce11dd721,3e409ab9a50..00000000000
--- a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
+++ b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
@@@ -324,20 -352,20 +352,32 @@@ public class VectorizedColumnReader
              }
            }
          } else if (originalType == OriginalType.TIMESTAMP_MILLIS) {
-           if (rebaseDateTime) {
+           if ("CORRECTED".equals(datetimeRebaseMode)) {
              for (int i = rowId; i < rowId + num; ++i) {
                if (!column.isNullAt(i)) {
++<<<<<<< HEAD
 +                long julianMillis = dictionary.decodeToLong(dictionaryIds.getDictId(i));
 +                long julianMicros = DateTimeUtils.fromMillis(julianMillis);
 +                long gregorianMicros = RebaseDateTime.rebaseJulianToGregorianMicros(julianMicros);
 +                column.putLong(i, gregorianMicros);
++=======
+                 long gregorianMillis = dictionary.decodeToLong(dictionaryIds.getDictId(i));
+                 column.putLong(i, DateTimeUtils.millisToMicros(gregorianMillis));
++>>>>>>> fd2d55c9919... [SPARK-31405][SQL] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files
                }
              }
            } else {
+             final boolean failIfRebase = "EXCEPTION".equals(datetimeRebaseMode);
              for (int i = rowId; i < rowId + num; ++i) {
                if (!column.isNullAt(i)) {
++<<<<<<< HEAD
 +                long gregorianMillis = dictionary.decodeToLong(dictionaryIds.getDictId(i));
 +                column.putLong(i, DateTimeUtils.fromMillis(gregorianMillis));
++=======
+                 long julianMillis = dictionary.decodeToLong(dictionaryIds.getDictId(i));
+                 long julianMicros = DateTimeUtils.millisToMicros(julianMillis);
+                 column.putLong(i, rebaseMicros(julianMicros, failIfRebase));
++>>>>>>> fd2d55c9919... [SPARK-31405][SQL] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files
                }
              }
            }
@@@ -485,27 -514,29 +526,38 @@@
        defColumn.readLongs(
          num, column, rowId, maxDefLevel, (VectorizedValuesReader) dataColumn);
      } else if (originalType == OriginalType.TIMESTAMP_MICROS) {
-       if (rebaseDateTime) {
-         defColumn.readLongsWithRebase(
-           num, column, rowId, maxDefLevel, (VectorizedValuesReader) dataColumn);
-       } else {
+       if ("CORRECTED".equals(datetimeRebaseMode)) {
          defColumn.readLongs(
            num, column, rowId, maxDefLevel, (VectorizedValuesReader) dataColumn);
+       } else {
+         boolean failIfRebase = "EXCEPTION".equals(datetimeRebaseMode);
+         defColumn.readLongsWithRebase(
+           num, column, rowId, maxDefLevel, (VectorizedValuesReader) dataColumn, failIfRebase);
        }
      } else if (originalType == OriginalType.TIMESTAMP_MILLIS) {
-       if (rebaseDateTime) {
+       if ("CORRECTED".equals(datetimeRebaseMode)) {
          for (int i = 0; i < num; i++) {
            if (defColumn.readInteger() == maxDefLevel) {
++<<<<<<< HEAD
 +            long micros = DateTimeUtils.fromMillis(dataColumn.readLong());
 +            column.putLong(rowId + i, RebaseDateTime.rebaseJulianToGregorianMicros(micros));
++=======
+             column.putLong(rowId + i, DateTimeUtils.millisToMicros(dataColumn.readLong()));
++>>>>>>> fd2d55c9919... [SPARK-31405][SQL] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files
            } else {
              column.putNull(rowId + i);
            }
          }
        } else {
+         final boolean failIfRebase = "EXCEPTION".equals(datetimeRebaseMode);
          for (int i = 0; i < num; i++) {
            if (defColumn.readInteger() == maxDefLevel) {
++<<<<<<< HEAD
 +            column.putLong(rowId + i, DateTimeUtils.fromMillis(dataColumn.readLong()));
++=======
+             long julianMicros = DateTimeUtils.millisToMicros(dataColumn.readLong());
+             column.putLong(rowId + i, rebaseMicros(julianMicros, failIfRebase));
++>>>>>>> fd2d55c9919... [SPARK-31405][SQL] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files
            } else {
              column.putNull(rowId + i);
            }
diff --cc sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala
index 08fbca2995c,201ee16faeb..00000000000
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala
@@@ -291,19 -289,10 +289,26 @@@ private[parquet] class ParquetRowConver
          }

        case TimestampType if parquetType.getOriginalType == OriginalType.TIMESTAMP_MILLIS =>
++<<<<<<< HEAD
 +        if (rebaseDateTime) {
 +          new ParquetPrimitiveConverter(updater) {
 +            override def addLong(value: Long): Unit = {
 +              val micros = DateTimeUtils.fromMillis(value)
 +              val rebased = rebaseJulianToGregorianMicros(micros)
 +              updater.setLong(rebased)
 +            }
 +          }
 +        } else {
 +          new ParquetPrimitiveConverter(updater) {
 +            override def addLong(value: Long): Unit = {
 +              updater.setLong(DateTimeUtils.fromMillis(value))
 +            }
++=======
+         new ParquetPrimitiveConverter(updater) {
+           override def addLong(value: Long): Unit = {
+             val micros = DateTimeUtils.millisToMicros(value)
+             updater.setLong(timestampRebaseFunc(micros))
++>>>>>>> fd2d55c9919... [SPARK-31405][SQL] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files
            }
          }

diff --cc sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetWriteSupport.scala
index e367b9cc774,6c333671d59..00000000000
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetWriteSupport.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetWriteSupport.scala
@@@ -187,24 -198,15 +198,29 @@@ class ParquetWriteSupport extends Write
                buf.order(ByteOrder.LITTLE_ENDIAN).putLong(timeOfDayNanos).putInt(julianDay)
                recordConsumer.addBinary(Binary.fromReusedByteArray(timestampBuffer))

-           case SQLConf.ParquetOutputTimestampType.TIMESTAMP_MICROS if rebaseDateTime =>
-             (row: SpecializedGetters, ordinal: Int) =>
-               val rebasedMicros = rebaseGregorianToJulianMicros(row.getLong(ordinal))
-               recordConsumer.addLong(rebasedMicros)
-
            case SQLConf.ParquetOutputTimestampType.TIMESTAMP_MICROS =>
              (row: SpecializedGetters, ordinal: Int) =>
++<<<<<<< HEAD
 +              recordConsumer.addLong(row.getLong(ordinal))
 +
 +          case SQLConf.ParquetOutputTimestampType.TIMESTAMP_MILLIS if rebaseDateTime =>
 +            (row: SpecializedGetters, ordinal: Int) =>
 +              val rebasedMicros = rebaseGregorianToJulianMicros(row.getLong(ordinal))
 +              val millis = DateTimeUtils.toMillis(rebasedMicros)
 +              recordConsumer.addLong(millis)
 +
 +          case SQLConf.ParquetOutputTimestampType.TIMESTAMP_MILLIS =>
 +            (row: SpecializedGetters, ordinal: Int) =>
 +              val millis = DateTimeUtils.toMillis(row.getLong(ordinal))
++=======
+               val micros = row.getLong(ordinal)
+               recordConsumer.addLong(timestampRebaseFunc(micros))
+
+           case SQLConf.ParquetOutputTimestampType.TIMESTAMP_MILLIS =>
+             (row: SpecializedGetters, ordinal: Int) =>
+               val micros = row.getLong(ordinal)
+               val millis = DateTimeUtils.microsToMillis(timestampRebaseFunc(micros))
++>>>>>>> fd2d55c9919... [SPARK-31405][SQL] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files
                recordConsumer.addLong(millis)
          }

Yea, all the conflicts are caused by the different name of DateTimeUtils.toMillis in 3.0

retest this please

Test build #122616 has started for PR 28526 at commit 9f14144.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27278/
Test PASSed.

Test build #122616 has finished for PR 28526 at commit 9f14144.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AvroDeserializer(

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122616/
Test FAILed.

FYI, the dependency failure will be fixed by the following.

https://github.com/apache/spark/pull/28532

retest this please

Test build #122646 has started for PR 28526 at commit 9f14144.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27308/
Test PASSed.

Test build #5006 has started for PR 28526 at commit 9f14144.

Test build #5007 has started for PR 28526 at commit 9f14144.

Test build #5008 has started for PR 28526 at commit 9f14144.

FYI,tThis seems not completely fixed yet due to the cached .m2/repository, see also https://issues.apache.org/jira/browse/SPARK-31693?focusedCommentId=17107856&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17107856

Test build #122646 has finished for PR 28526 at commit 9f14144.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AvroDeserializer(

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122646/
Test FAILed.

Test build #5006 has finished for PR 28526 at commit 9f14144.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AvroDeserializer(

Ya. The failure jobs are on the same broken Jenkins machine, amp-jenkins-worker-05.

Test build #5008 has finished for PR 28526 at commit 9f14144.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AvroDeserializer(

Test build #5007 has finished for PR 28526 at commit 9f14144.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AvroDeserializer(

@cloud-fan . You can see HiveExternalCatalogVersionsSuite in the above two runs.
cc @viirya .

Test build #5013 has started for PR 28526 at commit 9f14144.

Test build #5014 has started for PR 28526 at commit 9f14144.

Test build #5015 has started for PR 28526 at commit 9f14144.

Test build #5016 has started for PR 28526 at commit 9f14144.

Test build #5013 has finished for PR 28526 at commit 9f14144.

This patch fails build dependency tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
class AvroDeserializer(

Test build #5014 has finished for PR 28526 at commit 9f14144.

This patch fails build dependency tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
class AvroDeserializer(

Test build #5016 has finished for PR 28526 at commit 9f14144.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AvroDeserializer(

Test build #5015 has finished for PR 28526 at commit 9f14144.

This patch fails build dependency tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
class AvroDeserializer(

Test build #5017 has started for PR 28526 at commit 9f14144.

Test build #5018 has started for PR 28526 at commit 9f14144.

Test build #5019 has started for PR 28526 at commit 9f14144.

Test build #5020 has started for PR 28526 at commit 9f14144.

Test build #5021 has started for PR 28526 at commit 9f14144.

Test build #5022 has started for PR 28526 at commit 9f14144.

Test build #5023 has started for PR 28526 at commit 9f14144.

Test build #5024 has started for PR 28526 at commit 9f14144.

Test build #5017 has finished for PR 28526 at commit 9f14144.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AvroDeserializer(

Test build #5018 has finished for PR 28526 at commit 9f14144.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AvroDeserializer(

Test build #5019 has finished for PR 28526 at commit 9f14144.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AvroDeserializer(

Test build #5020 has finished for PR 28526 at commit 9f14144.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AvroDeserializer(

Test build #5024 has finished for PR 28526 at commit 9f14144.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AvroDeserializer(

retest this please

Test build #122666 has started for PR 28526 at commit 9f14144.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27328/
Test PASSed.

Test build #122666 has finished for PR 28526 at commit 9f14144.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AvroDeserializer(

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122666/
Test FAILed.

Test build #5025 has started for PR 28526 at commit 9f14144.

Test build #5025 has finished for PR 28526 at commit 9f14144.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AvroDeserializer(

Test build #5022 has finished for PR 28526 at commit 9f14144.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AvroDeserializer(

Test build #5023 has finished for PR 28526 at commit 9f14144.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AvroDeserializer(

Test build #5021 has finished for PR 28526 at commit 9f14144.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AvroDeserializer(

Test build #5031 has started for PR 28526 at commit 9f14144.

Test build #5032 has started for PR 28526 at commit 9f14144.

Test build #5033 has started for PR 28526 at commit 9f14144.

Test build #5034 has started for PR 28526 at commit 9f14144.

Test build #5035 has started for PR 28526 at commit 9f14144.

Test build #5036 has started for PR 28526 at commit 9f14144.

Test build #5037 has started for PR 28526 at commit 9f14144.

Test build #5032 has finished for PR 28526 at commit 9f14144.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AvroDeserializer(

Test build #5034 has finished for PR 28526 at commit 9f14144.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AvroDeserializer(

Test build #5036 has finished for PR 28526 at commit 9f14144.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AvroDeserializer(

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27364/
Test PASSed.

Test build #122705 has started for PR 28526 at commit 9f14144.

Test build #122705 has finished for PR 28526 at commit 9f14144.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AvroDeserializer(

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122705/
Test FAILed.

retest this please

Test build #122724 has started for PR 28526 at commit 9f14144.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27384/
Test PASSed.

Test build #122724 has finished for PR 28526 at commit 9f14144.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AvroDeserializer(

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122724/
Test PASSed.

finally, jenkins pass... merging to 3.0!


Add Brasil API

Open source project that aims to provide access to resources data available on Brazil.
https://github.com/filipedeschamps/BrasilAPI

@SEdilson Can you open a PR to include it?

Absolutely @felipefialho, I'll do it right away.


Core Window System Decorations is not supported or broken

Describe the bug
There is multiple bugs when system decorations is set to None with CoreWindow class at least on Win10.
The border is showed with a large band, size of the window is ignored, resizing the window is completely broken.
Screenshots

Desktop/Platform (please complete the following information):

OS: Windows 10
FluentAvalonia Version: 1.4.1
Avalonia Version: 0.10.17

CoreWindow doesn't support messing with the system decorations by design. It has custom logic to do its own thing for rendering the titlebar and resizing areas so its closer to a UWP app which doesn't work with other custom window frame stuff. Use a normal Avalonia Window if you want control over this stuff.


Change of time set event to 'all day' causes error

Describe the bug
When I create an event with a set time, like 12.00-13.00 in a specified day, and then open the event, setting the 'allday' to true, save it again I get "Device calendar plugin ran into an issue. Platform specific exception [500], with message :"The start date must be before the end date.", has been thrown." even though when I check the start and end date, this error statement is not true.
To Reproduce
See above.
Expected behavior
Edit: No error should be thrown. It should be saved as an all day event. If you later open the same all day event and remove the all day flag, then the time set from the beginning should be the shown.
Device(s) tested
This can be very important as not all device vendors do calendar in the same way.

Device: iPhone 13 mini
OS: iOS 16.1.2
Plugin version  4.3.1

Flutter doctor
flutter doctor -v
[✓] Flutter (Channel beta, 3.9.0-0.2.pre, on macOS 13.2.1 22D68 darwin-arm64, locale sv-SE)
• Flutter version 3.9.0-0.2.pre on channel beta at /Library/flutter
• Upstream repository https://github.com/flutter/flutter.git
• Framework revision 0df8557c56 (3 veckor sedan), 2023-03-23 22:02:01 -0500
• Engine revision 1cc4b2d280
• Dart version 3.0.0 (build 3.0.0-290.3.beta)
• DevTools version 2.22.2
[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
• Android SDK at /Users/Joakim/Library/Android/sdk
• Platform android-33, build-tools 30.0.3
• Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
• Java version OpenJDK Runtime Environment (build 11.0.15+0-b2043.56-8887301)
• All Android licenses accepted.
[✓] Xcode - develop for iOS and macOS (Xcode 14.3)
• Xcode at /Applications/Xcode.app/Contents/Developer
• Build 14E222b
• CocoaPods version 1.12.0
[✓] Chrome - develop for the web
• Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
[✓] Android Studio (version 2022.1)
• Android Studio at /Applications/Android Studio.app/Contents
• Flutter plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/6351-dart
• Java version OpenJDK Runtime Environment (build 11.0.15+0-b2043.56-8887301)
[✓] Connected device (3 available)
• EnSE (mobile)   • XXXXXXXX-XXXXXXXXXXXXXX • ios            • iOS 16.1.2 20B110
• macOS (desktop) • macos                     • darwin-arm64   • macOS 13.2.1 22D68 darwin-arm64
• Chrome (web)    • chrome                    • web-javascript • Google Chrome 112.0.5615.49
[✓] Network resources
• All expected network resources are available.
• No issues found!
Additional context
Example code:
Event ev = Event(DM.defaultWriteCalendar?.id,
title: 'Test',
start: TZDateTime(DM.location, 2023, 4, 12, 12),
end: TZDateTime(DM.location, 2023, 4, 12, 13)
);
var createEventResult = await DM.deviceCalendarPlugin
.createOrUpdateEvent(ev);
ev.eventId = createEventResult!.data;
if (createEventResult!.isSuccess) {
print('Save successful');
} else {
String errors =
createEventResult.errors[0].errorMessage;
for (int i = 1; i < createEventResult.errors.length; i++) {
errors =
'\n- ' + createEventResult.errors[i].errorMessage;
}
print('Errors: ' + errors);
}
await Future.delayed(Duration(seconds: 2));
ev.allDay = true;
createEventResult = await DM.deviceCalendarPlugin
.createOrUpdateEvent(ev);
if (createEventResult!.isSuccess) {
print('Save successful');
} else {
String errors =
createEventResult.errors[0].errorMessage;
for (int i = 1; i < createEventResult.errors.length; i++) {
errors =
'\n- ' + createEventResult.errors[i].errorMessage;
}
print('Errors: ' + errors);
}

Now this works without me changing the code. Something done in the latest update? Closing the issue.


Question groups

Add the ability to group questions into categories. These provide higher level aggregation levels. As used in SDQ.

 Groups of questions with sum or mean aggregations

Each should have a name and aggregation function.

Should not impact the current question set structure


Fix default 'settings.py' path

Use __file__ to get the absolute path of the cli.py path and define the default settings.py path
according to it.

Closes #238

@jenkins-cscs retry all


wrong error when checking derived class members

Hi
I get an error (problem) working with an Angular2 project:
[Angular] Identifier 'myService' refers to a private member of the component
Scenario:
I have a derived service class which inherits from a base class with a public method. So the error is detected by vscode when I declare this in a html file:
<myComponent [show]="myService.loading()"></myComponent>
It seems vscode can't reach base class method, despite this method is declared as public.
Cheers!!

mmm, I think this is caused by "Angular Language Service" plugin


rkt: add exec/enter subcommand

Discussed in this rocket-dev thread - we should add a rkt exec and/or rkt enter command that allows users to either exec a specified binary within an image, or do an nsenter-like operation on a running container.

@vcaputo is there anything remaining here? if not please close it out.

implemented in https://github.com/coreos/rocket/pull/330


cannot run accio cli

my ubuntu is Ubuntu 18.04.5 LTS
why?

can you suply the source code of accio cli? i guess thart os is diff and make it not work

can you suply the source code of accio cli? i guess thart os is diff and make it not work

Hi @madman188, thank you for reporting this issue.
We are currently in the process of reorganizing the source code for accio-cli. It will be released in the next few days.


Modpack Version
1.6.0及之前版本
Describe your issue
机械炸锅和熔化器识别不了加了吸管的燃烧室
Have you modified the modpack?
No
User Modifications
No response
Did the issue happen in singleplayer or on a server?
Singleplayer

mod原因无法修复(<_<


Lock to the pre-promise version of documentation

Fixes #13 temporarily.

Any updates on this? Any way I can help get it merged?

I'll merge this as a stopgap and start working on getting compatibility with the modern version up to speed. Thanks for the PR!

Thanks @tmcw


pre-linear Gaussian dropout from Srivastava

According to Section 5 of the paper, the correlated weight noise from section 3.2 corresponds to the type of Gaussian dropout described in Srivastava's 2014 paper. Unsure if this is correct, but depends on the meaning of "activation":

This new form of dropout amounts to adding a Gaussian distributed random variable with zero mean and activation equal to the standard deviation of the unit.

Usually the convention is that activation refers to the pre-nonlinearity value (right?), while the situation described in section 3.2 is post-nonlinearity.
Regardless though, I'll just take the paper at it's word, and Srivastava Gaussian dropout will be post-nonlinearity; while Wang is pre. Otherwise, Srivastava and Wang Gaussian dropout end up being pretty much the same (if we take Wang's to be parameterized as in Appendix B of the paper and not the Gaussian approximation to Bernoulli dropout in the 2011 paper (which I originally coded)).

The dropout implemented matches the paper, even if this ambiguity means it doesn't match what Srivastava implemented.


No way to add rules for premium account with an error

Storage Explorer Version: 1.1.0 RC
Platform: Windows/Linux/Mac
Architecture: x64
Build Number: 20180503.1
Commit: 253b437b
Regression From: Not a regression
Steps to Reproduce:

Launch Storage Explorer and expand Storage Accounts.
Right click 'Blob Containers' node under a premium storage account and select "Configure CORS Settings...".
On the 'CORS' Settings dialog, click 'Save' directly. Verify whether save successfully or not.

Expected Experience:
Save successfully.
Actual Experience:
Pop up an error dialog. You can see the screenshot.

Same root as #133. Already fixed.

The issue #133 has been fixed. But we have no way to add rules for premium account with the posted error.
Could you help to take look at it?

This issue still reproduces on 20180704.1. But it displays as the below different error

CORS is not supported for Premium Storage accounts (see documentation).
We just need to remove this action.

Hi @craxal , The action 'Configure CORS Settings...' still shows on the premium context menu with build 20180724.5. So we re-open this issue.

For the original premium account, the action 'Configure CORS Setting...' is removed. But for one attached premium account, the action still displays. So we re-opened this issue.

The visibility of this option depends on information that can only be retrieved from the management endpoint, which we don't have for attached resources. This is not a high-priority issue, so I'm closing as Won't Fix.


Libretro : controls nonresponsive when polling behavior set to "Normal"

Hi,
If you change the latency polling in RA from Settings > Input > Polling behavior to Normal, controls are not responsive.
The other Arduboy Libretro core suffered the same issue which was fixed with this commit.
Thank you.

Thanks a bunch for this, I was able to reproduce the problem, and it should be fixed in master with #70.

Thank you, I confirm it is fixed with this commit.


NoMethodError: undefined method `current=' for Locale:Module

I'm getting this error when trying to run bundle exec rake --trace translation:init:
** Invoke translation:init (first_time)
** Invoke environment (first_time)
** Execute environment
rake aborted!
NoMethodError: undefined method `current=' for Locale:Module
/usr/local/opt/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/gettext-3.2.2/lib/gettext.rb:274:in `set_current_locale'
/usr/local/opt/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/translation-1.9/lib/translation_io/railtie.rb:28:in `locale='
/usr/local/opt/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/i18n-0.7.0/lib/i18n.rb:35:in `locale='
/usr/local/opt/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/translation-1.9/lib/translation_io/railtie.rb:16:in `block in <class:Railtie>'
/usr/local/opt/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/lazy_load_hooks.rb:36:in `execute_hook'
/usr/local/opt/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/lazy_load_hooks.rb:45:in `block in run_load_hooks'
/usr/local/opt/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/lazy_load_hooks.rb:44:in `each'
/usr/local/opt/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/lazy_load_hooks.rb:44:in `run_load_hooks'
/usr/local/opt/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/application/finisher.rb:62:in `block in <module:Finisher>'
/usr/local/opt/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/initializable.rb:30:in `instance_exec'
/usr/local/opt/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/initializable.rb:30:in `run'
/usr/local/opt/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/initializable.rb:55:in `block in run_initializers'
...

I tried disabling gettext, but it had no effect.
Ideas?
Ruby: 2.3.3
Rails: 4.2.8

Hi,
Would you give us your Gemfile.lock and your translation initializer (without the API key of course).
You disabled gettext this way ? https://translation.io/blog/disable-gettext-in-your-project

Here's the code: https://github.com/churchio/onebody/compare/master...translation-io

I found out the issue here. It's this file that was causing some trouble: https://github.com/churchio/onebody/blob/translation-io/lib/locale.rb
If you rename it (and rename the call here: https://github.com/churchio/onebody/blob/translation-io/config/initializers/_libs.rb) it will be fine. After the changes, I was able to successfully sync your translations with Translation.io.
FIY, Gettext was requiring the same name (we need gettext for internal stuff when initialising or syncing) and your file was loaded instead of the one of Locale gem.
PS : initialising your project may take some time, be patient. Once it's done, syncing will be faster.

@MichaelHoste thank you!!!!


check empty array and trasform it in set as null

Q
A

Bug fix?
[X]

New feature?
[ ]

New sample?
[ ]

Related issues?
N.A.

What's in this Pull Request?
Fixes an error raised by SharePoint update API when updating an item that contains a field that requires an array. In this UC SharePoint requires a null value instead of an empty array.

@SamueleFurnari , Thank you for the fix.


Adding information about Spring Boot Endpoints to the README

Coverage remained the same at 79.708% when pulling f475f85932a97c487ae86c4ddd48abe330947d2f on readme-url into 9acc0491baf2abd99622c75d7bbf49d962ed7b73 on master.


[Bug]: npm i fails for new React/TypeScript/ESLint/Prettier project

Version
System:
    OS: macOS 14.2.1
    CPU: (8) x64 Intel(R) Core(TM) i7-1068NG7 CPU @ 2.30GHz
    Memory: 702.71 MB / 32.00 GB
    Shell: 5.9 - /bin/zsh
  Browsers:
    Brave Browser: <IP_ADDRESS>
    Chrome: 128.0.6613.120
    Safari: 17.2.1
  npmPackages:
    N/A due to this bug
  Node version: v20.11.1
  NPM version: 10.2.4

Details
There seems to be a problem with npm i when you try to create a React/TypeScript/ESLint/Prettier project. However, if I do the install without checking "ESLint" and "Prettier", I am able to NPM install successfully.
Reproduce link
N/A
Reproduce Steps
❯ npm create rsbuild@latest

◆  Create Rsbuild Project
│
◇  Project name or path
│  portal
│
◇  Select framework
│  React
│
◇  Select language
│  TypeScript
│
◇  Select additional tools (Use <space> to select, <enter> to continue)
│  Add ESLint for code linting, Add Prettier for code formatting
│
◇  Next steps ──╮
│               │
│  cd portal    │
│  npm i        │
│  npm run dev  │
│               │
├───────────────╯
│
└  Done.

Then I do the "Next steps" mentioned above and get:
❯ npm i
npm ERR! code ERESOLVE
npm ERR! ERESOLVE unable to resolve dependency tree
npm ERR!
npm ERR! While resolving<EMAIL_ADDRESS>npm ERR! Found<EMAIL_ADDRESS>npm ERR! node_modules/eslint
npm ERR!   dev<EMAIL_ADDRESS>from the root project
npm ERR!
npm ERR! Could not resolve dependency:
npm ERR! peer<EMAIL_ADDRESS>|| ^4.0.0 || ^5.0.0 || ^6.0.0 || ^7.0.0 || ^8.0.0-0" from<EMAIL_ADDRESS>npm ERR! node_modules/eslint-plugin-react-hooks
npm ERR!   dev<EMAIL_ADDRESS>from the root project
npm ERR!
npm ERR! Fix the upstream dependency conflict, or retry
npm ERR! this command with --force or --legacy-peer-deps
npm ERR! to accept an incorrect (and potentially broken) dependency resolution.
npm ERR!
npm ERR!
npm ERR! For a full report see:
npm ERR! /Users/me/.npm/_logs/2024-09-09T20_04_39_034Z-eresolve-report.txt

npm ERR! A complete log of this run can be found in: /Users/me/.npm/_logs/2024-09-09T20_04_39_034Z-debug-0.log

Wait for eslint-plugin-react-hooks to release 5.0 stable, or you can use pnpm instead.
https://github.com/web-infra-dev/rspack/issues/7802


Package phpunit/php-token-stream is abandoned, you should avoid using it. No replacement was suggested.

Package phpunit/php-token-stream is abandoned, you should avoid using it. No replacement was suggested.

Please look at these comments.
I think it's about the warning message for phpunit/phpunit and just ignore them.

ho to overcome "Package phpunit/php-token-stream is abandoned, you should avoid using it. No replacement was suggested"


setScriptTimeout (allScriptsTimeout) fails in SauceLabs (http://saucelabs.com) and there is no option to disable it

Bug report

Node Version: 8.11.3

Protractor Version: 5.3.0

Angular Version: 1.3.0

Browser(s): chrome

Operating System and Version windows 10

Your protractor configuration file
config.txt

A relevant example test

Output from running the test
 {"message":"unknown error: 'ms' must be a double\n (Session info: chrome=67.0.3396.62)\n (Driver info: chromedriver=2.39.562718 (9a2698cba08cf5a471a29d30c8b3e12becabb0e9),platform=Windows NT 6.3.9600 x86_64)"}

Steps to reproduce the bug
-- Just run any test in SauceLabs (http://saucelabs.com)

Protractor tries to set setScriptTimeout using value allScriptsTimeout. In SauceLabs, that causes the error reported above, so besides not working, it consumes time from the test run.
https://github.com/angular/protractor/blob/0fbc2c0ac12992bd61712188a96aef6684bef0c1/lib/runner.ts#L281
It could both:

convert the value to double to make it work
provide an option to disable this command completely, so I can save this time if I want to

The issue is that protractor runs setScriptTimeout, even if I don't set any value, using the default value 0.
Have a look at the default behavior on the code line I've mentioned, it's executed regardless if the parameter has been set or not.

@renatoch did you find a fix or workaround for this issue?

@joelgrim Unfortunately not, I ended up accepting the time loss and ignoring the error messages...


⚠ Turnos 24x7 has degraded performance

In 729ca52, Turnos 24x7 (https://reservas-dev.alternativasinteligentes.com/) experienced degraded performance:

HTTP code: 200
Response time: 939 ms

Resolved: Turnos 24x7 performance has improved in 050f7a9.


docs: update readme.md

Update code example in plugins/api.ts from app to nuxtApp

Hello, can you please explain why it should be preferred using nuxtApp rather then app? I don't have any issue on merging your request if there is any best practice behind.
Thank you.

Hello, sorry for the inconvenience, it was code from my VSCode, it put
nuxtApp in parameter and so I thought it was an error. It wasn't until
tomorrow that I realised it wasn't an error in the readme-md file. So I
deleted the pull-request. I'm sorry.
Le mer. 24 juil. 2024 à 07:20, sot1986 @.***> a écrit :

Hello, can you please explain why it should be preferred using nuxtApp
rather then app? I don't have any issue on merging your request if there is
any best practice behind.
Thank you.
—
Reply to this email directly, view it on GitHub
https://github.com/sot1986/nuxt-precognition/pull/1#issuecomment-2246923076,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ATYNQMRJPJZJ7CJQBTLPBBDZN42SFAVCNFSM6AAAAABLHCS776VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBWHEZDGMBXGY
.
You are receiving this because you modified the open/close state.Message
ID: @.***>