Fix SusPay condition check in Finish:

On a SusPayFinish, an additional check is added to make sure that the presented digest matches the digest in the SusPay ledger entry. This includes a regression unit test.
Reviewers: @rec @ximinez

Looks good. :+1:

Well, digest isn't a useful name - what would be interesting is whose digest it is.
Another option is
if (auto const& ledgerDigest = (*slep)[~sfDigest]))
    if (! (ctx_.tx[~sfMethod] && ledgerDigest == ctx_.tx[~sfDigest]))
        return tecNO_PERMISSION;

making it clearer that you're comparing the ledger digest with the transaction digest...
but it's not important.  :+1:

slep is completely clear: "S"erialized "L"edger "Entry" of the "P"ayment (versus just sle, which by convention is the account root for any function that loads 2 or more ledger entries where one of them is the account root). I admit that sle at first was not agreeable, but considering both the frequency which it comes up in business logic, and that it is a fundamental building block of rippled, it makes sense to presume that the reader is aware.

slep is completely clear:

It's completely clear to you because you invented all these acronyms.  For me at least, each time I see one I have to fill it in.  When it's an existing English word, like sled, I block a little longer.

sle, which by convention is the account root for any function that loads 2 or more ledger entries where one of them is the account root

I didn't know that!  "two or more ledger entries where one of them is the account root"...
How do you expect non-ripple engineers to ever read - and audit - this code when there is such a huge cognitive burden required to understand even the simplest parts?
The reason I keep pushing back on this is because

I find the code extremely hard to read and understand and it seems to be getting worse.

We keep finding errors caused by rippled engineers misreading the code.  The example that stood out for me was the test that didn't test anything - five engineers participated in that review, yet none of them saw it.

Thirty years of experience has shown me time and time again that the simplest and most readable code (and I mean readable to anyone) is usually also the most correct, reliable, and (most of the time) the most performant.

(If I were naming this stuff, it'd be just LedgerEntry - I never understood why you needed to advertise the serialization, if you couldn't serialize it, you couldn't put it in a ledger, could you? - and the variable would be paymentLedgerEntry - conceivably paymentEntry.)
Anyway, none of this is "actionable" - I already :+1: 'ed this - but you should understand where I'm coming from here.
I will do anything to avoid risk, and to my mind, it is a risk to rely on engineers to understand "sle, which by convention is the account root for any function that loads 2 or more ledger entries where one of them is the account root" - and to correctly reason with that understanding each time they see the variable name.

@rec I didn't pick SLE, it was there already and I considered changing it but ultimately just accepted it since David understands it, its reasonably short (SLE is currently an alias for STLedgerEntry), and the rename would be largely gratuitous. I don't think that the STObject` class hierarchy (which we now call "st") is particularly bad although there is certainly room for tidying.


Avoid creating DNS resolver for DNS filter in data plane

Create DNS resolver for DNS filter in control plane config parsing, not in data plane.
Signed-off-by: Yanjun Xiang<EMAIL_ADDRESS>
Commit Message:
Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]

/assign @yanavlasov @botengyao

This change is causing a crash due to thread safety check.
"
[2022-06-25 12:42:57.418][27][debug][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:341] dns resolution for www.google.com started
[2022-06-25 12:42:57.418][27][critical][assert] [source/common/event/timer_impl.cc:31] assert failure: dispatcher_.isThreadSafe().
[2022-06-25 12:42:57.418][27][critical][backtrace] [./source/server/backtrace.h:104] Caught Aborted, suspect faulting address 0xfffe00000010
"
It looks like DNS filter resolving by design is in worker threads.  It doesn't have the logic as in dynamic forward proxy which post the resolving to main threads:  https://github.com/envoyproxy/envoy/blob/326b205af6b16dbf75f6bfbf246082f5966bc0b0/source/extensions/common/dynamic_forward_proxy/dns_cache_impl.cc#L110 .
Also, for DFP, after resolving finished, it posts the results back to all worker threads:
https://github.com/envoyproxy/envoy/blob/326b205af6b16dbf75f6bfbf246082f5966bc0b0/source/extensions/common/dynamic_forward_proxy/dns_cache_impl.cc#L386
DNS filter only needs to send the resolved data back to client, and doesn't need to post the data to other worker threads. Therefore , such worker-thread--->main-thread---->worker-thread resolving process is not needed in DNS filter.  And with that, the DNS resolver has to be created in worker threads during DNS filter creation, not in the main threads during DNS filter config parsing.  Close this PR.


Plugin couldn't save trade with a custom banner

Stack trace
[15:43:28 ERROR]: [TradeShop] com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Unterminated string at line 26 column 76 path $.l::world::19979.0::4.0::20296.0.product[0].itemStack.meta.display-name
--------------------------------------------------------------------------------------
{
"l::world::19979.0::4.0::20296.0": {
"shopLoc": {
"div": "::",
"worldName": "world",
"x": 19979.0,
"y": 4.0,
"z": 20296.0
},
"owner": {
"player": "a6da4ce6-afa3-400f-a610-d29f70eabeee",
"role": "OWNER"
},
"managers": [],
"members": [],
"shopType": "ITRADE",
"product": [
{
"itemStack": {
"==": "org.bukkit.inventory.ItemStack",
"v": 3465,
"type": "PURPLE_BANNER",
"meta": {
"==": "ItemMeta",
"meta-type": "BANNER",
"display-name": "{\"italic\":false,\"text\":\"Clan Rat Banner\"
com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Unterminated string at line 26 column 76 path $.l::world::19979.0::4.0::20296.0.product[0].itemStack.meta.display-name
at com.google.gson.internal.Streams.parse(Streams.java:61) ~[gson-2.10.jar:?]
at com.google.gson.JsonParser.parseReader(JsonParser.java:102) ~[gson-2.10.jar:?]
at com.google.gson.JsonParser.parseReader(JsonParser.java:70) ~[gson-2.10.jar:?]
at org.shanerx.tradeshop.data.storage.Json.JsonConfiguration.loadFile(JsonConfiguration.java:115) ~[TradeShop-514985 (PP).jar:?]
at org.shanerx.tradeshop.data.storage.Json.JsonShopConfiguration.loadFile(JsonShopConfiguration.java:145) ~[TradeShop-514985 (PP).jar:?]
at org.shanerx.tradeshop.data.storage.Json.JsonConfiguration.<init>(JsonConfiguration.java:63) ~[TradeShop-514985 (PP).jar:?]
at org.shanerx.tradeshop.data.storage.Json.JsonShopConfiguration.<init>(JsonShopConfiguration.java:51) ~[TradeShop-514985 (PP).jar:?]
at org.shanerx.tradeshop.data.storage.DataStorage.getShopConfiguration(DataStorage.java:292) ~[TradeShop-514985 (PP).jar:?]
at org.shanerx.tradeshop.data.storage.DataStorage.getShopConfiguration(DataStorage.java:287) ~[TradeShop-514985 (PP).jar:?]
at org.shanerx.tradeshop.data.storage.DataStorage.loadShopFromSign(DataStorage.java:180) ~[TradeShop-514985 (PP).jar:?]
at org.shanerx.tradeshop.shop.Shop.loadShop(Shop.java:168) ~[TradeShop-514985 (PP).jar:?]
at org.shanerx.tradeshop.shop.Shop.loadShop(Shop.java:178) ~[TradeShop-514985 (PP).jar:?]
at org.shanerx.tradeshop.shop.listeners.ShopProtectionListener.onBlockBreak(ShopProtectionListener.java:208) ~[TradeShop-514985 (PP).jar:?]
at com.destroystokyo.paper.event.executor.asm.generated.GeneratedEventExecutor113.execute(Unknown Source) ~[?:?]
at org.bukkit.plugin.EventExecutor$2.execute(EventExecutor.java:77) ~[paper-api-1.20.1-R0.1-SNAPSHOT.jar:?]
at co.aikar.timings.TimedEventExecutor.execute(TimedEventExecutor.java:81) ~[paper-api-1.20.1-R0.1-SNAPSHOT.jar:git-Paper-196]
at org.bukkit.plugin.RegisteredListener.callEvent(RegisteredListener.java:70) ~[paper-api-1.20.1-R0.1-SNAPSHOT.jar:?]
at io.papermc.paper.plugin.manager.PaperEventManager.callEvent(PaperEventManager.java:54) ~[paper-1.20.1.jar:git-Paper-196]
at io.papermc.paper.plugin.manager.PaperPluginManagerImpl.callEvent(PaperPluginManagerImpl.java:126) ~[paper-1.20.1.jar:git-Paper-196]
at org.bukkit.plugin.SimplePluginManager.callEvent(SimplePluginManager.java:615) ~[paper-api-1.20.1-R0.1-SNAPSHOT.jar:?]
at net.minecraft.server.level.ServerPlayerGameMode.destroyBlock(ServerPlayerGameMode.java:378) ~[?:?]
at net.minecraft.server.level.ServerPlayerGameMode.destroyAndAck(ServerPlayerGameMode.java:337) ~[?:?]
at net.minecraft.server.level.ServerPlayerGameMode.handleBlockBreakAction(ServerPlayerGameMode.java:215) ~[?:?]
at net.minecraft.server.network.ServerGamePacketListenerImpl.handlePlayerAction(ServerGamePacketListenerImpl.java:1927) ~[?:?]
at net.minecraft.network.protocol.game.ServerboundPlayerActionPacket.handle(ServerboundPlayerActionPacket.java:42) ~[?:?]
at net.minecraft.network.protocol.game.ServerboundPlayerActionPacket.a(ServerboundPlayerActionPacket.java:15) ~[?:?]
at net.minecraft.network.protocol.PacketUtils.lambda$ensureRunningOnSameThread$0(PacketUtils.java:51) ~[?:?]
at net.minecraft.server.TickTask.run(TickTask.java:18) ~[paper-1.20.1.jar:git-Paper-196]
at net.minecraft.util.thread.BlockableEventLoop.doRunTask(BlockableEventLoop.java:153) ~[?:?]
at net.minecraft.util.thread.ReentrantBlockableEventLoop.doRunTask(ReentrantBlockableEventLoop.java:24) ~[?:?]
at net.minecraft.server.MinecraftServer.doRunTask(MinecraftServer.java:1338) ~[paper-1.20.1.jar:git-Paper-196]
at net.minecraft.server.MinecraftServer.d(MinecraftServer.java:197) ~[paper-1.20.1.jar:git-Paper-196]
at net.minecraft.util.thread.BlockableEventLoop.pollTask(BlockableEventLoop.java:126) ~[?:?]
at net.minecraft.server.MinecraftServer.pollTaskInternal(MinecraftServer.java:1315) ~[paper-1.20.1.jar:git-Paper-196]
at net.minecraft.server.MinecraftServer.pollTask(MinecraftServer.java:1308) ~[paper-1.20.1.jar:git-Paper-196]
at net.minecraft.util.thread.BlockableEventLoop.runAllTasks(BlockableEventLoop.java:114) ~[?:?]
at net.minecraft.server.MinecraftServer.tickServer(MinecraftServer.java:1424) ~[paper-1.20.1.jar:git-Paper-196]
at net.minecraft.server.MinecraftServer.runServer(MinecraftServer.java:1170) ~[paper-1.20.1.jar:git-Paper-196]
at net.minecraft.server.MinecraftServer.lambda$spin$0(MinecraftServer.java:317) ~[paper-1.20.1.jar:git-Paper-196]
at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: com.google.gson.stream.MalformedJsonException: Unterminated string at line 26 column 76 path $.l::world::19979.0::4.0::20296.0.product[0].itemStack.meta.display-name
at com.google.gson.stream.JsonReader.syntaxError(JsonReader.java:1657) ~[gson-2.10.jar:?]
at com.google.gson.stream.JsonReader.nextQuotedValue(JsonReader.java:1042) ~[gson-2.10.jar:?]
at com.google.gson.stream.JsonReader.nextString(JsonReader.java:824) ~[gson-2.10.jar:?]
at com.google.gson.internal.bind.TypeAdapters$28.readTerminal(TypeAdapters.java:747) ~[gson-2.10.jar:?]
at com.google.gson.internal.bind.TypeAdapters$28.read(TypeAdapters.java:791) ~[gson-2.10.jar:?]
at com.google.gson.internal.bind.TypeAdapters$28.read(TypeAdapters.java:725) ~[gson-2.10.jar:?]
at com.google.gson.internal.Streams.parse(Streams.java:49) ~[gson-2.10.jar:?]
... 39 more
[15:43:28 ERROR]: Could not pass event BlockBreakEvent to TradeShop v2.6.1-STABLE
java.lang.NullPointerException: Cannot invoke "com.google.gson.JsonObject.entrySet()" because "this.jsonObj" is null
at org.shanerx.tradeshop.data.storage.Json.JsonShopConfiguration.loadFile(JsonShopConfiguration.java:147) ~[TradeShop-514985 (PP).jar:?]
at org.shanerx.tradeshop.data.storage.Json.JsonConfiguration.<init>(JsonConfiguration.java:63) ~[TradeShop-514985 (PP).jar:?]
at org.shanerx.tradeshop.data.storage.Json.JsonShopConfiguration.<init>(JsonShopConfiguration.java:51) ~[TradeShop-514985 (PP).jar:?]
at org.shanerx.tradeshop.data.storage.DataStorage.getShopConfiguration(DataStorage.java:292) ~[TradeShop-514985 (PP).jar:?]
at org.shanerx.tradeshop.data.storage.DataStorage.getShopConfiguration(DataStorage.java:287) ~[TradeShop-514985 (PP).jar:?]
at org.shanerx.tradeshop.data.storage.DataStorage.loadShopFromSign(DataStorage.java:180) ~[TradeShop-514985 (PP).jar:?]
at org.shanerx.tradeshop.shop.Shop.loadShop(Shop.java:168) ~[TradeShop-514985 (PP).jar:?]
at org.shanerx.tradeshop.shop.Shop.loadShop(Shop.java:178) ~[TradeShop-514985 (PP).jar:?]
at org.shanerx.tradeshop.shop.listeners.ShopProtectionListener.onBlockBreak(ShopProtectionListener.java:208) ~[TradeShop-514985 (PP).jar:?]
at com.destroystokyo.paper.event.executor.asm.generated.GeneratedEventExecutor113.execute(Unknown Source) ~[?:?]
at org.bukkit.plugin.EventExecutor$2.execute(EventExecutor.java:77) ~[paper-api-1.20.1-R0.1-SNAPSHOT.jar:?]
at co.aikar.timings.TimedEventExecutor.execute(TimedEventExecutor.java:81) ~[paper-api-1.20.1-R0.1-SNAPSHOT.jar:git-Paper-196]
at org.bukkit.plugin.RegisteredListener.callEvent(RegisteredListener.java:70) ~[paper-api-1.20.1-R0.1-SNAPSHOT.jar:?]
at io.papermc.paper.plugin.manager.PaperEventManager.callEvent(PaperEventManager.java:54) ~[paper-1.20.1.jar:git-Paper-196]
at io.papermc.paper.plugin.manager.PaperPluginManagerImpl.callEvent(PaperPluginManagerImpl.java:126) ~[paper-1.20.1.jar:git-Paper-196]
at org.bukkit.plugin.SimplePluginManager.callEvent(SimplePluginManager.java:615) ~[paper-api-1.20.1-R0.1-SNAPSHOT.jar:?]
at net.minecraft.server.level.ServerPlayerGameMode.destroyBlock(ServerPlayerGameMode.java:378) ~[?:?]
at net.minecraft.server.level.ServerPlayerGameMode.destroyAndAck(ServerPlayerGameMode.java:337) ~[?:?]
at net.minecraft.server.level.ServerPlayerGameMode.handleBlockBreakAction(ServerPlayerGameMode.java:215) ~[?:?]
at net.minecraft.server.network.ServerGamePacketListenerImpl.handlePlayerAction(ServerGamePacketListenerImpl.java:1927) ~[?:?]
at net.minecraft.network.protocol.game.ServerboundPlayerActionPacket.handle(ServerboundPlayerActionPacket.java:42) ~[?:?]
at net.minecraft.network.protocol.game.ServerboundPlayerActionPacket.a(ServerboundPlayerActionPacket.java:15) ~[?:?]
at net.minecraft.network.protocol.PacketUtils.lambda$ensureRunningOnSameThread$0(PacketUtils.java:51) ~[?:?]
at net.minecraft.server.TickTask.run(TickTask.java:18) ~[paper-1.20.1.jar:git-Paper-196]
at net.minecraft.util.thread.BlockableEventLoop.doRunTask(BlockableEventLoop.java:153) ~[?:?]
at net.minecraft.util.thread.ReentrantBlockableEventLoop.doRunTask(ReentrantBlockableEventLoop.java:24) ~[?:?]
at net.minecraft.server.MinecraftServer.doRunTask(MinecraftServer.java:1338) ~[paper-1.20.1.jar:git-Paper-196]
at net.minecraft.server.MinecraftServer.d(MinecraftServer.java:197) ~[paper-1.20.1.jar:git-Paper-196]
at net.minecraft.util.thread.BlockableEventLoop.pollTask(BlockableEventLoop.java:126) ~[?:?]
at net.minecraft.server.MinecraftServer.pollTaskInternal(MinecraftServer.java:1315) ~[paper-1.20.1.jar:git-Paper-196]
at net.minecraft.server.MinecraftServer.pollTask(MinecraftServer.java:1308) ~[paper-1.20.1.jar:git-Paper-196]
at net.minecraft.util.thread.BlockableEventLoop.runAllTasks(BlockableEventLoop.java:114) ~[?:?]
at net.minecraft.server.MinecraftServer.tickServer(MinecraftServer.java:1424) ~[paper-1.20.1.jar:git-Paper-196]
at net.minecraft.server.MinecraftServer.runServer(MinecraftServer.java:1170) ~[paper-1.20.1.jar:git-Paper-196]
at net.minecraft.server.MinecraftServer.lambda$spin$0(MinecraftServer.java:317) ~[paper-1.20.1.jar:git-Paper-196]
at java.lang.Thread.run(Unknown Source) ~[?:?]

Plugin list

Chunky
CoreProtect
Essentials
EssentialsChat
EssentialsProtect
EssentialsSpawn
GriefPrevention
ImageFrame
LibertyBans
LuckPerms
Maintenance
PluginPortal
ProtocolLib
sleep-most
TradeShop
Vault
WarpSystem

Actions to reproduce (if known)

Make any kind of trade with either TRADE or iTRADE
Add any kind of Cost
Create a custom banner with custom name (for example, through: GamerGeeks
Add as product to the trade
Trade will work as intented.
Close the server and let the trade save
After a restart, the trade will crash with the exception mentioned.

If I would guess, the plugin has a problem with the specific name, being not italic. But that is just a guess on my side.
TradeShop version and Minecraft version (CraftBukkit/Spigot/Paper/...)
This server is running Paper version git-Paper-196 (MC: 1.20.1) (Implementing API version 1.20.1-R0.1-SNAPSHOT) (Git: 773dd72)
You are running the latest version
Previous version: git-Paper-137 (MC: 1.20.1)
TradeShop 2.6.1-STABLE by Lori00 & KillerOfPie
/tradeshop Display help message
/ts help Display help message
/ts setup Display shop setup tutorial
/ts bugs Report bugs to the developers
/ts metrics Displays the plugins counted metrics.
/ts reload Reload configuration files
Agreements

[X] I have searched for and ensured there isn't already an open issue regarding this.
[X] My version of Minecraft and TradeShop are supported by the TradeShop Dev Team (when in doubt, first ask on the Discord).

Other
While I have searched the already open issues, it was claimed to be fixed here:
#153
Since it seems to be not or the error just seems to be similar, I want to give more feedback here.
I have experience in programming and java programming, so if more information is needed, feel free to ask.

Hi there, thanks for your report. We are aware of this issue, but haven't fixed it yet (we are stretched thin at the moment). You can try to fix the newlines manually in the display-name fields and see if that works. If not, let us know.
If you manage to identify the issue and fix it feel free to open a PR, that would be extremely appreciated.

Hey, I don't think I'll have the time to read myself into a new code base.
But I think I narrowed down the problem. As long as the item doesn't have a display name of any kind, the saving of the trades work. Doesn't matter if it's a banner, book or anything really.
And fixing the display-name doesn't work really consistent, since it can delete the content after that.

We are aware the issue is related to the serialization of the item display name. We are currently stretched thin though, which is why it is taking time.

I believe the trade works if a regular/colored banner is set as the product, and then the custom banners are added to the chest. Not suggesting this as a solution to the error of course.


Incorrect implementation of burstable and best effort pods

Incorrect implementation of burstable pods
Today Kata containers starts a pod with 1 vCPU.
We then ramp up the number of vCPUs based on the cpu.shares or cpu requests we see come in for the containers.
In the case of burstable pods with upper limits the upper limit is indicated by the quota and not the shares.
So to exhibit correct behavior we should scale up the CPUs based on shares and quota.
However we will still not have the right behavior for unbounded pods.
However at the very least we should do the right vCPU setup for all bounded pods.
If you see below the quota is available for bounded burstable CPU pods. Hence we should use the vCPUs based on that value.
However unbounded pods will continue to be an issue for Kata and documented as a limitation.
Here is how the shares are setup when using runc
/sys/fs/cgroup/cpu/kubepods/besteffort/cpu.shares
2
/sys/fs/cgroup/cpu/kubepods/besteffort/pod55148103-5666-11e9-9de2-525400123456/02f5cf17015d31beb2462857e1773754221712389d17df5a3a1e636bc04daaac/cpu.shares
2
/sys/fs/cgroup/cpu/kubepods/besteffort/pod55148103-5666-11e9-9de2-525400123456/1dad0913e3a373e1742ef9dbc707fbcaab65fa1d394d5dc5c7df6c6d36569db7/cpu.shares
2
/sys/fs/cgroup/cpu/kubepods/besteffort/pod55148103-5666-11e9-9de2-525400123456/6d50e7ca6b84fbd4195155e812e04e7b5976666818fe69e6d28832e63fae639f/cpu.shares
2
/sys/fs/cgroup/cpu/kubepods/besteffort/pod55148103-5666-11e9-9de2-525400123456/cpu.shares
2

/sys/fs/cgroup/cpu/kubepods/burstable/cpu.shares
1075
/sys/fs/cgroup/cpu/kubepods/burstable/pod41a37c57-5665-11e9-9de2-525400123456/31e7f4b8bfe5a23919d507c8d683ea97a83698f3e5f4abc13e4f5b401a26f3f1/cpu.shares
256
/sys/fs/cgroup/cpu/kubepods/burstable/pod41a37c57-5665-11e9-9de2-525400123456/62c5bf39f9bbd6dacb93b0a34e78f6c8db2ff0962bb90ad9ee564fb50b9c5554/cpu.shares
51
/sys/fs/cgroup/cpu/kubepods/burstable/pod41a37c57-5665-11e9-9de2-525400123456/cpu.shares
307

/sys/fs/cgroup/cpu/kubepods/besteffort/cpu.cfs_quota_us
-1
/sys/fs/cgroup/cpu/kubepods/besteffort/pod55148103-5666-11e9-9de2-525400123456/02f5cf17015d31beb2462857e1773754221712389d17df5a3a1e636bc04daaac/cpu.cfs_quota_us
-1
/sys/fs/cgroup/cpu/kubepods/besteffort/pod55148103-5666-11e9-9de2-525400123456/1dad0913e3a373e1742ef9dbc707fbcaab65fa1d394d5dc5c7df6c6d36569db7/cpu.cfs_quota_us
-1
/sys/fs/cgroup/cpu/kubepods/besteffort/pod55148103-5666-11e9-9de2-525400123456/6d50e7ca6b84fbd4195155e812e04e7b5976666818fe69e6d28832e63fae639f/cpu.cfs_quota_us
-1
/sys/fs/cgroup/cpu/kubepods/besteffort/pod55148103-5666-11e9-9de2-525400123456/cpu.cfs_quota_us
-1

/sys/fs/cgroup/cpu/kubepods/burstable/cpu.cfs_quota_us
-1
/sys/fs/cgroup/cpu/kubepods/burstable/pod41a37c57-5665-11e9-9de2-525400123456/31e7f4b8bfe5a23919d507c8d683ea97a83698f3e5f4abc13e4f5b401a26f3f1/cpu.cfs_quota_us
50000
/sys/fs/cgroup/cpu/kubepods/burstable/pod41a37c57-5665-11e9-9de2-525400123456/62c5bf39f9bbd6dacb93b0a34e78f6c8db2ff0962bb90ad9ee564fb50b9c5554/cpu.cfs_quota_us
10000
/sys/fs/cgroup/cpu/kubepods/burstable/pod41a37c57-5665-11e9-9de2-525400123456/cpu.cfs_quota_us
60000
/sys/fs/cgroup/cpu/kubepods/burstable/pod41a37c57-5665-11e9-9de2-525400123456/f876aa7433c71aec840afc385a39e1f3c1541c4521e148c3771e8679f040b788/cpu.cfs_quota_us
-1
/sys/fs/cgroup/cpu/kubepods/burstable/pod439651677ca7971bec7b2a9a0df5a512/1c3e8063c7bba7cf0e4b6e777704e56e5bdcbb15a794d36423a4734e6d5cb751/cpu.cfs_quota_us
-1
/sys/fs/cgroup/cpu/kubepods/burstable/pod439651677ca7971bec7b2a9a0df5a512/a0ff4137e61acf00ed6dc94212dbfba5161a7cb5fe20217e2cfaa1ba91474ae8/cpu.cfs_quota_us
-1
/sys/fs/cgroup/cpu/kubepods/burstable/pod439651677ca7971bec7b2a9a0df5a512/cpu.cfs_quota_us
-1

/cc @egernst @bergwolf @devimc @jcvenegas

A potential fix for this can be

unbounded pods continue to get the number of vCPUs as setup by the cluster admin in the kata toml file

That way the cluster admin control the behavior of unbounded pods
Setting it to -1 will give you all CPUs i.e. approximate the runc behaviour

For bounded pods use quota if quota is higher than shares rounded up to the next higher CPU count

Within the VM we enforce both the quota and the shares at container level, so we will get correct behavior within the VM

There is a related issue upstream in kubernetes which we also need to track https://github.com/kubernetes/kubernetes/issues/70585

@mcastelino you also need the period to calculate the number of vcpus

@mcastelino you also need the period to calculate the number of vcpus

@devimc agree

So to exhibit correct behavior we should scale up the CPUs based on shares and quota.

you mean period and not shares, right?  because I don't understand how shares can affect the number of vcpus

cpu.shares
contains an integer value that specifies a relative share of CPU time available to the tasks in a cgroup. For example, tasks in two cgroups that have cpu.shares set to 100 will receive equal CPU time, but tasks in a cgroup that has cpu.shares set to 200 receive twice the CPU time of tasks in a cgroup where cpu.shares is set to 100. The value specified in the cpu.shares file must be 2 or higher.

@devimc @mcastelino  - actually, today we size just based off of limits:
https://github.com/kata-containers/runtime/blob/master/virtcontainers/sandbox.go#L1703-L1714:
func (s *Sandbox) calculateSandboxCPUs() uint32 {
	mCPU := uint32(0)

	for _, c := range s.config.Containers {
		if cpu := c.Resources.CPU; cpu != nil {
			if cpu.Period != nil && cpu.Quota != nil {
				mCPU += utils.CalculateMilliCPUs(*cpu.Quota, *cpu.Period)
			}

		}
	}
	return utils.CalculateVCpusFromMilliCpus(mCPU)

@mcastelino @jcvenegas @devimc -- I'm not sure if there are any changes required.  We already size according to limits, and don't have a real capability to scale shares, AFAIU.
@mcastelino - agreed?

Given this pod:
apiVersion: v1
kind: Pod
metadata:
  name: test-cpumanager-burstable-integer-limit-runc
spec:
  restartPolicy: Never
  containers:
  - name: busy
    image: busybox
    command: [ "top" ]
    resources:
      requests:
        cpu: 2
        memory: 100Mi

Looking to the cgroups cpu limits for that pod we can see at pod level (set by containerd/kubelet):
A pod with two containers is created (top and pause), the sum of cpu requested by the pod is 2
#Pod period and quota
/sys/fs/cgroup/cpu/kubepods/burstable/podeb10e69e-8e49-11e9-ba74-525400123456$ cat cpu.cfs_period_us
100000
/sys/fs/cgroup/cpu/kubepods/burstable/podeb10e69e-8e49-11e9-ba74-525400123456$ cat cpu.cfs_quota_us
-1
#Containers period and quota
/sys/fs/cgroup/cpu/kubepods/burstable/podeb10e69e-8e49-11e9-ba74-525400123456$ cat */cpu.cfs_period_us
100000
100000
/sys/fs/cgroup/cpu/kubepods/burstable/podeb10e69e-8e49-11e9-ba74-525400123456$ cat */cpu.cfs_quota_us
-1
-1

With that information we use today we are not able to calculate the vcpus needed for the VM
But if we look to shares of pod we see:
sys/fs/cgroup/cpu/kubepods/burstable/podeb10e69e-8e49-11e9-ba74-525400123456$ cat cpu.shares
2048

And for its containers we see:
/sys/fs/cgroup/cpu/kubepods/burstable/podeb10e69e-8e49-11e9-ba74-525400123456$ cat */cpu.shares
2048
2

Where `2048` should be the 2 cpus requested and `2` is just an small number for the pause container.

Now my question is what should be the formula to calculate this?

`( shares /1024  )`

@jcvenegas  - it depends -- its relative to what the parent cgroup provides and what the siblings request:
 cpu.shares: The weight of each group living in the same hierarchy, that translates into the amount of CPU it is expected to get. Upon cgroup creation, each group gets assigned a default of 1024. The percentage of CPU assigned to the cgroup is the value of shares divided by the sum of all shares in all cgroups in the same level.

Oh, true, so to achive that it is needed to calculate the cpus for all levels
/fs/cgroups/cpu/ - all cpus
kubepods  - shares/(sum(siblings))
...
until we reach the container level
does it worth  to do this calculation for this cases? by doing a recursive calculation

Well this calculation will work while threre is not cpu overcommit  -  If there are many   burstable pods
this calculation based in a global parent and siblings data is going to be  not reliable - right ? @mcastelino @egernst @devimc


feat: add wdyr

Description

Notice

[ ] Have you followed the guidelines in our Contributing guide?
[ ] Have you checked to ensure there aren't other open Pull Requests for the same update/change?

Pull Request Type

[ ] :bug: Bug fix (Non-breaking Change: Fixes an issue)
[ ] :hammer_and_wrench: Chore (Non-breaking Change: Doc updates, pkg upgrades, typos, etc..)
[ ] :nail_care: New Feature (Breaking/Non-breaking Change)

Issue (if applicable)

closes #
Risk

Testing

Engineering

Operations

Screenshots (if applicable)

Current dependencies on/for this PR:

develop

PR #2319

PR #3252   👈
PR #3248

This comment was auto-generated by Graphite.

Ab-solutely: https://github.com/shapeshift/web/pull/3252/commits/47c5b900e32eb03b55f713be26d3fd3759f45d29
This will also pave the way for a selectors stability README, including reselect-tools


ValueError on most offer requests api calls

Hi there :)
I'm getting a ValueError("time data '2023-05-13T03:59:00.000Z' does not match format '%Y-%m-%dT%H:%M:%SZ'") on most of my api calls with the python client.
Here is the code to reproduce:
from duffel_api import Duffel

duffel = Duffel(access_token="MY_TOKEN")

response = (duffel.offer_requests.create()
            .cabin_class("economy")
            .passengers([{"type": "adult"}])
            .max_connections(2)
            .return_offers()
            .slices([
                    {"origin": "PAR", "destination": "LON", "departure_date": "2023-06-14"},
                    {"origin": "LON", "destination": "PAR", "departure_date": "2023-06-16"},
                ]
            )
            .execute())

Expected behavior
No exception is raised
System

OS: MacOS
Library version: 0.5.0
Python Version: 3.11.2

Same issue here on Windows and all versions of Python 3.8-11.
The issue seems to be inconsistent date formatting in the API; dates sometimes have milliseconds and sometimes don't.
lines 104
lambda value: datetime.strptime(value, "%Y-%m-%dT%H:%M:%SZ")
and 245-55 in offer.py
arriving_at=datetime.strptime(json["arriving_at"], "%Y-%m-%dT%H:%M:%S"), departing_at=datetime.strptime(json["departing_at"], "%Y-%m-%dT%H:%M:%S"),
don't seem to account for this. There's a function in utils parse_datetime that handles this situation (just "parse_datetime" seems to work as a replacement for 104, and was used in line 99 preceding it, so I'm unaware of why 104 was left with strptime), but regarding 245-55, it expects a "Z", assuming UTC.
If it's to be assumed that arriving_at and departing_at are in UTC, then maybe you could just append a "Z" in parse_datetime if it doesn't yet exist (this, alongside the earlier changes removed errors for me, at least at a glance). But it won't pass test_parse_datetime which expects a value error to be raised if there's no "Z". Not sure if this matters if everything using it actually is UTC, but I can't tell if that's the case.

Hi folks, we'll take a look at this shortly. We do have some differing datetimes that we try to resolve through auto resolution in the libraries and we clearly missed something here.
Thanks for the report.

@felixmeziere oddly enough I don't seem to be able to trigger this. If you can still trigger it, would you be so kind as to intercept the data payload and send it over?
I'll keep looking at how this inconsistency is coming through for you but not for me.

@brpeterson the second example you gave:
arriving_at=datetime.strptime(json["arriving_at"], "%Y-%m-%dT%H:%M:%S"), departing_at=datetime.strptime(json["departing_at"], "%Y-%m-%dT%H:%M:%S"),

Although they might seem inconsistent, in this case that format should be correct. For arrival and departure, we don't have timezones because it's always the local time for where you're either departing or arriving. (This isn't to say we may be missing a trick somewhere but that's why they come without timezone information)

Hi @nlopes thanks for looking into this! I'm still getting this all the time in Sentry so I guess it'll be easy to reproduce

Would you be so kind as to send me the (anonymised if needed) content of
that sentry trace? Especially the search parameters and the path and params
of the query.
On Sat, 20 May 2023, 10:44 Felix Meziere, @.***> wrote:

Hi @nlopes https://github.com/nlopes thanks for looking into this! I'm
still getting this all the time in Sentry so I guess it'll be easy to
reproduce
[image: image]
https://user-images.githubusercontent.com/11617630/239680280-4179cfb5-6db9-481d-a620-aa88d0af0354.png
—
Reply to this email directly, view it on GitHub
https://github.com/duffelhq/duffel-api-python/issues/271#issuecomment-1555875210,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAE7L5WYE7QCZEPTA4TFLCDXHCG7RANCNFSM6AAAAAAX6FHBOI
.
You are receiving this because you were mentioned.Message ID:
@.***>

I just did a call that reproduces this locally:
    (
            duffel.offer_requests.create()
            .passengers([{"type": "adult"}])
            .max_connections(2)
            .return_offers()
            .slices(
                [{'origin': 'PAR', 'destination': 'BER', 'departure_date': '2023-06-06'}, {'origin': 'BER', 'destination': 'PAR', 'departure_date': '2023-06-08'}]
            )
            .execute()
        )

Is this enough, or do you need more?

I still can't reproduce (which is not great) but I've pushed #275. If you could try with the branch in that PR and let me know the results I'd be forever grateful!

Damn, ok will check right now, thanks

@nlopes just tried with your PR's branch, so far it seems to be working with it, thanks!

This auto closed but we'll try to get a new version out soon.


Add admin and host info

https://devguide.python.org/testing/new-buildbot-worker/#conventional-always-on-machines

https://github.com/search?q=repo%3Atiran%2Fansible-wasm-buildbot admin&type=code


How do i clear the R interactive terminal in VSCODE ??

Hi I know this may sound trivial to most, but i've been looking for a way to clear the R terminal in Vscode, using ctrl + L doesnt seem to work ??
I appreciate any help.
Thanks

Ctrl+L works fine for me on Windows. (with radian if that's relevant)


futures: make the exchange a direct param of basic_publish

It doesn't make sense to have it in BasicPublishOptions as it's one of the most important parameters.
This makes the API comprehension more straightforward wrt other libs.
Fixes #49
Based on #48 to fix example/tls.rs build

This looks reasonable.
Routing key and exchange are not properties of a message ("basic properties") in the protocol, they are basic.publish (the protocol method) proeprties.


Added support for group neutral factor analysis

I am working on a sector neutral factor and I discovered that Alphalens analysis on a group neutral factor is currently limited. With group neutral factor I mean a factor intended to rank stocks across the same group, so that it makes sense to compare performance of top vs bottom stocks in the same group but it doesn't make sense to compare performance of a stock in one group with performance of another group.
The main shortcoming is the return analysis: as there is no way to demean the returns by group,  the statistics shown are of little use. Also when the plots are broken down by group, the results are not useful either as there are 2 bugs:

API documentation claims that perf.mean_return_by_quantile was performing is performing the demeaning at group level when splitting the plots by group, but even in that case it is not true
The same goes for perf.mean_information_coefficient

Also changed the API arguments names in a consistent way:

the tears.* functions use 'long_short' and 'group_neutral' all around, as those functions are intended to be the top level ones. 'by_group' is used if the function support multiple plots output, one for each group
the remaining API (mostry performance.*) use 'demeaned' and  'group_adjust' as they used to be

In  alphalens/tests/test_tears.py I removed few parameterized tests from:

test_create_summary_tear_sheet
test_create_turnover_tear_sheet
test_create_information_tear_sheet
test_create_returns_tear_sheet

Those fuctions are called by test_create_full_tear_sheet anyway, so no need to be paranoic

@twiecki I also think that the information analysis doesn't need the group_adjust (now group_neutral) flag.
Currently and historically that flag is used to perform returns demeaning before computing the IC between factor values and returns. Subtracting a constant (the mean value) to the returns doesn't affect the value of IC (IC is Spearman's rank correlation coefficient), so we should get rid of this flag all around the information analysis API, unless there is a plan to change the way IC is computed (but we can still restore the flag in the future if needed).
Please let me know what you think and in case I'll add a commit in this PR to remove the flag.

I couldn't resist, I removed the group_adjust/group_neutral flags from  information analysis API, but I didn't
squash the commit so it's a change on its own

This looks great! I think the code could use a few more comments in general and I wonder which backwards compatibility we're breaking here and if we should check for that.

Yeah, we break few APIs here, let me list all the changes

On a high level view, the new API work as before if the user was running Alhalens with default values.
The only scenario where the changes have an impact is if the user was using the by_group options to view plots by group
Main user API brekage:

tears.create_summary_tear_sheet: added optional group_neutral (no breakage)
tears.create_returns_tear_sheet:  added optional group_neutral (brekage because it's not the last argument)
tears.create_information_tear_sheet:  removed group_adjust (brekage)
tears.create_full_tear_sheet: renamed group_adjust->group_neutral (brekage)
tears.create_event_returns_tear_sheet:  added optional group_neutral (brekage because it's not the last argument)

Lower level API breakage:

utils.get_clean_factor_and_forward_returns:  renamed by_group->binning_by_group
perf.factor_information_coefficient: removed group_adjust
perf.mean_information_coefficient: removed group_adjust
perf.factor_returns: rename long_short->demeaned, renamed group_neutral->group_adjust
perf.factor_alpha_beta: renamed long_short->demeaned, added optional group_adjust
perf.mean_return_by_quantile: added group_adjust
-perf.average_cumulative_return_by_quantile: added group_adjust, added by_group, changed quantized_factor->factor_data

I think we should add deprecation warnings for our changes in the high-level API. For example, for renaming a kwarg, we should check the old name, raise the warning, but internally just remap to the new name.

There is a simpler solution if we prefer backward compatibility over nicer arguments name:

there will be no breakage in create_returns_tear_sheet and create_event_returns_tear_sheet if the new argument 'group_neutral' is added as last argument (now is the last but one because I liked to have it close to long_short as they are very related)
We keep the name group_adjust instead of group_neutral

Actually we can also leave the argument 'group_adjust' to tears.create_information_tear_sheet and document it as deprecated. It doesn't affect the computation so we keep it only to avoid API change.
This and all the above, we can avoid API breakage

I wouldn't shy away from breaking API if it makes things easier to understand (which I think this does). And the deprecation warnings should be pretty straight-forward to implement so that would be my preference.

Good, I'll go for  deprecation warnings then

This looks great, as this adds API change, we should get Quantopian engineering to take a look.  The last commit add backward compatibility for the main functions, the ones in tears.*:

functions call that use keyword arguments are fully backward compatible
function calls that use positional arguments can be safe most of the time, but for those scenarios (and only on those) where the call is ambiguous and a mapping to the new API was not possible, I made sure to raise a warning.

Thanks @luca-s . I think quantopian's internal apps don't depend on these interfaces, just users' notebooks.
@abhijeetkalyan This means that updating the version of alphalens could mean users need to update their notebooks. I assume that's fine, right?

@richafrank @luca-s it sounds like with https://github.com/quantopian/alphalens/pull/206/commits/362612cf32abde6367b2b789f526ce7a7bb0fceb, this PR is fully backwards compatible. Is that correct, or is there still any API breakage from https://github.com/quantopian/alphalens/pull/206#issuecomment-341097404 that would occur?

@abhijeetkalyan two scenarios are still not backward compatible and a warning is raised in that case:

tears.create_returns_tear_sheet when run with exactly 3 positional arguments
tears.create_event_returns_tear_sheet when run with exactly 5 positional arguments

I'd like to go on and merge this PR. The scenarios where the API are not backward compatible are very limited in scope and and they are confined to the cases where the users wanted to break down the plots by group(e.g. sector) AND they didn't provide keyword arguments. Even if those scenarios are broken by this API change Alphalens will raise a warning and the output won't provide aplot for each group, making evident that something went wrong and so encouraging the users to look at the warning.

@luca-s Yep, seems like we're all set. Thanks!


Custom inputs

@jhrehanoaa
Note: #185 will interact with this PR. Let's coordinate at your convenience.

Closing. Replaced by material in #185, as noted.


Directly implements /health endpoint, making actuator optional

Fixes #2788

Verified the correct roll-up status when one component is down:
$ curl -sv localhost:9411/actuator/health
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 9411 (#0)
> GET /actuator/health HTTP/1.1
> Host: localhost:9411
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< content-type: application/vnd.spring-boot.actuator.v2+json; charset=utf-8
< content-length: 336
<
* Connection #0 to host localhost left intact
{"status":"DOWN","zipkin":{"status":"DOWN","details":{"ElasticsearchStorage{initialEndpoints=http://localhost:9200, index=zipkin}":{"status":"DOWN","details.error":"io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/<IP_ADDRESS>:9200"},"ScribeCollector{port=9410, category=zipkin}":{"status":"UP"}}}}


Wrong return type for openai.audio.speech.create API

Confirm this is a Node library issue and not an underlying OpenAI API issue

[X] This is an issue with the Node library

Describe the bug
I'm following the tutorial for TTS on the OpenAI website: https://platform.openai.com/docs/guides/text-to-speech?lang=node
When I try to use that code in a typescript file, typescript always infers a return value of "never".  Ignoring TS for that line makes it work.

To Reproduce
Save the code from https://platform.openai.com/docs/guides/text-to-speech?lang=node in a .ts file.
Code snippets
No response
OS
macOS
Node version
Node v19
Library version
v4.17.17

I can't reproduce this; can you share a stackblitz, replit, or similar that reproduces the problem?

@rattrayalex I can't reproduce this on Replit either. It's weird because I'm using other APIs in my project, and the types all work except this one. Either way, you can close this issue as it's not reproducible, I appreciate the prompt response.

Okay, sorry for the difficulty. You might try deleting node_modules and reinstalling or something.
If others encounter this as well, please comment with your tsconfig and other details of your installation.

I have the same issue

I am facing the same issue:

@tan-dd were you able to reproduce this one on a replit?

I am getting the same issue, here is my tsconfig file
{   "compilerOptions": {     "lib": ["ES2020"],     "module": "commonjs",     "strict": true,     "esModuleInterop": true,     "skipLibCheck": true,     "skipDefaultLibCheck": true,       "moduleResolution": "node",     "noUnusedParameters": false,     "resolveJsonModule": true,     "allowSyntheticDefaultImports": true,     "forceConsistentCasingInFileNames": true,     "experimentalDecorators": true,      "target": "ES2020",     "outDir": "./dist",     "rootDir": "./src"   },   "include": ["src/**/*.ts"],   "exclude": [     "node_modules/**/*",     ".serverless/**/*",     ".webpack/**/*",     "_warmup/**/*",     ".vscode/**/*"   ] }

@DisorderedJJ could you please share a codesandbox, replit, or other reproduction link?


Firebug lite breaks when using angular.js - Issue #3

Had been missing firebug lite while working on an angular project.  Finally took a closer look at why it wouldn't start and turned out to be a simple fix.  It works again now.
I messed up the first commit and fixed it in the second, sorry.

This fix also worked for me.

Top result when you search for error message.
Too bad noone seem to be mantaining the project - it still has some use cases - debugging some obscure browsers with no access to developer tools...


Add the KidsPod app

re: https://podnews.net/press-release/kidspod-app
Here's the app's site https://kidspod.app/
This looks like a likely example US string I see in our logs:
KidspodMobileClient/111 CFNetwork/1408.0.4 Darwin/22.5.0

hmm, I did some testing, they are not setting the ua, I get generic requests from:
AppleCoreMedia/<IP_ADDRESS>F75 (iPhone; U; CPU OS 16_5_1 like Mac OS X; en_us)
I'll write them.

Installed it, and it looks like it does send their custom UA only when the episode is downloaded, not played on demand - like many other iOS apps
Added to the AppleCoreMedia list: https://docs.google.com/spreadsheets/d/1YKQgt7xwLSdHmgtZew1TVD_RqzuA9W0ec0ab9BF9E0c/edit?usp=sharing
Since this is a new app, hopefully they can dedicate some resources to fixing it.

I heard back from their CEO, Jessica, and she passed this along to their dev team to work on (including the link to this issue).

ok, thanks Andrew!


add resources reference if missing

This PR fixes following problem:
Log group retention remains Never as CFN is not aware of AWS::Logs::LogGroup resource, if serverless.yml file does not have resources defined.
As a workaround serverless.yml must have empty block, which is not very convenient to have:
resources: {}

Master has nco already on service.resources: https://github.com/ArtificerEntertainment/serverless-plugin-log-retention/blob/master/add-log-retention.js#L38
services.resources can be an Array as well as an Object.  Can I suggest rather than trying to merge via resources.Resources, you just use the compiledCloudFormationTemplate object to modify the LogGroup directly?

Thanks Doug!

Thanks Doug!

@ArtificerEntertainment is there an issue or PR which addresses @dougmoscrop solution?
Cause the problem remains unsolved while there is no resources block in yaml file.


Potential memory leak

I used the IBM Rhapsody LightSwitch.rpy file and multiplied the content a couple of times to get it up to 2.7MB. Run it with caution memory consumption is very high.
Sorry i had to rename it github is not allowing upload of *.rpy files.
LightSwitch.txt

Excessive memory usage is in memoization support in Arpeggio. I should rework that code. In the mean time I've introduced new parser parameter memoization which is False by default. For some common grammar's I tested it gives even slightly better speed performance and memory usage is considerably lower.
Please, test with Arpeggio code from master branch.

I was running your Arpeggio master and the example from perf/calgraph/. It matches my own parser memeory consumtion:
% ../../../../env/bin/python -m memory_profiler light.py
Filename: light.py

Line #    Mem usage    Increment   Line Contents
================================================
     3   19.055 MiB    0.000 MiB   @profile
     4                             def main():
     5   19.984 MiB    0.930 MiB       mm = metamodel_from_file('rhapsody.tx')
     6  837.910 MiB  817.926 MiB       model = mm.model_from_file('LightSwitch.rpy')

Here are some results in Arpeggio testing. I find it interesting, especially differences between py2 and py3.
I will make the same testset for textX to see what is the overhead of textX's magic.
Arpeggio from master branch disables memoization by default so you should have smaller memory consumption, especially if you are using Python 3.

Here are reports from textX tests after fix in Arpeggio memoization (Arpeggio's master branch). Now, memory consumption overhead with memoization is much smaller. But, anyway, for this grammar there is no point in enabling memoization as there are no cache hits at all.
Python 3 seems to handle memory much better. And it is faster too.
I will run some more tests in the following days and make a new release.

I played a bit with tracemalloc for python3 and it found one potential "leak". This results are with current Arpeggio master.
Before removing _expr_cache (python2):
 % ../../../env/bin/python test_memory_nomemoization.py                                 :(
Filename: test_memory_nomemoization.py

Line #    Mem usage    Increment   Line Contents
================================================
    16     13.9 MiB      0.0 MiB   @profile
    17                             def no_memoization():
    18
    19     13.9 MiB      0.0 MiB       parser = ParserPython(rhapsody, memoization=False)
    20
    21                                 # Smaller file
    22     13.9 MiB      0.0 MiB       file_name = join(dirname(__file__), 'test_inputs', 'LightSwitch.rpy')
    23     13.9 MiB      0.0 MiB       with codecs.open(file_name, "r", encoding="utf-8") as f:
    24     16.7 MiB      2.8 MiB           content = f.read()
    25
    26    221.2 MiB    204.5 MiB       small = parser.parse(content)
    27
    28                                 # File that is double in size
    29    221.2 MiB      0.0 MiB       file_name = join(dirname(__file__), 'test_inputs', 'LightSwitchDouble.rpy')
    30    221.2 MiB      0.0 MiB       with codecs.open(file_name, "r", encoding="utf-8") as f:
    31    227.5 MiB      6.4 MiB           content = f.read()
    32
    33    628.0 MiB    400.5 MiB       large = parser.parse(content)

../../../env/bin/python test_memory_nomemoization.py  87,43s user 0,30s system 99% cpu 1:27,78 total

After (python2):
% ../../../env/bin/python test_memory_nomemoization.py                                 :(
Filename: test_memory_nomemoization.py

Line #    Mem usage    Increment   Line Contents
================================================
    16     14.2 MiB      0.0 MiB   @profile
    17                             def no_memoization():
    18
    19     14.2 MiB      0.0 MiB       parser = ParserPython(rhapsody, memoization=False)
    20
    21                                 # Smaller file
    22     14.2 MiB      0.0 MiB       file_name = join(dirname(__file__), 'test_inputs', 'LightSwitch.rpy')
    23     14.2 MiB      0.0 MiB       with codecs.open(file_name, "r", encoding="utf-8") as f:
    24     17.0 MiB      2.8 MiB           content = f.read()
    25
    26    168.8 MiB    151.8 MiB       small = parser.parse(content)
    27
    28                                 # File that is double in size
    29    168.8 MiB      0.0 MiB       file_name = join(dirname(__file__), 'test_inputs', 'LightSwitchDouble.rpy')
    30    168.8 MiB      0.0 MiB       with codecs.open(file_name, "r", encoding="utf-8") as f:
    31    175.1 MiB      6.3 MiB           content = f.read()
    32
    33    470.8 MiB    295.7 MiB       large = parser.parse(content)

../../../env/bin/python test_memory_nomemoization.py  84,44s user 0,20s system 99% cpu 1:24,67 total

Before (python3):
% ../../../env3/bin/python test_memory_nomemoization.py                                :(
Filename: test_memory_nomemoization.py

Line #    Mem usage    Increment   Line Contents
================================================
    16     25.4 MiB      0.0 MiB   @profile
    17                             def no_memoization():
    18
    19     25.4 MiB      0.0 MiB       parser = ParserPython(rhapsody, memoization=False)
    20
    21                                 # Smaller file
    22     25.4 MiB      0.0 MiB       file_name = join(dirname(__file__), 'test_inputs', 'LightSwitch.rpy')
    23     25.4 MiB      0.0 MiB       with codecs.open(file_name, "r", encoding="utf-8") as f:
    24     26.1 MiB      0.7 MiB           content = f.read()
    25
    26     99.9 MiB     73.8 MiB       small = parser.parse(content)
    27
    28                                 # File that is double in size
    29     99.9 MiB      0.0 MiB       file_name = join(dirname(__file__), 'test_inputs', 'LightSwitchDouble.rpy')
    30     99.9 MiB      0.0 MiB       with codecs.open(file_name, "r", encoding="utf-8") as f:
    31    102.4 MiB      2.6 MiB           content = f.read()
    32
    33    242.5 MiB    140.1 MiB       large = parser.parse(content)

../../../env3/bin/python test_memory_nomemoization.py  103,16s user 0,14s system 99% cpu 1:43,37 total

After (python3):
% ../../../env3/bin/python test_memory_nomemoization.py
Filename: test_memory_nomemoization.py

Line #    Mem usage    Increment   Line Contents
================================================
    16     24.9 MiB      0.0 MiB   @profile
    17                             def no_memoization():
    18
    19     24.9 MiB      0.0 MiB       parser = ParserPython(rhapsody, memoization=False)
    20
    21                                 # Smaller file
    22     24.9 MiB      0.0 MiB       file_name = join(dirname(__file__), 'test_inputs', 'LightSwitch.rpy')
    23     24.9 MiB      0.0 MiB       with codecs.open(file_name, "r", encoding="utf-8") as f:
    24     25.6 MiB      0.7 MiB           content = f.read()
    25
    26     85.2 MiB     59.6 MiB       small = parser.parse(content)
    27
    28                                 # File that is double in size
    29     85.2 MiB      0.0 MiB       file_name = join(dirname(__file__), 'test_inputs', 'LightSwitchDouble.rpy')
    30     85.2 MiB      0.0 MiB       with codecs.open(file_name, "r", encoding="utf-8") as f:
    31     87.8 MiB      2.6 MiB           content = f.read()
    32
    33    199.2 MiB    111.4 MiB       large = parser.parse(content)

../../../env3/bin/python test_memory_nomemoization.py  100,34s user 0,12s system 99% cpu 1:40,53 total

Another approach i will try is from pympler.tracker import SummaryTracker.

Nice work! It does looks like memory leak. _expr_cache is used to speed up parse tree node navigation using rule names. I suspect that it keep references to parse tree nodes that are thrown away during backtracking.

After some quick testing it looks that there is no memory leaking involved here. _expr_cache dicts are created per each NonTerminal and although they remain empty most of the time that is memory expensive considering the number of NonTerminal instances. Try moving _expr_cache instantiation to __getattr__. That will reduce memory usage while still providing parse tree navigation feature. If everything is right send me a PR.
BTW, I'm also thinking about using __slots__ to remove one dict more and try some speed optimization techniques. I'll try some of those when catch up with some spare time.

Check out master branch of Arpeggio. Some nice figures in perf. reports.

That some nice improvements. Thanks for thank. I give it monday a shot with my big model.

Works perfect even with a more complex and bigger sbs file. Thanks :)

Great! You can close this issue if everything works ok.

This ticket can be closed. But i the grammer seems to have an error. ID is parsed as parent for the following objects [1]. This has to do with the lines [2] as it says everything following a ; is a array.
[1]
ANOTHER_TEST = r"""MI MA MO C++ 12345
{ MA
    - VAL = { MOO
        - ID = 12;
        { BOO
            - _name = "1";
        }
        { FOO
            - _count = 8;
        }
        { BOO
            - _name = "kras";
        }
        { FOO
            - _count = -1; 1; 3;
        }
        - MORE = 23;
    }
}
"""

[2]
Object:
    '{' name=ID
        properties+=Property
    '}'
;
Property:
    '-' name=ID '=' (values=Value (';'? !('-'|'}') values=Value)*)? ';'?
;
Value:
     DATE | STRING | INT | FLOAT | GUID | Object | ID
;

It is a grammar issue, not a bug in textX. Better to discuss that at textX group. You can use github gist for code snippets and just link in the group post.


Set GOGC=100 to prometheus-operator to reduce CPU usage

towards https://github.com/giantswarm/giantswarm/issues/29530

Can you explain what's thé tradeoffs hère ? Because sure we're reducing memory in favor of CPU and it's not always better. You're gonfiguring it to 100 instead of what ?

Have you considered other values than 100 in between the chart default value and this one? Maybe 75 or something would provide better résults?


Ru-gpts for chit-chat bot

I have seen this service:
https://sbercloud.ru/ru/datahub/rugpt3family/demo-boltalka
How it can be implemented through Ru-gpts?
Are there code snippets or implementation examples?
Thank you in advance

Hey @LEv145 ,
The demo-boltalka model is a fine-tuned version of ruGPT-3 large model that can be accessed via API.
If you have your own chit-chat dataset, you can fine-tune the model as shown here


Update Yellow Highlight Dismissal Behavior

Details

This update modifies the behavior of the yellow highlight for linked report actions to dismiss under various conditions.
Fixed Issues

$ https://github.com/Expensify/App/issues/42165
PROPOSAL: https://github.com/Expensify/App/issues/42165#issuecomment-2111017989
Tests

Click into a thread
Click back into the main conversation by clicking the the room title in the top left of the thread window
Observe that the thread is highlighted in yellow after you exit it
The yellow highlight should fade after:

Clicking out of the chat into another chat
Clicking into another thread
Refreshing the page
Sending a new comment in the same chat (but not receiving a comment from another user)
Taking any action from the + menu while in the chat

[x] Verify that no errors appear in the JS console

Offline tests

Same as Tests
QA Steps

Same as Tests
PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:

[x] I verified that all the inputs inside a form are aligned with each other.
[x] I added Design label and/or tagged @Expensify/design so the design team can review the changes.

[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.

Screenshots/Videos

Android: Native

https://github.com/user-attachments/assets/d12a1eb4-79f2-40eb-991b-4a2e0cc16d8c

Android: mWeb Chrome

https://github.com/user-attachments/assets/d501f2b4-ab2b-469c-aa82-6a434d42b5b2

iOS: Native

https://github.com/user-attachments/assets/0f27cd91-4982-44d7-84de-306a9623691d

iOS: mWeb Safari

https://github.com/user-attachments/assets/eb7472f7-4ae0-4b6d-a443-6a495cb8f27f

MacOS: Chrome / Safari

https://github.com/user-attachments/assets/32e37331-4050-443a-97ca-c025cb324cea

MacOS: Desktop

https://github.com/user-attachments/assets/9cd326a8-2e24-433e-bf16-355560bf6b82

@sobitneupane I think found a bug, not related to this PR, however making it not possible to test correctly:
Only in native, when we return to the main chat, the yellow highlight appears as expected, but the new message button appears no matter the size of the conversation. Even if the user takes action, the scroll doesn't go down, and the yellow highlight doesn't disappear. I did a quick search and didn't find a related issue, I'll look further.

Hello @sobitneupane, I need some help.
So far our "Yellow Highlight Dismissal Behavior" works as expected for the following scenarios:

[x] Clicking out of the chat into another chat
[x] Clicking into another thread
[ ] Refreshing the page
[x] Sending a new comment in the same chat (but not receiving a comment from another user)
[x] Taking any action from the + menu while in the chat

However, I’m having trouble with dismissing the highlight after refreshing the page. I've tried using window/performance detection, useState/useEffect logic, and the beforeunload event, but nothing seems to work perfectly.
Could you give me some suggestions on how can we achieve this? Thank you very much.

Sorry for the delay @brunovjk. I running quite busy. I will try to get to it asap.

Okay, I think I found a safe way, I'm testing it and I'll let you know. Thanks for your patience.

@sobitneupane PR, finally, ready for review. Looking forward to hearing your thoughts on the hook I created.

Taking any action from the + menu while in the chat

@brunovjk Should we remove the yellow highlight as soon as we begin taking any action (such as Submit expense), or after the action is complete? I am inclined towards the former.

Cool, let me change it now. One moment

@sobitneupane now available for review. Thank you very much.

@brunovjk I can still reproduce this issue.
https://github.com/user-attachments/assets/8f2e6998-8f52-4e16-878b-c158e197d506

Thank you, Im on it

Screen.Recording.2024-08-16.at.14.00.30.mov

@sobitneupane Thank you for clarifying. I misunderstood your point about "removing as soon as we start taking any action." It's more accurate now. However, when cleaning the reportActionID (which also clears the yellow highlight) by starting an action in a chat with many messages, the highlighted item, which might be well above the visible area, causes the view to scroll to the bottom:
https://github.com/user-attachments/assets/832f7249-f188-464a-8567-9b7309671581
What do you think? Should we keep the current behavior of scrolling to the bottom, or should we only remove the highlight while keeping the reportActionItem visible? Thanks :D

What do you think? Should we keep the current behavior of scrolling to the bottom, or should we only remove the highlight while keeping the reportActionItem visible?

@brunovjk I think we should only remove the highlight.

@brunovjk Any update? Let's try to get this merged asap.

Still, there's just one detail missing now. In short: The yellow highlight is linked to the linkedReportActionId, we must 'clear the route parameters' with this function I created:
https://github.com/Expensify/App/blob/67ff90351f41be0666f79d8aae575b02c1c563da/src/libs/actions/Report.ts#L441-L450
But we only have one problem now, if the highlighted message is in a chat with many messages below it, when clearing the reportActionID we do scroll down, as indicated here.
I'm focused on this, it should have been closed a long time ago, but do you have any idea how to prevent scrolling when we 'clear' the reportActionID?? Thanks.

Making good progress here, I need to do some more testing to make sure everything is ok and no regressions, and we'll have it for review soon.

Misclicked above.

Hey @sobitneupane :D could you please take a look at the changes I made? Everything looks good to me now.

Edit: I'm investigating the Reassure Performance Tests fail, do you have any idea what might be causing it?

I am not sure. @brunovjk Is it related to this PR? Which part of the code do you think is causing the issue?

@sobitneupane could you take a look at the changes now? I tested it and everything seems to work fine, but I'm still a little unsure about the names of the variables and functions we created.

Thanks for the update @brunovjk

Refreshing the page

@brunovjk I can still reproduce the issue. The highlight doesn't fade after refreshing the page.
https://github.com/user-attachments/assets/70eeaeff-1951-44a0-b175-69600d04fe89

Update: Working on fixing/updating the hook to a more robust approach, I'll update again by the end of the week.


sklearn deprecated

Hello !
On installation, pip throw an error because of this :
Here is how to fix this error in the main use cases:
- use 'pip install scikit-learn' rather than 'pip install sklearn'
- replace 'sklearn' by 'scikit-learn' in your pip requirements files
(requirements.txt, setup.py, setup.cfg, Pipfile, etc ...)
- if the 'sklearn' package is used by one of your dependencies,
it would be great if you take some time to track which package uses
'sklearn' instead of 'scikit-learn' and report it to their issue tracker
- as a last resort, set the environment variable
SKLEARN_ALLOW_DEPRECATED_SKLEARN_PACKAGE_INSTALL=True to avoid this error
It might be good to update requirements.txt file
Cheers,

Seems to works when using :
python -m pip install lasso-python
instead of :
pip install -r requirements.txt

Yes indeed I updated the pip package to now use the right package :+1:


Can't make it work with nonNullable fields

interface M {
  id: string;
}

type MF = FormModel<M>;

function getMf(): MF {
  return new FormGroup({
    id: new FormControl<string|undefined>('a', {nonNullable: true})
  });
}

error TS2322: Type 'FormGroup<{ id: FormControl<string | undefined>; }>' is not assignable to type 'FormGroup<FormModelInnerKeyofTraverse<M, "group", DefaultInferMode, { id: string | null; }>>'.
  Types of property 'controls' are incompatible.
    Type '{ id: FormControl<string | undefined>; }' is not assignable to type 'FormModelInnerKeyofTraverse<M, "group", DefaultInferMode, { id: string | null; }>'.

TS: 4.8.4

I've tried different Infer modes.

Hi, lets a look

undefined !=null in typescript it is a different types
always put controls type in FormGroups, like FormGroup<MF['controls']> it is better for debug type errors and you no need to describe nested FormControls types it will be inferred automatically.

It is works example:
interface M {
  id?: string | null;
}

type MF = FormModel<M, 'group', InferModeRequired & InferModeNonNullable>;

function getMf(): MF {
  return new FormGroup<MF['controls']>({
    id: new FormControl('a', { nonNullable: true }),
  });
}

also stackblitz https://stackblitz.com/edit/angular-ngx-mf-i8wxvq?file=src%2Fapp%2Fapp.component.ts

id?: string | null;
It is no matter, you can write id: string

@e-oz  Now I understand what you mean, you need type with string | undefined why you can't use null instead of undefined? You can try to use InferModeFromModel like type MF = FormModel<M, 'group', InferModeFromModel>; then all will be infer from your model.

Ok, I will check, thanks for issue.

Hello!
First of all, thank you for the library.
@iamguid did you have chance to take a look and provide a fix for this issue?

@IgorMaystrenko Hello, thank you for the comment, I'll check it today.

Seems it is one line fix, sorry guys for late fix. Commit with fix will be here today.

But I need to clarify requirements. There is a test with expected behavior, and as I understand, there is what you want, yes? @e-oz  @IgorMaystrenko
    it('Undefined property and optional value https://github.com/iamguid/ngx-mf/issues/4', () => {
        interface Optional {
            optionalA?: number;
            optionalB?: number;
            optionalC: number | undefined;
            optionalD?: number | undefined;
            optionalE?: number | undefined;
        }

        type OptionalForm = FormModel<Optional, null, InferModeFromModel>;

        const fb = new FormBuilder().nonNullable;

        const wf = fb.group<OptionalForm['controls']>({
            // optionalA - when field is optinal, control should be optional
            optionalB: fb.control(123), // when field is optinal, control should be optional
            optionalC: fb.control(undefined), // when undefined on value, control value should be undefinable too
            optionalD: fb.control(undefined), // when undefined on value and field is optional, control value should be undefinable too
            // optionalE - when undefined on value and field is optional, control should be optional
        });
    })

There is a commit, you can test your cases, if you want https://github.com/iamguid/ngx-mf/commit/f172dae2efcef0c14fca4802b6bc757a623efe01

Thanks a lot for your efforts and time!
I've tested with that commit.
Good news: it does fix the issue! :)
But, it raised an error for a few of my forms.
But, it was easy to fix by replacing
return new FormGroup({

with
return new FormGroup<EnvironmentForm['controls']>({

So maybe it's breaking change, and maybe this nice trick with Form['controls'] (you taught me in this issue responses) should be documented (I didn't check if it's documented already).

@e-oz

But, it raised an error for a few of my forms.

You need to check your types when you define FormControl<string | undefined> maybe you just missed a type.
Yes, better way to define your forms is use Form['controls'], because then you don't need to define FormControl types for each field in FormGroup. You can leave it just FormControl('test') without types.

I'll publish a new version of package, you can upgrade.

You need to check your types when you define FormControl<string | undefined> maybe you just missed a type.

yes, it was a place where I was using <string|null> instead, so the highlight was correct.

@iamguid thank you for the quick response and fix. I've checked, it works in my test project.
I will continue injecting this into another project and let you know if find something. I have a bunch of different cases there, so maybe find something.


[SPARK-13033][ML][PySpark] Add import/export for ml.regression

Add export/import for all estimators and transformers(which have Scala implementation) under pyspark/ml/regression.py.
@yanboliang Please help to review.
For doctest, I though it's enough to add one since it's common usage. But I can add to all if we want it.

Can one of the admins verify this patch?

@yanboliang Sorry for last PR that I didn't check scala side.
For regression, there are only three algorithm support MLRead/MLWrite:
LinearRegression
IsotonicRegression
AFTSurvivalRegression
I add export/import api, and doc test currently.
But there is one issues here that doctest failed with below exception. It was caused we didn't set default value for "weightCol" (IsotonicRegression), "quantilesCol"(AFTSurvivalRegression) on scala code side. I add value when constructure to pass doctest, but I thought we should submit a jira for this. How about your idea?
Exception detail.
ir2 = IsotonicRegression.load(ir_path)
Exception raised:
Traceback (most recent call last):
File "C:\Python27\lib\doctest.py", line 1289, in __run
compileflags, 1) in test.globs
File "<doctest main.IsotonicRegression[11]>", line 1, in
ir2 = IsotonicRegression.load(ir_path)
File "C:\aWorkFolder\spark\spark-1.6.0-bin-hadoop2.6\spark-1.6.0-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\ml\util.py", line 194, in load
return cls.read().load(path)
File "C:\aWorkFolder\spark\spark-1.6.0-bin-hadoop2.6\spark-1.6.0-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\ml\util.py", line 148, in load
instance._transfer_params_from_java()
File "C:\aWorkFolder\spark\spark-1.6.0-bin-hadoop2.6\spark-1.6.0-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\ml\wrapper.py", line 82, in _tran
fer_params_from_java
value = java2py(sc, self.java_obj.getOrDefault(java_param))
File "C:\aWorkFolder\spark\spark-1.6.0-bin-hadoop2.6\spark-1.6.0-bin-hadoop2.6\python\lib\py4j-0.9-src.zip\py4j\java_gateway.py", line 813, in
call
answer, self.gateway_client, self.target_id, self.name)
File "C:\aWorkFolder\spark\spark-1.6.0-bin-hadoop2.6\spark-1.6.0-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\sql\utils.py", line 45, in deco
return f(*a, **kw)
File "C:\aWorkFolder\spark\spark-1.6.0-bin-hadoop2.6\spark-1.6.0-bin-hadoop2.6\python\lib\py4j-0.9-src.zip\py4j\protocol.py", line 308, in get
eturn_value
format(target_id, ".", name), value)
Py4JJavaError: An error occurred while calling o351.getOrDefault.
: java.util.NoSuchElementException: Failed to find a default value for weightCol
at org.apache.spark.ml.param.Params$$anonfun$getOrDefault$2.apply(params.scala:647)
at org.apache.spark.ml.param.Params$$anonfun$getOrDefault$2.apply(params.scala:647)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.ml.param.Params$class.getOrDefault(params.scala:646)
at org.apache.spark.ml.PipelineStage.getOrDefault(Pipeline.scala:43)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)
at py4j.Gateway.invoke(Gateway.java:259)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:209)
at java.lang.Thread.run(Thread.java:745)

@Wenpei It looks like _transfer_params_from_java did not consider the params which do not have default value and we should handle them. Would you mind to create a jira to track this issue?

Sure, I will submit a jira, I thought we need fix it in scala side that ensure all parameter has default value.

It should not make all parameters have default value because of some params are not setting default value on purpose. I think we should modify _transfer_params_from_java to make it not to get the params which do not have default values.

Sure, good catch. I have submit a jira 13153 and submit a pr later

I need do a minor change for this PR after 13153 get resoved

@yanboliang  @mengxr
Done for this PR.  Please help review and launch test.

@Wenpei Please pay attention to the status of #11197 and update this PR corresponding when it get merged.

@yanboliang OK.

@yanboliang I complete this pr, please take a look

Looks good except minor issues.

Jenkins, test this please.

@mengxr @srowen Can you add me to white list ? or help to launch a jenkins test for this?

Jenkins, test this please.

Test build #51873 has started for PR 11000 at commit 3646b36.

Test build #51873 has finished for PR 11000 at commit 3646b36.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51873/
Test PASSed.

Merged build finished. Test PASSed.

LGTM

LGTM

LGTM

LGTM

Merged into master. Thanks!


simplified txsize calculation

This PR replaces the txsize function with something simpler and less error prone.

I see that the fee tests are failing

I would like to wait to merge this until @redxaxder has had a chance to look at it.


The mod is spamming this error message

The mod spamming this error message in my console,
Encountered 429 on route PATCH/channels/{channel_id} with bucket 6e836da6cef38ba2f3dfd8568a4e9631:channel_id=1121458058082660372 Retry-After: 5000 ms Scope: user

Please help me to fix this error.

What's your channel topic update time in your config?

Oh, I changed it to 3

Should I change it to default?

That's going to update the topic every 3 ticks, not seconds. Discord has a rate limit for topic update, which is two times per 10 minutes. Exceeding that rate causes issues. I suggest you change it to default or disable it altogether.

Ok alr, thanks.


Added the ability to filter events based on boolean expression(s) in the Events Window.

Summary
Added the ability to filter events in the event view using boolean expression(s). This PR Addresses a large part of https://github.com/microsoft/perfview/issues/1629 (CC: @Maoni0). In a nutshell, this PR allows the ability to specify a filter query (currently only) in the Columns To Display textbox to apply to all events so that we can easily filter the data to retrieve pertinent events.
Design Details
In a nutshell, here's what's happening under the hood:

If the user specifies a filter query within [] in the "Columns To Display" textbox, we extract the intended query and create a FilterQueryExpressionTree out of it based on the string provided by the user.

A FilterQueryExpressionTree encapsulates the collection of FilterQueryExpressions and the matching of the trace event on the collection of these expressions.
FilterQueryExpressions are of the following format: LeftOperand Operator RightOperand where:

LeftOperand can either be a name of a property to search on in any event e.g. ThreadID or be specified specifically for a particular event name (we are simply checking if the event name passed in by the user is contained in the full EventName property of the TraceEvent) by using ::. For example: CSwitch::OldThreadID.
Operator can be any of the following: !=, =, <, <=, >=, >, Contains.
RightOperand can either be a string or a numeric that's used as a double.

Multiple FilterQuerieExpressions can be combined using either || or &&.

The algorithm used to deduce the logical expression is a modification of the Shunting Yard Algorithm that takes in an infix expression, converts it into it's postfix (aka Reverse Polish Notation (RPN)) equivalent.

The design decision to convert to the RPN equivalent is based on making the boolean expression reduction more performant when we iterate through all TraceEvents to discern which ones to display.

Once the RPN expression is obtained, and the user invokes a UI update, the entire RPN expression is applied to the TraceEvents and a single boolean is returned on the basis of which, we decide to display or discard the particular event.

While working on this PR, the solution was prototyped here. A number of unit tests have also been added with this PR.
Current Implementation Limitations

The way I have designed this, I have imposed a restriction of 26 Filter Query Expressions. This is implementation specific and I'll be happy to change this, if need be. The trade-off here was to keep the construction of final RPN expression quick and easy to reduce in the more perf demanding part of the code.
To construct an expression, you have to add a space between the Left Operand, Operator and Right Operand. e.g. Depth<1 will not work - instead you'll need to enter: Depth < 1. This is an implementation detail, as well; to construct FilterQueryExpressions, I am splitting on empty spaces to distinguish between the different parts of the expression.
Right now, only one query will be used to filter the event i.e. you can't specify multiple [..]s in the Columns To Display textbox.

Demo
Demo of Greater + Less Than or Equal to Operators:
https://user-images.githubusercontent.com/4951960/168762353-416a619c-0c2f-4b59-887c-213a7fe2f77f.mp4
Demo of More Complex Operations With A Bunch of Nesting:
https://user-images.githubusercontent.com/4951960/168765726-f6644215-77e3-4f5a-a70f-a1359a82cdc0.mp4
Some of the queries I demo'd based on the GC/Start and GC/Stop events are:

[(Count > 10 && (Depth >= -1) && (Count <= 30) && (Count <= 30 || ProcessorNumber = 2))] Count ProcessorNumber Depth
[(Count > 10  && (Count <= 30) && (Count <= 30 || ProcessorNumber = 2))] Count ProcessorNumber Depth
[(Count > 10  && (Count <= 30) && (Count <= 30 || ProcessorNumber = 2))] Count ProcessorNumber
[(Count > 10  && (Count <= 30)  && (Count <= 30 || ProcessorNumber = 2))] Count ProcessorNumber

Happy to address any feedback!

@MokoSan, this is super cool.  I am playing around with this locally and starting to look at the code.

@MokoSan, this is super cool. I am playing around with this locally and starting to look at the code.

Thanks for taking a look! Please let me know if you have any feedback. I am also working on getting this functionality working from the Find text box; might submit that as a separate PR.

Hi @brianrob, I think this PR is ready to be merged sans the seemingly unrelated build failures; have added the ability to do the query search for the Find text box and added a plethora of unit tests. I was unable to figure out how to rerun the CI checks. How can I do so? Thanks for your help!

Hi @brianrob, thank you very much for taking the time to review the PR!
Here are my responses:

I tried this out on my machine and it's super cool.  One thing I noticed is that the code is very whitespace particular.  For example, [(ProcessorNumber = 2)] works but [(ProcessorNumber =2)] throws an exception.  I also noticed that if you don't specify any fields then you get an empty Rest column.  Is that intentional?

Yes, this is intentional - I mentioned this in the first comment in the PR but let me know if I should be adding this anywhere else to better broadcast the usage.
Details: The way I have this currently implemented is by splitting on empty spaces to deduce the operands and the operator. Do let me know if this should be changed. I have a couple of ideas but will require some more work.

I am a bit leary of perf.  I see lots of allocations of dictionaries, calls to string.Split, etc. and some N-squared algorithms (e.g. looking up each field in a set of field names, which results in a O(N) lookup per-field).  I am wondering how this behaves over a reasonably large trace.  I'm thinking that there is likely room for optimization, but I'm wondering if you've profiled any of this.  I want to be pragmatic and not force perfection here - just want to make sure that we aren't egregious.  :)

Definitely will be happy to iterate on this further. I have addressed a majority of your comments with the latest update.
Microbenchmarking
I used Benchmark.NET to get details about the wall-clock time and allocations based on creating and matching simple and complex queries.
NOTE: By clicking on the ▸in any of the sections below, you should have access to all the Benchmark.NET results and code.
For Simple Queries
For simple queries, the average wall-clock time it takes to create a filter query tree and to match on it by a simple query is ~1.5μs allocating ~1.2KB per call.

Results

BenchmarkDotNet=v0.13.1, OS=Windows 10.0.22000
11th Gen Intel Core i7-1165G7 2.80GHz, 1 CPU, 8 logical and 4 physical cores
  [Host]                                    : .NET Framework 4.8 (4.8.4510.0), X86 LegacyJIT  [AttachedDebugger]
  FilterQueryTreePerf_CreateAndMatch_Simple : .NET Framework 4.8 (4.8.4510.0), X86 LegacyJIT

Job=FilterQueryTreePerf_CreateAndMatch_Simple  InvocationCount=100  IterationCount=100
UnrollFactor=1

Method
SimpleExpression
Mean
Error
StdDev
Median
Min
Q1
Q3
Max
Allocated

CreateAndMatch_SimpleExpression
ThreadID = 1,001
1,245.3 ns
105.53 ns
311.17 ns
1,136.0 ns
830.0 ns
1,019.5 ns
1,464.0 ns
1,961.0 ns
901 B

CreateAndMatch_SimpleExpression
ThreadID > 1,000
1,599.4 ns
128.19 ns
373.93 ns
1,547.5 ns
974.0 ns
1,304.0 ns
1,889.8 ns
2,407.0 ns
901 B

CreateAndMatch_SimpleExpression
ThreadID >= 1,001
1,428.0 ns
82.64 ns
239.75 ns
1,447.5 ns
1,033.5 ns
1,207.5 ns
1,641.5 ns
1,921.5 ns
983 B

CreateAndMatch_SimpleExpression
ThreadID Contains 1
962.4 ns
27.08 ns
76.82 ns
945.0 ns
831.0 ns
903.0 ns
1,013.0 ns
1,200.0 ns
983 B

Code
    [SimpleJob(targetCount: 100, invocationCount: 100, id: "FilterQueryTreePerf_CreateAndMatch_Simple")]
    [MinColumn, Q1Column, MedianColumn, Q3Column, MaxColumn]
    [MemoryDiagnoser]
    [MarkdownExporter]
    public class FilterQueryTreePerf_CreateAndMatch_Simple
    {
        [Params("ThreadID = 1,001", "ThreadID >= 1,001", "ThreadID > 1,000", "ThreadID Contains 1")]
        public string SimpleExpression { get; set; }

        [Benchmark]
        public void CreateAndMatch_SimpleExpression()
        {
            FilterQueryExpressionTree tree = new FilterQueryExpressionTree(SimpleExpression);
            var data = new Dictionary<string, string>()
            {
                { "ThreadID", "1,001" },
            };
            tree.Match(data, "GC/Start");
        }
    }

For Complex Queries
For more complex queries, the average wall-clock time varies fairly significantly depending on the query. E.g. for a fairly nested complex query such as: ((GC/Start::Depth = 10 && (GC/Start::Depth <= 11 && (GC/Start::Depth <= 12 && (Depth <= 13 && (GC/Start::Depth <= 14)))))) , we average around 12.5μs of wall-clock time and allocate 9KB of memory as opposed to a query like: ```"((Depth <= 10) && (Depth <= 20)) || ((Depth != 15) && (Depth > 5))"`` where we average around 7.22μs and allocate 6KB.

Results

BenchmarkDotNet=v0.13.1, OS=Windows 10.0.22000
11th Gen Intel Core i7-1165G7 2.80GHz, 1 CPU, 8 logical and 4 physical cores
  [Host]                            : .NET Framework 4.8 (4.8.4510.0), X86 LegacyJIT  [AttachedDebugger]
  FilterQueryTreePerf_Match_Complex : .NET Framework 4.8 (4.8.4510.0), X86 LegacyJIT

Job=FilterQueryTreePerf_Match_Complex  InvocationCount=100  IterationCount=100
UnrollFactor=1

Method
ComplexExpression
Mean
Error
StdDev
Median
Min
Q1
Q3
Max
Allocated

CreateAndMatch_ComplexExpression
((Dep(...)> 5)) [66]
7.224 μs
0.1879 μs
0.5269 μs
7.043 μs
6.484 μs
6.893 μs
7.424 μs
8.817 μs
6 KB

CreateAndMatch_ComplexExpression
((GC(...))))) [122]
12.576 μs
0.8231 μs
2.3350 μs
11.783 μs
9.009 μs
10.781 μs
13.876 μs
20.172 μs
9 KB

CreateAndMatch_ComplexExpression
((GC/(...) 10)) [92]
8.980 μs
0.4311 μs
1.2299 μs
8.584 μs
7.514 μs
8.077 μs
9.632 μs
12.423 μs
7 KB

CreateAndMatch_ComplexExpression
((Thr(...)1001) [68]
5.686 μs
0.2163 μs
0.6171 μs
5.372 μs
5.008 μs
5.232 μs
5.922 μs
7.278 μs
5 KB

CreateAndMatch_ComplexExpression
((Thr(...)1001) [45]
4.266 μs
0.3320 μs
0.9631 μs
3.907 μs
3.209 μs
3.526 μs
4.858 μs
7.092 μs
3 KB

CreateAndMatch_ComplexExpression
(Dept(...)= 20) [30]
3.746 μs
0.1663 μs
0.4744 μs
3.582 μs
3.216 μs
3.379 μs
3.997 μs
5.059 μs
3 KB

CreateAndMatch_ComplexExpression
(GC/S(...)s 10) [50]
4.714 μs
0.2925 μs
0.8202 μs
4.503 μs
3.763 μs
4.039 μs
5.058 μs
7.478 μs
4 KB

CreateAndMatch_ComplexExpression
(GC/S(...)= 20) [48]
5.564 μs
0.4654 μs
1.3577 μs
5.129 μs
3.916 μs
4.441 μs
6.520 μs
9.515 μs
4 KB

CreateAndMatch_ComplexExpression
(OldP(...)est2) [77]
6.068 μs
0.2337 μs
0.6704 μs
5.792 μs
5.370 μs
5.574 μs
6.674 μs
8.434 μs
6 KB

CreateAndMatch_ComplexExpression
Depth(...)!= 10 [41]
4.936 μs
0.3329 μs
0.9552 μs
4.509 μs
3.970 μs
4.222 μs
5.545 μs
7.585 μs
4 KB

CreateAndMatch_ComplexExpression
Depth(...)!= 10 [41]
5.365 μs
0.3979 μs
1.1289 μs
4.941 μs
3.942 μs
4.538 μs
5.749 μs
8.388 μs
4 KB

CreateAndMatch_ComplexExpression
Depth(...)<= 20 [26]
3.672 μs
0.2279 μs
0.6576 μs
3.416 μs
2.957 μs
3.165 μs
4.032 μs
5.578 μs
3 KB

Code
    [SimpleJob(targetCount: 100, invocationCount: 100, id: "FilterQueryTreePerf_Match_Complex")]
    [MinColumn, Q1Column, MedianColumn, Q3Column, MaxColumn]
    [MemoryDiagnoser]
    [MarkdownExporter]
    public class FilterQueryTreePerf_CreateAndMatch_Complex
    {
        [Params("(Depth >= 10) && (Depth <= 20)",
        "Depth <= 10 || Depth <= 20",
        "((Depth <= 10) && (Depth <= 20)) || ((Depth != 15) && (Depth > 5))",
        "Depth != 10 && Depth != 10 && Depth != 10",
        "Depth <= 10 && Depth <= 10 && Depth != 10",
        "((ThreadData != 1,0033) && ThreadData = 1001)",
        "((ThreadData != 1,0033 && ThreadData != 1,002) && ThreadData = 1001)",
        "(OldProcessName = test && OldProcessName != test1 || OldProcessName != test2)",
        "(GC/Start::Depth >= 10 && GC/Start::Depth <= 20)",
        "((GC/Start::Depth = 10 && (GC/Start::Depth <= 11 && (GC/Start::Depth <= 12 && (Depth <= 13 && (GC/Start::Depth <= 14))))))",
        "((GC/Start::Depth = 10) && (Depth <= 10)) || ((GC/Start::ThreadData != 10) || (Depth >= 10))",
        "(GC/Start::Depth = 10) && (ThreadData Contains 10)")]
        public string ComplexExpression { get; set; }

        [Benchmark]
        public void CreateAndMatch_ComplexExpression()
        {
            FilterQueryExpressionTree tree = new FilterQueryExpressionTree(ComplexExpression);
            var data = new Dictionary<string, string>
                {
                    { "Depth", "10" },
                    { "ThreadData", "10" }
                };
            tree.Match(data, "GC/Start");
        }
    }

CPU Analysis Based On Tracing PerfView
I collected a trace that examined another trace with the events window open with ~1M+ CSwitch events with the following Columns to display textbox filled:
Cols: OldProcessName ProcessorNumber [(ProcessName = devenv) && (ProcessorNumber = 5)]

The results indicated that to traverse 1M events and to filter 10k out of them for a 2 expression query, it takes ~1 second of CPU time to Match (this was much lesser than what I expected). Additionally, for the Shunting Yard Algorithm, the lifeblood of the boolean expression deduction, string[] rawTokens = postfixNotation.Split(' '); and  if (_operandRegex.IsMatch(rawToken)) were the two most expensive calls. For Matching individual queries, Splitting the query expression and looking in the payload of the Trace Events took the most time.
I wouldn't call these egregious with these latest changes but do let me know if you'd want me to optimize further.
Further details:

Name
Inc %
Inc
Exc %
Exc

microsoft.diagnostics.tracing.traceevent!FilterQueryExpressionTree.Match
28.3
1,817
0.3
18

+   microsoft.diagnostics.tracing.traceevent!ShuntingYard.Match
14.3
919
0.2
10

+   microsoft.diagnostics.tracing.traceevent!FilterQueryExpression.Match
11.2
721
0.3
17

+   mscorlib.ni!System.String.CtorCharCount(Char, Int32)
1.1
70
0.0
0

+   mscorlib.ni!System.Collections.Generic.Dictionary`2[System.__Canon,System.Boolean].Insert(System.__Canon,   Boolean, Boolean)
0.6
39
0.5
31

+   mscorlib.ni!System.Collections.Generic.Dictionary`2[System.__Canon,System.Boolean].Clear()
0.5
34
0.0
3

+   mscorlib.ni!System.Collections.Generic.Dictionary`2+Enumerator[System.Char,System.__Canon].MoveNext()
0.2
13
0.2
12

Zooming in on the Shunting Yard algorithm, the lifeblood of the matching side of things:

Name
Inc %
Inc
Exc %
Exc

+   microsoft.diagnostics.tracing.traceevent!ShuntingYard.Match
14.3
919
0.2
10

+   system.ni!Regex.IsMatch
6.6
422
0.0
0

+   mscorlib.ni!String.SplitInternal
3.8
247
0.0
2

Zooming in on the individual query Match side of things:

Name
Inc %
Inc
Exc %
Exc

+   microsoft.diagnostics.tracing.traceevent!FilterQueryExpression.Match
11.2
721
0.3
17

+ mscorlib.ni!String.Split
5.5
351
0.1
9

+   microsoft.diagnostics.tracing.traceevent!FilterQueryUtilities.ExtractPayloadByName
5.4
349
0.1
9

+ mscorlib.ni!String.EqualsHelper
0.0
2
0.0
2

The command I used to collect the trace: .\PerfView.exe /nogui /accepteula collect withperf.etl.

Hi @brianrob,
I believe I have addressed the following issues in the latest commit:

Added documentation in the Columns To Display and Find sections.
Removed the requirement of providing spaces amongst the operands and the operator per expression i.e. we can now write expressions such as [(ProcessName=devenv) && (GC/Start::Depth >=2)].
Added more robustness to all the Filter Query Expression calls by wrapping around try-catch blocks and writing to the log file in case of an exception rather than bubbling it up to the user.

Do let me know if you have any further feedback! Thanks for your help!


Pin bettermc to last cran-approved version in workflow

Description
Make the same change as https://github.com/cmu-delphi/covidcast-indicators/pull/1828 in the backfill corrections workflow.
The bettermc package, required both for backfill corrections and as a dependency of evalcast, was recently removed from CRAN. Pin to the last working version.
Changelog

.github/workflows/backfill-corr-ci.yml

Fixes
Backfill corrections CI is failing.

This works, but a 1-hr (uncached) runtime is annoying to work with. Will try to reduce that in a separate PR.


msstats conversion and  factor value validation

Is there a test for it? Would be very helpful also to judge the correctness of the output of this PR.


[QUERY] get outputstream from datalakeclient is missing

Hi,
In azure blob storage we can get the outputstream from blob and write any data to the stream with parallel transfer options.
I dont see this option in datalakeclient
I only found uploadwithresponse but enforce size of the stream or file.
Thanks,
Itay

Hi @ihudedi someone from the Storage team will follow up with your question / feature-request shortly.  Thanks

Hi, @ihudedi. Thanks for opening this feature request. We'll work on adding support for this feature in a future release

Hi @rickle-msft
Is there any updates?

@ibrahimrabab are there any updates on this? I know a while back we added openInputStream to DataLakeFileClient, were there plans on adding this API as well?

Hi @ihudedi @alzimmermsft
Thanks for reaching out, we don't have a set plan on when we'll have this feature done, but we anticipate getting to this in a future release soon.

Hi @ibrahimrabab
Which version does this fix is applied?
Thanks,
Itay

Hi @ihudedi
We are planning to release a beta version of this feature today. I will update on this thread again once the release is complete :)

@ihudedi
The output stream should be in the newest beta release available to use,
https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/storage/azure-storage-file-datalake/CHANGELOG.md#12160-beta1-2023-05-30


btcpay update to v1.13.0, signed by web-flow

fixes: #4515

I ran into an error:
# Install NBXplorer v2.5.2
# Download the NBXplorer source code v2.5.2
/home/admin/config.scripts/bonus.btcpayserver.sh: line 491: cd: NBXplorer: No such file or directory

Might be a network issue an my side. Will retry.

OK now ran thru. Merging.


Experiment: No retries for RESET MASTER on errant GTID cleanup

Observing recurring server crashes when applying gtid-errant-reset-master on production replicas, this PR removes the retries by way of experimentation.
cc @ggunson

closed in favor of #762