load data failure for memcached (Caused by: java.lang.NullPointerException)

Hello, there,
I am following the instructions to run memcached benchmarking.
I would be grateful if somebody can give me a hint to to work around this. I don't have any experience on Java. But it looks like a option or setting issue that is easy to fix. Thanks!
The failure occurs at the 4th step:

./bin/ycsb load memcached -s -P workloads/workloada > outputLoad.txt
[WARN]  Running against a source checkout. In order to get our runtime dependencies we'll have to invoke Maven. Depending on the state of your system, this may take ~30-45 seconds
[DEBUG]  Running 'mvn -pl site.ycsb:memcached-binding -am package -DskipTests dependency:build-classpath -DincludeScope=compile -Dmdep.outputFilterFile=true'
java -cp /home/MYSELF/bench/YCSB/memcached/conf:/home/MYSELF/bench/YCSB/memcached/target/memcached-binding-0.18.0-SNAPSHOT.jar:/home/MYSELF/.m2/repository/org/apache/htrace/htrace-core4/4.1.0-incubating/htrace-core4-4.1.0-incubating.jar:/home/MYSELF/.m2/repository/org/hdrhistogram/HdrHistogram/2.1.4/HdrHistogram-2.1.4.jar:/home/MYSELF/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:/home/MYSELF/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar:/home/MYSELF/.m2/repository/net/spy/spymemcached/2.11.4/spymemcached-2.11.4.jar:/home/MYSELF/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.4/jackson-core-asl-1.9.4.jar:/home/MYSELF/bench/YCSB/core/target/core-0.18.0-SNAPSHOT.jar site.ycsb.Client -db site.ycsb.db.MemcachedClient -s -P workloads/workloada -load
Command line: -db site.ycsb.db.MemcachedClient -s -P workloads/workloada -load
YCSB Client 0.18.0-SNAPSHOT

Loading workload...
log4j:WARN No appenders could be found for logger (org.apache.htrace.core.Tracer).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Starting test.
2020-06-17 19:05:07:107 0 sec: 0 operations; est completion in 0 second
site.ycsb.DBException: java.lang.NullPointerException
at site.ycsb.db.MemcachedClient.init(MemcachedClient.java:133)
at site.ycsb.DBWrapper.init(DBWrapper.java:86)
at site.ycsb.ClientThread.run(ClientThread.java:91)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at site.ycsb.db.MemcachedClient.createMemcachedClient(MemcachedClient.java:166)
at site.ycsb.db.MemcachedClient.init(MemcachedClient.java:122)
... 3 more
2020-06-17 19:05:07:132 0 sec: 0 operations; est completion in<PHONE_NUMBER>67300 days 15 hours

Have you fixed this problem?

This worked for me by appointing memcached.properties file. Run this command under YCSB directory:
bin/ycsb load memcached -s -P workloads/workloada -P memcached/conf/memcached.properties > outputLoad.txt

or
./bin/ycsb load memcached -s -P workloads/workloada \
    -p "memcached.hosts=<IP_ADDRESS>" > outputLoad.txt

as the end of README suggested.


[ISSUE #1110] Enable Consumer logging

Which Issue(s) This PR Fixes(Closes)

Fixes #1110
Brief Description

How Did You Test This Change?

🚀Thanks for your contribution🎉. CodeRabbit(AI) will review your code first🔥


Decide on approaches for ephemeral tables, events, etc.

Ephemeral tables are essentially just events that take advantage of the same network stack as tables. We should rename them to events to make their use case more clear.

Todos:

separate from tables in config
StoreEphemeralRecord -> StoreEvent
MyEvent.emit instead of MyEvent.emitEphemeral
Still use existing encoding/packing

I agree that offchain tables is a superset of ephemerals/events in terms of usefulness, the only downside is it's more complicated

We should make sure it's not possible to emit an ephemeral event for a non-ephemeral table, because it would lead to offchain state to be different from onchain state

I don't think we need to protect the user here, it's already easy to corrupt onchain data if you circumvent codegened libs and intentionally incorrectly call Store directly

it's already easy to corrupt onchain data if you circumvent codegened libs and intentionally incorrectly call Store directly

Can you elaborate on this / give an example for how to corrupt onchain data via external Store methods? For root systems it's of course possible to emit whatever event to make the onchain state diverge from the indexer state, and this is fine because only the root owner (usually the initial deployer of the World) can register root systems. The more important piece is that it should not be possible to make the onchain state diverge from the indexer state when writing data via the external Store methods, and in my mind this is currently true: whatever bytes data is provided to the methods is written to storage and emitted in events, and the indexer updates its state in the same way as the onchain state was updated. Every bytes value is "valid" in the sense that it can be interpreted as some type, and the interpretation is the same onchain and on the indexer. (Whether the value is semantically correct is a different question, but at least the value should be the same onchain and the indexer.)

If we wanted to support all table methods (including setField, pushToField, etc) for ephemeral tables, we would either have to add a set of function signatures for each (e.g. emitSetField, emitPushToField) to all relevant interfaces (StoreCore, Store, StoreSwitch, World), or add a bool ephemeral parameter to all functions. The former would bloat the interfaces a lot and add a lot of bytecode (maybe to the point where the World would no longer fit the contract size limit anymore since it's already close), and the latter feels kinda clunky and would increase the gas cost for all methods.
I feel like the cleanest solution that would still fit most use cases is to only support setRecord and deleteRecord for ephemeral tables. Emitting events for these is very simple and cheap, and use cases like arrays can still be solved by changing the consumer's data model (e.g. adding an index key instead of using a dynamic field).
These ephemeral tables would mostly solve the offchain table use case. For the separate use case of "custom events" we could add a separate solution: an optional root module that allows users to register custom events (each event type can be registered once, is bound to a namespace and can only be emitted by systems in that namespace). This would even allow for things like token interfaces to be implemented directly on the World contract, rather than of the workaround of an interface proxy (note that there could only be one of these interfaces per World since each event can only be registered once).

The more important piece is that it should not be possible to make the onchain state diverge from the indexer state when writing data via the external Store methods, and in my mind this is currently true

true, I suppose I don't see the divergence as a fundamentally different problem - if the contract deployer wants to scam the user, they can. And devs who use generated libs wouldn't stumbler onto this problem in the first place. But for devs who do complicated stuff with Store directly this may cause a problem

an optional root module that allows users to register custom events

I think a root system can already just emit events that will be emitted by World because they're delegatecalled

if the contract deployer wants to scam the user, they can. And devs who use generated libs wouldn't stumble onto this problem in the first place. But for devs who do complicated stuff with Store directly this may cause a problem

I think the main thing we should avoid is the possibility for diverging state on vanilla Worlds without root systems (= it shouldn't be possible to have diverging state by using the permissionless external Store methods)

I think a root system can already just emit events that will be emitted by World because they're delegatecalled

I should clarify - the proposed root module adds the ability for anyone (not just the root namespace owner) to register a custom event, which can then be emitted by systems in the namespace for which the event was registered (not just root systems) by calling a method on the World.

Specific proposed changes to ephemeral tables:

Change the name to "offchain table" ("ephemeral" implies the state is not stored anywhere, but with this change offchain tables would be synced by indexers by default just like regular tables)
StoreCore

change emitEphemeralRecord to emitSetRecord
add emitDeleteRecord
add a flag to the Tables table to store whether a table is onchain or offchain (could be called offchainOnly or something else)
In emitSetRecord and emitDeleteRecord revert if the table isn't an offchain table
Remove StoreSetEphemeral and instead use the same StoreSetRecord and StoreDeleteRecord events as for regular tables

tablegen

change the config flag from ephemeral to whatever flag name we decide on (eg offchainOnly)
generate a set and deleteRecord method (just like regular tables but without the methods for fields)

I feel like in terms of indexing, it might be worth having the separate events so that we don't have to look up the table registration/definition to determine if it's an offchain only table. Although the sync stack currently combines these/treats them as a single event, that was just me being lazy. I think there are decent use cases for knowing when a table is offchain only. cc @Kooshaba @yonadaaa


Stop estimating transitions for drinkd_stat

Currently, both drinkd_stat and drinkd are being transitioned. This doesn't make sense, as drinkd_stat is dependant on drinkd, so if both are being transitioned they may become distinct. Instead, stop estimating transitions for drinkd_stat and do the accounting in the HealthModule (or Transitions module, whichever fits best).

drinkd_stat has now been replaced with the problem_drinker variable.


How to detect a contract account in the contract.

I want to exclude all contract account to use my contract. How can I detect the account deployed contract in my contract?

@taokayan @sergmetelin @heifner

The first parameter of your contract's void apply() function is uint64_t receiver, which is the name of the account to which the contract has been deployed.  If you are using the EOSIO_ABI macro, you will need to remove it and create your own implementation of the apply() function.  See the macro definition at the bottom of dispatcher.hpp for details.  If you include all of the macro, you may use if(self == N(accountname)) to test if the account named 'accountname' matches the account on which the contract is deployed.

@dabdevelop curious if you got a solution or not. Can you share the idea? Thanks!
@jgiszczak I don't understand your suggestion. The original question was trying to prevent any user with a contract deployed to that user to use my contract.

@chuang39 If the condition I quote is violated (self == N(accountname)), reject the action in your contract, or do nothing.  Or conversely, all contract actions would be within that conditional, and so only work if the contract is running within the context of the account named "accountname".

@jgiszczak sorry for late response. Thanks for your reply first.
For (self == N(accountname)), self would be the name of my contract name. Assume I deploy my contract at localtest1234, self/receiver would be localtest123. What account name I should use? If a user called user12345678 (with a deployed contract) pushes a action, how your solution could catch it?

This is a contract entry point:
void apply( uint64_t receiver, uint64_t code, uint64_t action )

receiver: the account that own the current code (self)
code: if triggering code (if code == receiver, it's in main action, otherwise it's in the notification)
Unless specified in some parameters, there's no way to know which use who push the action. Also an action can be signed by multiple accounts.

Ultimately, you can not prevent someone else from using your code.  The suggestion I have made only prevents casual reuse.
if(self == N(localtest1234))
{
    function_normally();
}

That test will fail on account localtest12345678, so a deployment to that account would do nothing.  However, your code is always retrievable from the chain, as it is public information, and it can be converted from WASM to WAST format, edited to remove that check, and reconverted to WASM and uploaded.  You can not prevent this.
There are no secrets on the chain.

@jgiszczak get your point. Thanks for your explanation!


rename "browse projects" link or show some projects by default on "browse" page

The "browse projects" link just takes you to an empty search page. (This is unrelated to #3746, at the current time the search is working.)

@jribbens Can you include a screenshot of what you're seeing? Seems to be working to me.

@jribbens Thanks for clarifying! It sounds like this is an outcome of a change we made regarding #3062 -- I've renamed the issue to reflect your recent comment.


Create u-naziboni.txt

@YirisBadolo First stage of verification passed, official domain is univ-bobo.gov.bf as far as I verified. This is just an informational note. Pull request is still in the review. I appreciate your patience while waiting for the final decision. Proof of domain ownership: https://www.univ-bobo.gov.bf/

@YirisBadolo Pull request merged. Thank you.


function to collocate insitu and satellite data

take 1d xarray insitu data from Kyla and collocate each measurement to satellite
include SSS and SST

I finished one test code to collocate SMAP SSS data with one SIZR file for 2016. @kdrushka you can try to run it. At some point in the code, where i do the interpolation, i use a loop and that's not ideal at all but i couldn't find a way to make the 3D interpolation work with xarray interp() (but i'm sure it's possible).

Sweet! :raised_hands:
I think the 3d interp wasn't working because we had non-unique times for SMAP files so they were loading more than once.
I added two things, one to only load SMAP files once, and two to do a 3-d interp along a new dimension. Let me know if this is what you had in mind/make sense...

That's great! Thanks :-)

I have added the collocation to OISST L4 data in the test notebook.
I made a .py file to run the collocation with SIZRS file for a certain date/region of interest. It seems to work!

woohoo! nice work :1st_place_medal:


Bug World. Problems with generators worlds.

Why all used generator OverWorld (Nether-type and THe-End-type)? Use Sponge build 686.
I tested. I removed the sponge. It was good.
Details in the video:
https://youtu.be/-0lDVSUWBPg

I couldn't reproduce this. Remove any plugins and see if it still doesn't work.
If not, generate a new world and try again

@simon816 I left only a sponge. Problem still exists.
Screen (This Nether):

This fault sponge. 100%. I now removed the sponge. It was good. Worlds generated good.
Screen (This Nether, Without sponge.):

What forge Version? Your allow-nether setting is on true right?

@TBlueF forge 1515, allow-nether=true.
Read my previous comment. The problem occurs after you install a sponge.

Make sure you are using Forge build 1515.
Create an empty directory and set up a Sponge server in there, see if the problem still persists.

This issue is confirmed and we are looking into it.


Typo

"The .. syntax allows for either, both, or none of it's arguments to be absent."
"its", not "it's".

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: bcdddc9e-3060-4a72-95fa-93f8a4b6b741
Version Independent ID: a70329e0-1d30-2e54-e8b8-22f9733bd09f
Content: Ranges - C# 8.0 language proposals
Content Source: proposals/csharp-8.0/ranges.md
Product: dotnet-csharp
Technology: csharp-spec
GitHub Login: @dotnet-bot
Microsoft Alias: wiwagn

Thank you for opening this issue. I'll add it to the backlog. You can always submit the fix yourself by editing the original article. Click on the Content Source link at the bottom of your original comment. To learn how to edit, see the Editing files in a repository article from GitHub.
Thanks again!


Fix README.md

Comment in Dockerfile command line breaks builds.
Copying just kmod binary breaks Driver Container run as modprobe, rmmod, etc. are symlinks to it.

Dockerfile takes comment as an argument so build fails:
\[2/2\] STEP 4/7: COPY --from=builder /etc/driver-toolkit-release.json /etc/ # better to copy it from builder rather than pulling it from the internet so it work also with disconnected envs   error: build error: error building at STEP "COPY --from=builder /etc/driver-toolkit-release.json /etc/ # better to copy it from builder rather than pulling it from the internet so it work also with disconnected envs": checking on sources under "/var/lib/containers/storage/overlay/10fe403477c06dadd312c69ff55074a6840df0a7384073d3d0e0d1fb887965c1/merged": copier: stat: "/#": no such file or directory

Copying only kmod binary makes that Driver Container pod fails when running as modprobe is not found:
``
Warning  FailedPostStartHook  2m20s  kubelet            Exec lifecycle hook ([modprobe -v -a simple-kmod simple-procfs-kmod]) for Container "simple-kmod-driver-container" in Pod "simple-kmod-driver-container-vsbft_simple-kmod-demo(fa368274-beb6-471f-a196-29c7b924cf86)" failed - error: rpc error: code = Unknown desc = command error: time="2022-09-27T10:18:38Z" level=error msg="exec failed: unable to start container process: exec: \"modprobe\": executable file not found in $PATH"
``

/cc @qbarrand

/lgtm
Thank you @enriquebelarte!

/retest


Updated owners list

Hi @roopalik, I'm your friendly neighborhood Azure Pull Request Bot (You can call me AZPRBOT). Thanks for your contribution!

This seems like a small (but important) contribution, so no contribution license agreement is required at this point. Real humans will now evaluate your PR.

TTYL, AZPRBOT;

Please resolve the merge conflict. Thanks.

If you need guidance about how to resolve a local merge conflict, see http://sharepoint/sites/azurecontentguidance/wiki/Pages/Resolve a local merge conflict yourself.aspx

Closed due to inactivity.


ubuntu first opening's user passwd how to know

hoesetly i don't know much about linux
i use orbstack create a brand new ubuntu,and i don't know user passwd how to know firstly

There's no password by default, as orb and sudo are both configured not to require passwords when logging in from macOS. You can set one if you need it:
sudo passwd <username>

There's no password by default, as orb and sudo are both configured not to require passwords when logging in from macOS. You can set one if you need it:
sudo passwd <username>

cooper.@ubuntu:/home$ sudo passwd cooper.
[sudo] password for cooper.:
Sorry, try again.
[sudo] password for cooper.:
sudo: 1 incorrect password attempt
how? i don't know sorry.

Did your machine creation finish? If not, wait for it to finish first. sudoers is configured during setup.

i don't know whether it is finish, i create centos also.
[cooper.@centos cooper.]$ sudo adduser me
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.

[sudo] password for cooper.:
Sorry, try again.
[sudo] password for cooper.:

When you created the CentOS machine, did the command finish or not? (Or if you used the GUI, is it still loading?)
Can you share the output of:

id
cat /etc/sudoers
grep . /etc/sudoers.d/*
orb report
As well as logs from the OrbStack menu.

Also, you can always log in as root to debug (not recommended for general use): orb -m CentOS -u root

Also, you can always log in as root to debug (not recommended for general use): orb -m CentOS -u root

use this command is ok,thanks

Can you share the output of the other commands for debugigng?

cat /etc/sudoers

## Sudoers allows particular users to run various commands as
## the root user, without needing the root password.
##
## Examples are provided at the bottom of the file for collections
## of related commands, which can then be delegated out to particular
## users or groups.
##
## This file must be edited with the 'visudo' command.

## Host Aliases
## Groups of machines. You may prefer to use hostnames (perhaps using
## wildcards for entire domains) or IP addresses instead.
# Host_Alias     FILESERVERS = fs1, fs2
# Host_Alias     MAILSERVERS = smtp, smtp2

## User Aliases
## These aren't often necessary, as you can use regular groups
## (ie, from files, LDAP, NIS, etc) in this file - just use %groupname
## rather than USERALIAS
# User_Alias ADMINS = jsmith, mikem

## Command Aliases
## These are groups of related commands...

## Networking
# Cmnd_Alias NETWORKING = /sbin/route, /sbin/ifconfig, /bin/ping, /sbin/dhclient, /usr/bin/net, /sbin/iptables, /usr/bin/rfcomm, /usr/bin/wvdial, /sbin/iwconfig, /sbin/mii-tool

## Installation and management of software
# Cmnd_Alias SOFTWARE = /bin/rpm, /usr/bin/up2date, /usr/bin/yum

## Services
# Cmnd_Alias SERVICES = /sbin/service, /sbin/chkconfig, /usr/bin/systemctl start, /usr/bin/systemctl stop, /usr/bin/systemctl reload, /usr/bin/systemctl restart, /usr/bin/systemctl status, /usr/bin/systemctl enable, /usr/bin/systemctl disable

## Updating the locate database
# Cmnd_Alias LOCATE = /usr/bin/updatedb

## Storage
# Cmnd_Alias STORAGE = /sbin/fdisk, /sbin/sfdisk, /sbin/parted, /sbin/partprobe, /bin/mount, /bin/umount

## Delegating permissions
# Cmnd_Alias DELEGATING = /usr/sbin/visudo, /bin/chown, /bin/chmod, /bin/chgrp

## Processes
# Cmnd_Alias PROCESSES = /bin/nice, /bin/kill, /usr/bin/kill, /usr/bin/killall

## Drivers
# Cmnd_Alias DRIVERS = /sbin/modprobe

# Defaults specification

#
# Refuse to run if unable to disable echo on the tty.
#
Defaults   !visiblepw

#
# Preserving HOME has security implications since many programs
# use it when searching for configuration files. Note that HOME
# is already set when the the env_reset option is enabled, so
# this option is only effective for configurations where either
# env_reset is disabled or HOME is present in the env_keep list.
#
Defaults    always_set_home
Defaults    match_group_by_gid

# Prior to version 1.8.15, groups listed in sudoers that were not
# found in the system group database were passed to the group
# plugin, if any. Starting with 1.8.15, only groups of the form
# %:group are resolved via the group plugin by default.
# We enable always_query_group_plugin to restore old behavior.
# Disable this option for new behavior.
Defaults    always_query_group_plugin

Defaults    env_reset
Defaults    env_keep =  "COLORS DISPLAY HOSTNAME HISTSIZE KDEDIR LS_COLORS"
Defaults    env_keep += "MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE"
Defaults    env_keep += "LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES"
Defaults    env_keep += "LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE"
Defaults    env_keep += "LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY"

#
# Adding HOME to env_keep may enable a user to run unrestricted
# commands via sudo.
#
# Defaults   env_keep += "HOME"

Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin

## Next comes the main part: which users can run what software on
## which machines (the sudoers file can be shared between multiple
## systems).
## Syntax:
##
## 	user	MACHINE=COMMANDS
##
## The COMMANDS section may have other options added to it.
##
## Allow root to run any commands anywhere
root	ALL=(ALL) 	ALL

## Allows members of the 'sys' group to run networking, software,
## service management apps and more.
# %sys ALL = NETWORKING, SOFTWARE, SERVICES, STORAGE, DELEGATING, PROCESSES, LOCATE, DRIVERS

## Allows people in group wheel to run all commands
%wheel	ALL=(ALL)	ALL

## Same thing without a password
# %wheel	ALL=(ALL)	NOPASSWD: ALL

## Allows members of the users group to mount and unmount the
## cdrom as root
# %users  ALL=/sbin/mount /mnt/cdrom, /sbin/umount /mnt/cdrom

## Allows members of the users group to shutdown this system
# %users  localhost=/sbin/shutdown -h now

## Read drop-in files from /etc/sudoers.d (the # here does not mean a comment)
#includedir /etc/sudoers.d

grep . /etc/sudoers.d/*

cooper. ALL=(ALL) NOPASSWD:ALL

orb report

OrbStack info:
  Version: 0.5.1 (50100)
  Commit: 646c501f9b245f5bc61bae3036ff5e92aaa7840e (v0.5.1)

System info:
  macOS: 13.2.1 (22D68)
  CPU: arm64, 8 cores
  CPU model: Apple M2

log

[    0.464270] cacheinfo: Unable to detect cache hierarchy for CPU 0
[    0.464283] random: crng init done
[    0.464376] loop: module loaded
[    0.465043] virtio_blk virtio2: 1/0/0 default/read/poll queues
[    0.465187] virtio_blk virtio2: [vda] 297960 512-byte logical blocks (153 MB/145 MiB)
[    0.465574] virtio_blk virtio3: 1/0/0 default/read/poll queues
[    0.465726] virtio_blk virtio3: [vdb]<PHONE_NUMBER>4 512-byte logical blocks (8.80 TB/8.00 TiB)
[    0.467122]  vdb: vdb1
[    0.467209] virtio_blk virtio4: 1/0/0 default/read/poll queues
[    0.467335] virtio_blk virtio4: [vdc] 20971520 512-byte logical blocks (10.7 GB/10.0 GiB)
[    0.467701]  vdc: vdc1 vdc2
[    0.467824] zram: Added device: zram0
[    0.467890] wireguard: WireGuard 1.0.0 loaded. See www.wireguard.com for information.
[    0.467933] wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld<EMAIL_ADDRESS>All Rights Reserved.
[    0.468013] tun: Universal TUN/TAP device driver, 1.6
[    0.468309] PPP generic driver version 2.4.2
[    0.468358] PPP Deflate Compression module registered
[    0.468446] PPP MPPE Compression module registered
[    0.468468] usbcore: registered new interface driver rtl8187
[    0.468503] usbcore: registered new interface driver rtl8192cu
[    0.468534] usbcore: registered new interface driver rtl8150
[    0.468563] usbcore: registered new interface driver r8152
[    0.468587] usbcore: registered new interface driver asix
[    0.468611] usbcore: registered new interface driver ax88179_178a
[    0.468634] usbcore: registered new interface driver cdc_ether
[    0.468666] usbcore: registered new interface driver cdc_ncm
[    0.468696] usbcore: registered new interface driver r8153_ecm
[    0.468746] VFIO - User Level meta-driver version: 0.3
[    0.468855] usbcore: registered new interface driver uas
[    0.468883] usbcore: registered new interface driver usb-storage
[    0.468960] vhci_hcd vhci_hcd.0: USB/IP Virtual Host Controller
[    0.468992] vhci_hcd vhci_hcd.0: new USB bus registered, assigned bus number 1
[    0.469028] vhci_hcd: created sysfs vhci_hcd.0
[    0.469062] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.01
[    0.469102] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    0.469138] usb usb1: Product: USB/IP Virtual Host Controller
[    0.469171] usb usb1: Manufacturer: Linux 6.1.21-orbstack-00098-g7d48b03fef38 vhci_hcd
[    0.469208] usb usb1: SerialNumber: vhci_hcd.0
[    0.469279] hub 1-0:1.0: USB hub found
[    0.469300] hub 1-0:1.0: 8 ports detected
[    0.469350] vhci_hcd vhci_hcd.0: USB/IP Virtual Host Controller
[    0.469388] vhci_hcd vhci_hcd.0: new USB bus registered, assigned bus number 2
[    0.469439] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
[    0.469473] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.01
[    0.469509] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    0.469542] usb usb2: Product: USB/IP Virtual Host Controller
[    0.469582] usb usb2: Manufacturer: Linux 6.1.21-orbstack-00098-g7d48b03fef38 vhci_hcd
[    0.469620] usb usb2: SerialNumber: vhci_hcd.0
[    0.469680] hub 2-0:1.0: USB hub found
[    0.469701] hub 2-0:1.0: 8 ports detected
[    0.469804] rtc-pl031 20050000.pl031: registered as rtc0
[    0.469836] rtc-pl031 20050000.pl031: setting system clock to 2023-03-28T02:57:19 UTC<PHONE_NUMBER>)
[    0.469918] hid: raw HID events driver (C) Jiri Kosina
[    0.469951] usbcore: registered new interface driver usbhid
[    0.469974] usbhid: USB HID core driver
[    0.470011] GACT probability NOT on
[    0.470030] Mirror/redirect action on
[    0.470050] netem: version 1.3
[    0.475997] Initializing XFRM netlink socket
[    0.476068] NET: Registered PF_INET6 protocol family
[    0.476273] Segment Routing with IPv6
[    0.476298] In-situ OAM (IOAM) with IPv6
[    0.476339] NET: Registered PF_PACKET protocol family
[    0.476370] Bridge firewalling registered
[    0.476395] l2tp_core: L2TP core driver, V2.0
[    0.476426] 8021q: 802.1Q VLAN Support v1.8
[    0.476454] Key type dns_resolver registered
[    0.476506] NET: Registered PF_VSOCK protocol family
[    0.476896] Loading compiled-in X.509 certificates
[    0.477134] Btrfs loaded, crc32c=crc32c-generic, zoned=no, fsverity=no
[    0.480961] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[    0.481392] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[    0.481437] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
[    0.481480] cfg80211: failed to load regulatory.db
[    0.481639] erofs: (device vda): mounted with root inode @ nid 36.
[    0.481685] VFS: Mounted root (erofs filesystem) readonly on device 254:0.
[    0.481922] devtmpfs: mounted
[    0.482037] Freeing unused kernel memory: 3008K
[    0.564331] Run /opt/orb/preinit as init process
[BEGIN] preinit
[END] preinit

   OpenRC 0.46 is starting up Linux 6.1.21-orbstack-00098-g7d48b03fef38 (aarch64)

 * Mounting /proc ... [ ok ]
 * Mounting /run ... * /run/openrc: creating directory
 * /run/lock: creating directory
 * /run/lock: correcting owner
/lib/rc/sh/gendepends.sh: 12: [: unexpected operator
 * Caching service dependencies ... [ ok ]
 * Mounting /sys ... * Mounting devtmpfs on /dev ... [ ok ]
 [ ok ]
 * Mounting security filesystem ... [ ok ]
 * Mounting /dev/mqueue ... * Mounting fuse control filesystem ... [ ok ]
 [ ok ]
 * Mounting /dev/pts ... [ ok ]
 * Mounting /dev/shm ... [ ok ]
[BEGIN] vinit-early
2023-03-28T02:57:19.791319Z  INFO vcontrol: listening on <IP_ADDRESS>:103
/opt/orb/vinit-early: line 58: echo: write error: Invalid argument
/opt/orb/vinit-early: line 60: echo: write error: Invalid argument
[END] vinit-early
hostname         | * Setting hostname ...udev             | * Starting udev ... [ ok ]

2023-03-28T02:57:19.893054Z  INFO vcontrol: flag_data_resized
2023-03-28T02:57:19.893138Z ERROR vcontrol::error: Request failed: data not ready
2023-03-28T02:57:19.893083Z  INFO vcontrol: flag_data_resized
2023-03-28T02:57:19.893211Z ERROR vcontrol::error: Request failed: data not ready
 [ ok ]

[    0.974262] udevd[547]: starting version 3.2.11
[    0.976534] udevd[547]: starting eudev-3.2.11
2023-03-28T02:57:20.094608Z  INFO vcontrol: flag_data_resized
2023-03-28T02:57:20.094646Z ERROR vcontrol::error: Request failed: data not ready
2023-03-28T02:57:20.094619Z  INFO vcontrol: flag_data_resized
2023-03-28T02:57:20.094714Z ERROR vcontrol::error: Request failed: data not ready
udev-settle      | * Waiting for uevents to be processed ...fsck             | * Checking local filesystems  ... [ ok ]

 [ ok ]

root             | * Remounting filesystems ... [ ok ]

localmount       | * Mounting local filesystems ...[    1.158154] BTRFS: device label user-data-fs devid 1 transid 23 /dev/vdb1 scanned by mount (674)
[    1.158479] BTRFS info (device vdb1): using crc32c (crc32c-generic) checksum algorithm
[    1.158541] BTRFS info (device vdb1): turning on async discard
[    1.158586] BTRFS info (device vdb1): enabling ssd optimizations
[    1.158637] BTRFS info (device vdb1): setting nodatacow, compression disabled
[    1.158725] BTRFS info (device vdb1): using free space tree
 [ ok ]

networking       | * Starting networking ... *   lo .../etc/network/if-up.d/dad: 11: arithmetic expression: expecting primary: " counter-- "

networking       |run-parts: /etc/network/if-up.d/dad: exit status 2

networking       | [ ok ]

chronyd          | * Starting chronyd ...2023-03-28T02:57:20.296213Z  INFO vcontrol: flag_data_resized
2023-03-28T02:57:20.296237Z ERROR vcontrol::error: Request failed: data not ready
2023-03-28T02:57:20.296285Z  INFO vcontrol: flag_data_resized
2023-03-28T02:57:20.296307Z ERROR vcontrol::error: Request failed: data not ready
 [ ok ]

[BEGIN] vinit-late
[    1.340139] zram: setup backing device /dev/vdc1
/opt/docker-rootfs /
[    1.342067] zram0: detected capacity change from 0 to 10571776
/
Resize device id 1 (/dev/vdb1) from 228.27GiB to max
[END] vinit-late
Setting up swapspace version 1, size = 5 GiB<PHONE_NUMBER> bytes)
no label, UUID=c81ef284-3431-4992-b11d-26431a8ebbdb
[    1.366272] Adding 5285884k swap on /dev/zram0.  Priority:32767 extents:1 across:5285884k SSDsc
vm.swappiness = 100
vm.page-cluster = 1
[    1.368271] Adding 4194300k swap on /dev/vdc2.  Priority:1 extents:1 across:4194300k Dsc
INFO[03-28 02:57:20] started
2023-03-28T02:57:20.498006Z  INFO vcontrol: flag_data_resized
2023-03-28T02:57:20.498006Z  INFO vcontrol: flag_data_resized
rpcbind          | * Starting rpcbind ...[    1.540086] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
 [ ok ]

rpc.statd        | * Starting NFS statd ... [ ok ]

nfs              | * Mounting nfsd filesystem in /proc ... [ ok ]

nfs              | * Exporting NFS directories ... [ ok ]

nfs              | * Starting NFS mountd ... [ ok ]

nfs              | * Starting NFS daemon ...[    3.096786] NFSD: Using UMH upcall client tracking operations.
[    3.096900] NFSD: starting 1-second grace period (net f0000000)
 [ ok ]

nfs              | * Starting NFS smnotify ... [ ok ]

INFO[03-28 02:57:45] starting container                            container=ubuntu
[   26.842477] conbr0: port 1(vethILKNuf) entered blocking state
[   26.842573] conbr0: port 1(vethILKNuf) entered disabled state
[   26.842776] device vethILKNuf entered promiscuous mode
[   26.842955] conbr0: port 1(vethILKNuf) entered blocking state
[   26.843089] conbr0: port 1(vethILKNuf) entered forwarding state
[   26.846430] conbr0: port 1(vethILKNuf) entered disabled state
[   26.846607] eth0: renamed from vethoc6Vll
INFO[03-28 02:57:45] container started                             container=ubuntu
INFO[03-28 02:58:16] stopping container                            container=ubuntu
WARN[03-28 02:58:20] graceful shutdown failed                      container=ubuntu error="shutting down the container failed"
INFO[03-28 02:58:20] stopped container                             container=ubuntu
INFO[03-28 03:06:23] starting container                            container=ubuntu
INFO[03-28 03:06:24] container started                             container=ubuntu
INFO[03-28 03:50:44] creating container                            id=01GWK5DW4P2C4VZJQCCZCH1FGY name=centos
INFO[03-28 03:50:44] fetching image index
INFO[03-28 03:50:46] downloading images                            spec="{centos 9-Stream arm64 default}"
INFO[03-28 03:50:57] extracting rootfs                             container=centos
INFO[03-28 03:50:59] applying templates
INFO[03-28 03:50:59] starting container                            container=centos
INFO[03-28 03:50:59] container started                             container=centos
INFO[03-28 03:50:59] waiting for network before setup              container=centos
INFO[03-28 03:51:00] network is up                                 container=centos ips="[<IP_ADDRESS> fd00:30:31:0:53f6:6544:d884:48ab]"
INFO[03-28 03:51:00] running initial setup                         uid=501 username=cooper.
INFO[03-28 03:51:00] container created                             container=centos
ERRO[03-28 04:34:52] SSH error: container not found: CentOS
ERRO[03-28 05:53:24] SSH error: container not found: CentOS

What about id?

and by the way
orbstack open terminal always open my item2, i want to open warp, how can i do?

Thanks, reopening for investigation.
It's not possible to support Warp as a terminal for containers and machines. Warp doesn't support opening scripts: https://github.com/warpdotdev/Warp/discussions/612
I'd recommend running the CLI commands yourself.

@magickatt Did you create the machine with the default (latest) version of CentOS or a specific older version? What's in sudoers.d for you? grep . /etc/sudoers.d/*
I can't reproduce this even with a username that ends with a period, so I'm not sure what's going on. For @CooperHash everything looked fine in sudoers.d.

Fixed for the next version, thanks for reporting. In the meantime, rename /etc/sudoers.d/$USER to orbstack to fix the issue. sudo doesn't seem to like sudoers.d files that contain a dot (.) in the name.

Released in v0.5.2.

su root with empty password failed
alpine:~$ su root
Password:
su: incorrect password

alpine:~$ chmod +w /etc/apk/repositories
chmod: /etc/apk/repositories: Operation not permitted

report

alpine:~$ orb report
OrbStack info:
  Version: 0.13.0 (130000)
  Commit: ba57f21d6c60d45e0469196f4a67c79ec9911eea (v0.13.0)

System info:
  macOS: 13.4 (22F66)
  CPU: arm64, 8 cores
  CPU model: Apple M2
------------------------------

@image72 It's configured to use passwordless sudo by default, not su. Accounts with no password are different from accounts with an empty password.
You can use su to change the sudoers config or set any passwords you want.

always failed, and alpine not include package sudo
alpine:~$ apk RUN apk add --no-cache su-exec
ERROR: Unable to lock database: Permission denied
ERROR: Failed to open apk database: Permission denied
alpine:~$ apk --no-cache add sudo
ERROR: Unable to lock database: Permission denied
ERROR: Failed to open apk database: Permission denied
alpine:~$ sudo -h
-sh: sudo: not found
alpine:~$ sudo apk update
-sh: sudo: not found

@image72 It's configured to use passwordless sudo by default, not su. Accounts with no password are different from accounts with an empty password.
You can use su to change the sudoers config or set any passwords you want.

always failed, and alpine not include package sudo
alpine:~$ apk RUN apk add --no-cache su-exec
ERROR: Unable to lock database: Permission denied
ERROR: Failed to open apk database: Permission denied
alpine:~$ apk --no-cache add sudo
ERROR: Unable to lock database: Permission denied
ERROR: Failed to open apk database: Permission denied
alpine:~$ sudo -h
-sh: sudo: not found
alpine:~$ sudo apk update
-sh: sudo: not found

@image72 Which version and CPU architecture did you create the Alpine machine with? It should've been installed during creation, but maybe there was a network error or something.
Anyway, you can use orb -m alpine -u root to log in as root and fix sudo/su.

@image72 It's configured to use passwordless sudo by default, not su. Accounts with no password are different from accounts with an empty password.
You can use su to change the sudoers config or set any passwords you want.

always failed, and alpine not include package sudo
alpine:~$ apk RUN apk add --no-cache su-exec
ERROR: Unable to lock database: Permission denied
ERROR: Failed to open apk database: Permission denied
alpine:~$ apk --no-cache add sudo
ERROR: Unable to lock database: Permission denied
ERROR: Failed to open apk database: Permission denied
alpine:~$ sudo -h
-sh: sudo: not found
alpine:~$ sudo apk update
-sh: sudo: not found

@image72 Which version and CPU architecture did you create the Alpine machine with? It should've been installed during creation, but maybe there was a network error or something.
Anyway, you can use orb -m alpine -u root to log in as root and fix sudo/su.

resolved, thanks. please note that in docs
orb -m alpine -u root to log in as root and fix sudo/su.

@image72 orb -u has been added to the docs: https://docs.orbstack.dev/machines/#user-accounts
Just tested again and sudo was set up correctly in a new Alpine machine, so it'd still be good to know what version and architecture you used.

@image72 orb -u has been added to the docs: https://docs.orbstack.dev/machines/#user-accounts
Just tested again and sudo was set up correctly in a new Alpine machine, so it'd still be good to know what version and architecture you used.

issued version info

alpine:~$ orb report
OrbStack info:
  Version: 0.13.0 (130000)
  Commit: ba57f21d6c60d45e0469196f4a67c79ec9911eea (v0.13.0)

System info:
  macOS: 13.4 (22F66)
  CPU: arm64, 8 cores
  CPU model: Apple M2
------------------------------

@image72 I meant the version and architecture of your Alpine machine, not OrbStack or your Mac. orb list

$ cat /etc/issue
Welcome to Alpine Linux 3.17
Kernel \r on an \m (\l)

$ orb list
NAME    STATE    DISTRO  VERSION  ARCH
----    -----    ------  -------  ----
alpine  running  alpine  3.17     arm64
kali    stopped  kali    current  arm64


New Release plz

Hi,
the version the fix from #106 does work as excepted. however we dont want to use dev-master.
is it possible to create a new bugfix version?
thx

Done. use v2.8.4

Done. use v2.8.4


Slug in URL

Here is my Code that links to another view page:

Here is my Controller code:
------ public function getView($id){
return View::make('project.view')
->with('project', Project::find($id));
} -----------
The above code works fine in getting the project id in the url but i want to display the slug in the Url that links to the view page:
public function getView($slug) {
$project = Project::where('slug',$slug)->get();
if ($project) {
return View::make('project.view')->with('project', $project);
}
// no model with that slug found ...
}
The above code you gave me i used it as mention but it giveing me the following error in my laravel log message: Next exception 'ErrorException' with message 'Undefined property: Illuminate\Database\Eloquent\Collection::$pname (View: C:\xampp\htdocs\wpc\app\views\project\view.blade.php)'

Please what can i do to make everything work well? It has been giving me bottle neck with my application.
I can provide the view page code and the project page code if you need it.

Can you edit the question?

This has been resolved here #199
Can be closed


Any updates yet about the code release?

Hello,
Thank you very much for your interest in our paper!
The code release is currently at the last stage, and we are waiting for the final green light from the copyright owner to release the source code. I believe the source code will soon be uploaded.

thank you for the fast reply.
we are all patiently waiting!


Ingress API version downgrade breaks helm chart

Describe the bug
It looks to me like this commit downgrades the ingress to v1beta1, but does not undo the field name changes from this commit. This results in the following error when applying:
Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "service" in io.k8s.api.networking.v1beta1.IngressBackend

Steps to reproduce the issue:
I am applying the helm chart via Terraform:
resource "helm_release" "thanos-operator" {
  repository = "https://kubernetes-charts.banzaicloud.com"
  chart      = "thanos-operator"
  name       = "thanos"
  namespace  = "my-namespace"
  values     = [file("${path.module}/data/values.yaml")]
}

Setting ingress.enabled = true in values.yaml should trigger this bug.
Expected behavior
The ingress should be created.

Thanks for reporting! Are you actually using the ingress to access operator metrics? I'm not sure about the original intent when creating the chart, but I'm not sure that it's (setting up an ingress) that useful at all.

I'm mostly in the exploratory phase with Thanos at the moment, so wasn't sure exactly what to expect. I wanted to see if adding the ingress gave me anything useful to look at out of the operator.
I was able to get around this by disabling creating the ingress in the chart and just using Terraform to create the ingress instead.
I ran into some other issues with the operator after that (seems like some NPE issue in the operator after I created the example CRDs), so have put using the operator on hold for now, while I just get the various Thanos components up and running.

Thanks again! I'm going to remove the ingress, since it wasn't intentional just part of the default helm create template. I guess you don't have the logs for the NPE anymore but I will check the example CRs to see if I can reproduce.

If you are open to troubleshooting the operator, I would love to be able to use it at some point. I am going to be traveling abroad for the next 3 weeks starting tomorrow, but can open a new issue for that when I get back if you'd like.


Feature Request: Export the ChartTooltip

Description
Would it be possible to export the ChartTooltip as part of the saas-ui-charts package? This would make it easier to override the props and pass it in as the tooltip when making a custom tooltip off of the existing one. Exporting it in packages/saas-ui-charts/src/index.ts would be pretty helpful.
Link to Reproduction
No response
Steps to reproduce

Go to '...'
Click on '...'
Scroll down to '...'
See error

Saas UI Version
2.8.8
Chakra UI Version
2.8.2
Browser
No response
Operating System

[ ] macOS
[ ] Windows
[ ] Linux

Additional Information
No response

Feel free to make a PR!


Where is DispatcherQueueExtensions for WinUI3?

The Microsoft documentation implies DispatcherQueueExtensions should be available for WinUI3 apps (https://learn.microsoft.com/en-us/windows/communitytoolkit/extensions/dispatcherqueueextensions).
However, the documentation does not state (or at least I couldn't find it) which of the many Community Toolkit Nuget packages need to be installed to access those extensions.
Which nuget package do I use?

@markolbert you can find them in the CommunityToolkit.WinUI package. Basically anything that is Microsoft.Toolkit.Uwp for UWP/WinUI 2 becomes CommunityToolkit.WinUI for WindowsAppSDK/WinUI 3.
This will be a bit different with our next release, as we'll have our new unified infrastructure, but we're also working on refreshing how we do our docs as part of that too, so hopefully should be easier to find in the future.


AWS Auto Scaling Group watcher

While definitely nice, this can alternatively be achieved by tagging the ASG and propagating the tag to the instances (and then using the tag watcher).


wip: integrate binary data into payloads

Previously, the non-binary part of a message and the binary payloads in a message were represented separately: the non-binary portion was represented by a serde_json::Value, and could be converted to an arbitrary data structure.  That data structure would not include the binary data or any indication that there is any binary data at all.  The binary data would be provided in a Vec<Vec>.
There were a few problems with this:

The original code only supported cases where the payload was a flat array with some binary payloads in the root of the array, or a flat object where the root of the object was a binary payload.  Objects with more complicated structure and binary data embedded in various places in the structure were not supported.
Adding support for the above turned out to not be possible in a useful way, because the ordering of the Vec<Vec> matters, and it could never be clear where exactly in the possibly-complex structure each binary payload belonged.
One of the useful features of the socket.io protocol is that it lets users efficiently transmit binary data in addition to textual/numeric data, and have that handled transparently by the protocol, with either end of the connection believing that they just sent or received a single mixed textual/numeric/binary payload.  Separating the non-binary from the binary negates that benefit.

This introduces a new type, PayloadValue, that behaves similarly to serde_json::Value.  The main difference is that it has a Binary variant, which holds a numeric index and a Vec.  This allows us to include the binary data where the sender of that data intended it to be.
There is currently one wrinkle: serde_json does not appear to consistently handle binary data; when serializing a struct with Vec, I believe it will serialize it as an array of numbers, rather than recognize that it's binary data.  For now, I've included a Binary struct that wraps a Vec, which can be included as the type of a binary member, instead of using a Vec directly.  Hopefully I'll be able to figure out a better way to do this.
Unfinished tasks:

Testing: I have no idea if this even works yet.  All I've done is get it to compile.
Benchmarking: I've tried to ensure that I don't copy data any more than the existing library does, but it's possible I've introduced some performance regressions, so I'll have to look into that.
Documentation: the documentation still references the old way of doing things and needs to be updated.

Closes #276.

Hi @Totodore, here's what I have so far.  I hate PRs that have so many changed lines, but the changes to the library itself are actually not that large.  The first commit is all added lines: the addition of PayloadValue and its methods, serializers, and deserializers.  The second commit is much smaller (+382 -334 lines), and adapts the library to use PayloadValue.

Did some benchmarking and compared runs with critcmp:
group                                              0.11.0-encode                          integrated-data-binary-encode
-----                                              -------------                          -----------------------------
Encode packet ack on /                             1.01    176.3±2.50ns        ? ?/sec    1.00    174.7±1.77ns        ? ?/sec
Encode packet ack on /custom_nsp                   1.00    183.0±2.07ns        ? ?/sec    1.00    183.2±3.36ns        ? ?/sec
Encode packet binary ack (b64) on /                2.12   483.7±15.09ns        ? ?/sec    1.00    228.1±2.51ns        ? ?/sec
Encode packet binary ack (b64) on /custom_nsp      2.03    488.8±6.23ns        ? ?/sec    1.00    240.6±7.92ns        ? ?/sec
Encode packet binary event (b64) on /              1.78    605.1±2.86ns        ? ?/sec    1.00    339.2±6.57ns        ? ?/sec
Encode packet binary event (b64) on /custom_nsp    1.77    612.0±6.06ns        ? ?/sec    1.00    345.5±3.81ns        ? ?/sec
Encode packet connect on /                         1.01     75.9±1.23ns        ? ?/sec    1.00     75.1±0.52ns        ? ?/sec
Encode packet connect on /custom_nsp               1.01     77.8±0.74ns        ? ?/sec    1.00     77.1±0.68ns        ? ?/sec
Encode packet event on /                           1.00    185.9±2.94ns        ? ?/sec    1.00    185.3±7.23ns        ? ?/sec
Encode packet event on /custom_nsp                 1.00    196.0±2.12ns        ? ?/sec    1.00    195.8±1.89ns        ? ?/sec
Encode packet event with ack on /                  1.01    186.6±1.56ns        ? ?/sec    1.00    183.9±2.07ns        ? ?/sec
Encode packet event with ack on /custom_nsp        1.00    194.1±2.64ns        ? ?/sec    1.00    193.6±2.08ns        ? ?/sec

group                                              0.11.0-decode                          integrated-data-binary-decode
-----                                              -------------                          -----------------------------
Decode packet ack on /                             1.00    382.2±2.05ns        ? ?/sec    1.15    439.9±1.95ns        ? ?/sec
Decode packet ack on /custom_nsp                   1.00    379.0±4.84ns        ? ?/sec    1.20    455.3±3.34ns        ? ?/sec
Decode packet binary ack (b64) on /                1.56    685.7±3.14ns        ? ?/sec    1.00    438.5±3.82ns        ? ?/sec
Decode packet binary ack (b64) on /custom_nsp      1.49    717.5±9.37ns        ? ?/sec    1.00   480.7±30.18ns        ? ?/sec
Decode packet binary event (b64) on /              1.44    827.9±7.43ns        ? ?/sec    1.00    573.4±5.78ns        ? ?/sec
Decode packet binary event (b64) on /custom_nsp    1.42   872.6±27.01ns        ? ?/sec    1.00    614.2±3.47ns        ? ?/sec
Decode packet connect on /                         1.00     82.1±1.50ns        ? ?/sec    1.11     91.0±3.12ns        ? ?/sec
Decode packet connect on /custom_nsp               1.00     86.6±1.34ns        ? ?/sec    1.18    102.3±2.12ns        ? ?/sec
Decode packet event on /                           1.00    478.6±5.45ns        ? ?/sec    1.11   530.1±20.90ns        ? ?/sec
Decode packet event on /custom_nsp                 1.00    506.7±7.16ns        ? ?/sec    1.11    562.6±3.95ns        ? ?/sec
Decode packet event with ack on /                  1.00    479.5±8.71ns        ? ?/sec    1.10    527.2±4.28ns        ? ?/sec
Decode packet event with ack on /custom_nsp        1.00    511.0±7.13ns        ? ?/sec    1.10    562.2±3.83ns        ? ?/sec

The encoding numbers look really good, improved in a few cases by 75%-110%.  But decoding is a bit more mixed; a few large improvements of around 50%, but quite a few 10-20% regressions.
Since the decoding regressions are in the cases without binary payloads, my guess is that my PayloadValue implementation is less efficient than serde_json::Value.  Will try to spend some time this weekend figuring out why.

Did some profiling, looked at flamegraphs, made a few changes, but the performance regressions are still there.
This is very strange, though.  Decoding of Connect packets has regressed about 7%, but that makes no sense: Connect packets don't use ser/deser at all, and the code path for those packets hasn't changed at all.  So I'm wondering if some surrounding code has caused the compiler to optimize things differently.  If that's the case, I'm not sure what to do here.

I'm getting really confused by this, actually.  Different runs of the benchmark compare very differently.  For example, I just did a run where the Connect packet decoding showed as improved by 18% (whereas my prior run it had it regressing by 7%).  And then even sometimes during the same run, a decode of a particular packet type will have a wildly different change percent from the decode of that same packet type but with an ack.  I wonder if maybe my laptop isn't "quiet" enough to get good data; perhaps I'll try shutting down X and giving it another run.

I'm getting really confused by this, actually. Different runs of the benchmark compare very differently. For example, I just did a run where the Connect packet decoding showed as improved by 18% (whereas my prior run it had it regressing by 7%). And then even sometimes during the same run, a decode of a particular packet type will have a wildly different change percent from the decode of that same packet type but with an ack. I wonder if maybe my laptop isn't "quiet" enough to get good data; perhaps I'll try shutting down X and giving it another run.

I have aproximately the same weird results (but on a github codespace container so it is not stable). Either the benchmark code is not working properly (but according to me it is correct code) or it is the machine.
Maybe I'll try to benchmark on other machines and check the difference.

Ok, I quit X, ran a basline against main, then ran it against this branch, comparing with main, 3 times.  Numbers are the average change from main:
Encode packet connect on /                        +0.3244%	+1.0224%	+0.2637%
Encode packet connect on /custom_nsp              +7.9232%	-0.1176%	-0.7743%
Encode packet event on /                          +1.6022%	+1.8639%	+2.0130%
Encode packet event on /custom_nsp                +2.6121%	+2.9042%	+3.2108%
Encode packet event with ack on /                 +2.3220%	+1.9855%	+2.2505%
Encode packet event with ack on /custom_nsp       +2.3728%	+2.6042%	+2.4713%
Encode packet ack on /                            +0.9427%	+1.0778%	+1.2282%
Encode packet ack on /custom_nsp                  +2.4452%	+3.6130%	+2.1004%
Encode packet binary event (b64) on /             -40.989%	-43.376%	-43.861%
Encode packet binary event (b64) on /custom_nsp   -41.039%	-43.742%	-43.640%
Encode packet binary ack (b64) on /               -51.311%	-51.461%	-51.580%
Encode packet binary ack (b64) on /custom_nsp     -50.145%	-50.608%	-50.198%
Decode packet connect on /                        -18.877%	-18.676%	-19.289%
Decode packet connect on /custom_nsp              +4.8328%	+5.2217%	+4.9167%
Decode packet event on /                          +1.8359%	+1.3146%	+1.9475%
Decode packet event on /custom_nsp                +5.2678%	+2.1481%	+2.6367%
Decode packet event with ack on /                 +1.7432%	+1.3348%	+1.3040%
Decode packet event with ack on /custom_nsp       +5.2353%	+3.6659%	+2.4317%
Decode packet ack on /                            +5.5802%	+4.6327%	+5.3401%
Decode packet ack on /custom_nsp                  +8.4783%	+8.4328%	+8.2965%
Decode packet binary event (b64) on /             -34.712%	-34.472%	-34.677%
Decode packet binary event (b64) on /custom_nsp   -30.051%	-29.580%	-29.752%
Decode packet binary ack (b64) on /               -36.868%	-37.087%	-37.149%
Decode packet binary ack (b64) on /custom_nsp     -34.869%	-34.986%	-34.482%

With one exception (Encode packet connect on /custom_nsp), it's consistent across all 3 runs.  But there are still some oddities: "Decode packet connect on /" is somehow ~19% faster, even though that code hasn't changed at all from what's on main.  And even stranger, "Decode packet connect on /custom_nsp" is 5% slower, even though a) it does almost the exact same thing as without the custom nsp, and b) that code also hasn't changed from what's on main.
Unfortunately I'm at a loss as to what to do, though.  cargo flamegraph didn't show me anything obvious to target, and the ser/deser for PayloadValue is done in almost the exact same way as for serde_json::Value.  (The one notable difference is that if integer map keys are used, my version's deserializer has to do an extra copy of the string... but all of the tests use string map keys, so that's not what's causing this here.)
One thing that's notable, maybe: PayloadValue is a 56-byte struct, whereas serde_json::Value is only 32 bytes.  Not sure allocation overhead, or perhaps cache locality, might have something to do with it.  The difference appears to be because serde_json::Map<String, Value> only 24 bytes, while HashMap<String, PayloadValue> is 56.  Unfortunately serde_json::Map's methods are only defined when the value type is serde_json::Value, so i couldn't reuse it.

Oh, it looks like serde_json::Map is just a wrapper around either BTreeMap or IndexedMap (depending on if you compile with the preserve_order feature).  If I swap out HashMap for BTreeMap, it shaves the size down, though not all the way: 48 bytes vs. serde_json::Value's size of 32.
We're not going to get all the way down to 32, but I'm surprised it's not at least down to 40 bytes: the Binary variant holds a usize and Bytes; those are 8 and 32 bytes, respectively, which should give us 40 bytes.  I'm surprised it would require any padding for alignment, unless I'm forgetting something and it wants 16-byte alignment for things like usize.
At any rate, 48 bytes is still better than 56.  I'll try to rerun the benchmarks later tonight and see if that changes anything.  Feels like a bit of a long shot, though.

Ok, here are the same numbers as before (against the baseline of the main branch), but now with three more runs with BTreeMap as the holder in the PayloadValue::Object() variant, just like serde_json::Value::Object() uses.
                                                  HashMap	HashMap		HashMap		BTreeMap	BTreeMap	BTreeMap
Encode packet connect on /                        +0.3244%	+1.0224%	+0.2637%	+3.7118%	+0.4929%	-0.2489%
Encode packet connect on /custom_nsp              +7.9232%	-0.1176%	-0.7743%	+29.232%	-1.1302%	+14.978%
Encode packet event on /                          +1.6022%	+1.8639%	+2.0130%	+10.888%	+12.810%	+11.635%
Encode packet event on /custom_nsp                +2.6121%	+2.9042%	+3.2108%	+12.844%	+13.418%	+12.542%
Encode packet event with ack on /                 +2.3220%	+1.9855%	+2.2505%	+12.238%	+14.013%	+12.626%
Encode packet event with ack on /custom_nsp       +2.3728%	+2.6042%	+2.4713%	+12.288%	+13.965%	+13.041%
Encode packet ack on /                            +0.9427%	+1.0778%	+1.2282%	+12.757%	+15.188%	+12.443%
Encode packet ack on /custom_nsp                  +2.4452%	+3.6130%	+2.1004%	+13.417%	+11.715%	+11.726%
Encode packet binary event (b64) on /             -40.989%	-43.376%	-43.861%	-41.263%	-41.204%	-40.748%
Encode packet binary event (b64) on /custom_nsp   -41.039%	-43.742%	-43.640%	-41.187%	-41.331%	-40.808%
Encode packet binary ack (b64) on /               -51.311%	-51.461%	-51.580%	-50.663%	-50.812%	-49.995%
Encode packet binary ack (b64) on /custom_nsp     -50.145%	-50.608%	-50.198%	-49.477%	-49.555%	-49.184%
Decode packet connect on /                        -18.877%	-18.676%	-19.289%	-16.085%	-18.892%	-20.472%
Decode packet connect on /custom_nsp              +4.8328%	+5.2217%	+4.9167%	+4.4243%	+5.0582%	+3.3755%
Decode packet event on /                          +1.8359%	+1.3146%	+1.9475%	+2.2657%	+2.1780%	+2.7651%
Decode packet event on /custom_nsp                +5.2678%	+2.1481%	+2.6367%	+6.3334%	+3.2236%	+3.2511%
Decode packet event with ack on /                 +1.7432%	+1.3348%	+1.3040%	+2.2029%	+1.7505%	+4.1861%
Decode packet event with ack on /custom_nsp       +5.2353%	+3.6659%	+2.4317%	+3.6227%	+4.2471%	+8.0672%
Decode packet ack on /                            +5.5802%	+4.6327%	+5.3401%	+4.1947%	+4.3384%	+5.6753%
Decode packet ack on /custom_nsp                  +8.4783%	+8.4328%	+8.2965%	+2.8838%	+8.2279%	+6.6421%
Decode packet binary event (b64) on /             -34.712%	-34.472%	-34.677%	-34.424%	-34.413%	-32.392%
Decode packet binary event (b64) on /custom_nsp   -30.051%	-29.580%	-29.752%	-29.295%	-29.499%	-27.383%
Decode packet binary ack (b64) on /               -36.868%	-37.087%	-37.149%	-35.033%	-35.360%	-33.529%
Decode packet binary ack (b64) on /custom_nsp     -34.869%	-34.986%	-34.482%	-32.803%	-33.091%	-31.718%

I'm so confused.  Somehow the performance when encoding non-binary events are much much worse, despite now using the same map implementation serde_json::Value uses.  Decoding is... kinda the same?

Ok, here are the same numbers as before (against the baseline of the main branch), but now with three more runs with BTreeMap as the holder in the PayloadValue::Object() variant, just like serde_json::Value::Object() uses.

I don't understand what you changed exactly for this benchmark. It would not suprise me that your custom implementation is slower than serde_json (it has been probably optimized to hell).
What we could do though is only delegate serde to PayloadValue with binary payloads and otherwise use official serde_json::Value for normal events.
Appart from that, I also discovered that serde_json::RawValue exists and that it could be really useful to only partially serde the input payload ([event, ...]) and deserialize the RawValue immediately to T later. But it could be the subject of another PR.

I wonder if it would possible to use a custom with = "binary_deser" module for deserializing binary data rather than reimplementing entire a new PayloadValue. Because 90% of the code is basically a copy of serde_json::Value. What do you think?

I don't understand what you changed exactly for this benchmark.

I changed PayloadValue::Object(HashMap<String, PayloadValue>) to PayloadValue::Object(BTreeMap<String, PayloadValue>).  That mirrors json_value::Value::Object(), which also uses BTreeMap.  I was expecting better numbers after this change, but instead got worse numbers.

What we could do though is only delegate serde to PayloadValue with binary payloads and otherwise use official serde_json::Value for normal events.

Possibly... the downside there is that the public API would then be a little inconsistent.  E.g. Packet::event() would continue to accept a serde_json::Value while Packet::bin_event() would take a PayloadValue.  This would also complicate the internals quite a bit.

Appart from that, I also discovered that serde_json::RawValue exists and that it could be really useful to only partially serde the input payload ([event, ...]) and deserialize the RawValue immediately to T later. But it could be the subject of another PR.

Oh neat, I didn't know about that either.  I'm not sure that would work, though, since you'd need some kind of heterogeneous array/list implementation that is serde-compatible; that is, the example in the RawValue docs works because there's a wrapper object with keys and values.  Not sure how you'd represent something like:
[ String, RawValue(0), RawValue(1), ..., RawValue(n) ]

I wonder if it would possible to use a custom with = "binary_deser" module for deserializing binary data rather than reimplementing entire a new PayloadValue.

I assume you're talking about something that socketioxide would provide, but that users would add to their structs with a serde attribute?  I don't think that would work, since the deserializer would also need access to the list of binary payloads, and there's no way to get extra data to a deserializer specified that way.
Another thing I'm thinking about is sticking with serde_json::Value, and writing a sort of "wrapper deserializer".  Most of it would just delegate to Value's own deserializer implementation, but if we encounter a map with a placeholder in it, it will instead emit a Bytes.  I'm not sure if that's actually possible to do, though.

Another thing I'm thinking about is sticking with serde_json::Value, and writing a sort of "wrapper deserializer". Most of it would just delegate to Value's own deserializer implementation, but if we encounter a map with a placeholder in it, it will instead emit a Bytes. I'm not sure if that's actually possible to do, though.

That could be a good think to explore. For the moment I'm not fond of the entire replacement of serde_json because it is a lot of code and it will probably introduce a lot of complexity to the lib. That's why I won't merge all of this immediately. What we could do is either improve this PR by working on it or if you find completely different possibilities don't hesitate to create a new PR :).
Thanks a lot for all of your work and your time :)!

or the moment I'm not fond of the entire replacement of serde_json because it is a lot of code and it will probably introduce a lot of complexity to the lib.

Yes, I totally agree.  This got pretty large and more complicated than I expected or hoped.

What we could do is either improve this PR by working on it or if you find completely different possibilities don't hesitate to create a new PR

For now I'll probably continue to work here on this PR.  Even if I do something completely different I may just squash it all down.
I've sorta mostly written a ser/deser wrappers for serde_json::Value's serializer and deserializer.  The serializer was very easy, but the deserializer is a bit more tricky.  I'm having some issues with lifetimes (which I've temporarily gotten around by cloning, just so I can move forward to try to prove the concept; but I may need some help there at some point), but it's mostly compiling.  Unfortunately, though, it's still more new code than I'd hoped, as some useful things (like serde_json::value::de::MapKeyDeserizlizer) aren't public, and writing a Deserializer impl isn't trivial.
There is one major downside to this approach: since Value is still the user-facing representation, we can't have a unified text+binary representation in the case where a user maybe doesn't know the full payload format and needs to use Value instead of some struct that implements Deserialize.  So we'll still need the separate Data and Bin extractors.  And some subtle things will change, like the Value passed will need to still include the placeholder objects (otherwise there's no way for the user to know where the payloads go), while the current code removes them.
But overall I think this is a better way to go than PayloadValue.  I don't consider it a total waste of time, as I've learned a lot about how serde & serde_json work internally, and it's been fun to dig into that.

Thanks a lot for all of your work and your time :)!

  Got into this because I needed it for a personal project of mine, and even though it's been a diversion from that project, it's been fun and educational!

Decided to do a new PR after all; see #284.


thread terminated

Description
thread terminated when i wanna change to the Artists List.
Steps to reproduce
open rmpc
press 3
What terminal emulator are you running rmpc in?
wezterm 20240203-110809-5046fc22
rmpc debuginfo
thread 'main task' has overflowed its stack
fatal runtime error: stack overflow
fish: Job 1, 'rmpc' terminated by signal SIGABRT (Abort)
Relevant log output
rmpc.log

I am unable to reproduce this issue. It seems that MPD is closing the connection to rmpc to which rmpc reacts with an attempt to reconnect and this cycle repeats endlessly.
Is rmpc the only client which exhibits this behaviour? What is your MPD config? Do you have max_connections or connection_timeout ?

there is no error when i use ncmpcpp.
the "Queue" and "Directories" are normal, without restart mpd.
my Artists list is very long maybe 1000 item in it.

I tried rmpc with 10 000 artists over network on my old rpi3 and it seems to handle it just fine

10 000 artists in rmpc

You mention MPD restart, does it crash? Is there anything relevant in MPD's logs?

i can debug and reproduce it.
but i'm not a rust coder,
maybe you can tell me how to get more info of it.

Sorry for taking so long to respond, I have merged big changes to how rmpc connects to MPD, can you please try to build from master if anything changed for better or worse on your end?

still the same error.

rmpc.log

Thanks, can you try again from master with https://github.com/mierak/rmpc/pull/170 merged? Should either fix it or give us a better idea. Either way it should stop the endless reconnect loop.

thanks a lot for fix it.
when i use the latest master and open the Artists List.
didn't crash and get this error.

log in /tmp/rmpc.log
2024-12-06T12:54:51.118092660Z ERROR thread=main src/main.rs:572 message="GenericError: 'stream did not contain valid UTF-8'"

fixed


WebSocket can send and receive messages despite being closed

If a WebSocket client's connection is closed by the server after its subscription has been canceled, it's still able to send messages. I haven't read the spec or examined the protocol-level interactions in detail, but this seems wrong.
What's more, Dart's WebSocket server will also receive this message—despite having called both WebSocket.close() and HttpServer.close(). This is clearly wrong, since the documentation of WebSocket.close() says that it "closes the WebSocket connection".
Here's a reproduction:
import 'dart:async';
import 'dart:io';

main() async {
  var server = await HttpServer.bind('localhost', 0);
  server.transform(new WebSocketTransformer()).listen((socket) {
    socket.listen((msg) {
      print("got $msg");

      // As soon as the server's socket receives a message, close the server and
      // the socket. Once these complete there should be no server-side sockets
      // open at all.
      socket.close().then((_) => print("socket closed"));
      server.close().then((_) => print("server closed"));
    });
  });

  var ws = await WebSocket.connect('ws://localhost:${server.port}');

  // Create and cancel a subscription.
  ws.listen(null).cancel();

  // Send a message to the server that will cause it to close the WebSocket
  // connection.
  ws.add("foo");

  // Give the server time to shut down.
  await new Future.delayed(new Duration(seconds: 1));

  // Try to send another message.
  ws.add("bar");
}

I reproduced this as of 77101d63e3c661957b7829dd96d6b4324da0e768.
I'd expect the socket to continue listening to the server behind-the-scenes even after the subscription is cancelled, so that it can determine when the connection is closed.

I can confirm. Moreover, WebSocket.readyState always returns WebSocket.OPEN. https://github.com/dart-lang/sdk/issues/32876

I can confirm as well. I have just stumbled upon it while testing how my Android Flutter app behaves after turning on and off the airplane mode. The socket becomes unusable with no indication of the problem. As mentioned, even the readyState is OPEN.

Actually, my case described above seems to be resolved by reacting to ws.done. I can re-establish the connection then after an offline period.

any updates on this?


Data blobs missing, aborting prune to avoid data loss.

Output of restic version
restic 0.10.0 compiled with go1.15.1 on linux/amd64
How did you run restic exactly?
Environment variables:
# backup directory (ie what to back up)
export BACKUP_SOURCE="/backup_directory/"

# password for client side encryption
export RESTIC_PASSWORD="resticPassword"

# store URL (ie where to back up the data)
export STORE_URL="azure:xxx:/"

# azure credentials
export AZURE_ACCOUNT_NAME="azureName"
export AZURE_ACCOUNT_KEY="azureKey"

We execute:
restic -r "${STORE_URL}" forget --limit-download 200000 --limit-upload 200000 --prune --keep-last 5 --keep-daily 7 --keep-monthly 12 --keep-yearly 5
Output is:
Applying Policy: keep the last 5 snapshots, 7 daily, 12 monthly, 5 yearly snapshots
keep 14 snapshots:
[...]
14 snapshots

remove 1 snapshots:
ID        Time                 Host                  Tags        Paths
----------------------------------------------------------------------
35ce6a3e  2021-01-26 01:39:09  xxx              /backup_directory
----------------------------------------------------------------------
1 snapshots

[0:00] 100.00%  1 / 1 files deleted

counting files in repo
building new index for repo
pack file cannot be listed 267805c0377b66ef496154c315e086bccbf5a581555934c8b38e6d9015b73d1c: storage: service returned error: StatusCode=403, ErrorCode=AuthenticationFailed, ErrorMessage=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:6f99d0eb-c01e-002c-2f06-f91338000000
Time:2021-02-02T01:52:14.6651614Z, RequestInitiated=Tue, 02 Feb 2021 01:52:14 GMT, RequestId=6f99d0eb-c01e-002c-2f06-f91338000000, API Version=, QueryParameterName=, QueryParameterValue=
[16:50] 100.00%  29083 / 29083 packs

repository contains 29082 packs (1574299 blobs) with 140.661 GiB
processed 1574299 blobs: 0 duplicate blobs, 0 B duplicate
load all snapshots
find data that is still in use for 14 snapshots
[0:41] 100.00%  14 / 14 snapshots

Fatal: [<data/e10abcb7> <data/b560d0dc> <data/9bae48c9> <data/9cd925fd> <data/ac5c58ce> <data/08e40397> <data/c97ff7b6> <data/3385e1f2>] not found in the new index
Data blobs seem to be missing, aborting prune to prevent further data loss!
Please report this error (along with the output of the 'prune' run) at
https://github.com/restic/restic/issues/new/choose

The error message is clearly this:
Fatal: [<data/e10abcb7> <data/b560d0dc> <data/9bae48c9> <data/9cd925fd> <data/ac5c58ce> <data/08e40397> <data/c97ff7b6> <data/3385e1f2>] not found in the new index
Data blobs seem to be missing, aborting prune to prevent further data loss!
Please report this error (along with the output of the 'prune' run) at
https://github.com/restic/restic/issues/new/choose

The restic prune outputs this:
repository afe59b03 opened successfully, password is correct
counting files in repo
building new index for repo
[5:14] 100.00%  29083 / 29083 packs
repository contains 29083 packs (1574307 blobs) with 140.665 GiB
processed 1574307 blobs: 0 duplicate blobs, 0 B duplicate
load all snapshots
find data that is still in use for 14 snapshots
[0:47] 100.00%  14 / 14 snapshots
found 1573688 of 1574307 data blobs still in use, removing 619 blobs
will remove 0 invalid files
will delete 0 packs and rewrite 16 packs, this frees 5.141 MiB
[0:53] 100.00%  16 / 16 packs rewritten
counting files in repo
[6:11] 100.00%  29080 / 29080 packs
finding old index files
saved new indexes as [54bda4e1 b7f7e97e d658eb3a bd7428b0 73ece8cb b1ade6d9 0a9baa69 362f56c1 45ddeaae 5fee2104]
remove 12 old index files
[0:00] 100.00%  12 / 12 files deleted
remove 16 old packs
[0:00] 100.00%  16 / 16 files deleted
done

What backend/server/service did you use to store the repository?
Azure Storage
Expected behavior
Working without a crash. (it is the first time that restic is behaving like this)

Actual behavior
The "AuthenticationFailed" is weird because just before, the backup succeeded with the same environment variables. Moreover, it succeeded in listing the snapshots.

Steps to reproduce the behavior
Honestly, I do not know how.

Do you have any idea what may have caused this?
No. (the failed authentication?)
Do you have an idea how to solve the issue?
No and I wonder if I need to disable the tomorrow's backup.
Did restic help you today? Did it make you happy in any way?

The most important part of the error message is actually the error returned while rebuilding the index:
pack file cannot be listed 267805c0377b66ef496154c315e086bccbf5a581555934c8b38e6d9015b73d1c: storage: service returned error: StatusCode=403, ErrorCode=AuthenticationFailed, ErrorMessage=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

That looks like a temporary hick-up especially as it didn't occur again in the second prune run. So maybe a network packet got corrupted. The strange thing here is that restic apparently didn't retry the operation although it should have. Do you know at which time exactly the failing forget run was started?

No and I wonder if I need to disable the tomorrow's backup.

As the prune run after the failing forget did work, it looks like the repository itself is ok. The failure in during forget also happened during the initial integrity check which doesn't modify anything in the repository yet.

Sorry for not replying in time but I do not have the date/time for this failing.
The backup of the day after worked as expected. We had the same issue on another backup on the same day as the one above.
So this issue seemed to be temporary. It has been multiple days that the backup works as expected. So probably a network issue as you said.

Closing as the prune implementation has been rewritten in 0.12.0


Transition data

@ianlongden shouldn't the table to file script read also the new columns (actions and condition)?

They should now but these were only just added to stage and were not there when the script was made/used. So would have crashed earlier on. But I can fix this now.

Thanks @ianlongden !


⚠ Jenkins Plugins Artifactory has degraded performance

In f49020f, Jenkins Plugins Artifactory (https://get.jenkins.io/plugins/artifactory/) experienced degraded performance:

HTTP code: 200
Response time: 1275 ms

Resolved: Jenkins Plugins Artifactory performance has improved in 62e9288 after 32 minutes.


Demo: cross-edge-cloud communication between pods

KubeEdge community meeting Aug 18th, 2021 (Europe Time)
Today my topic is EdgeMesh: Cross-edge-cloud communication between pods demo. And I am here to make a brief summary of what was said at the meeting.
My Cluster Environment:

k8s-master is a k8s control-plane and also a k8s worker. k8s-node1 is a k8s worker. ke-edge1 and ke-edge2 are edge nodes of kubeedge, and they access the Internet through a NAT.
[root@k8s-master ~]# kubectl get nodes
NAME         STATUS   ROLES           AGE     VERSION
k8s-master   Ready    master,worker   3d18h   v1.18.20
k8s-node1    Ready    worker          3d18h   v1.18.20
ke-edge1     Ready    agent,edge      25h     v1.19.3-kubeedge-v1.7.2
ke-edge2     Ready    agent,edge      25h     v1.19.3-kubeedge-v1.7.2

edgemesh-server is deployed on the cloud node, and has a public IP. edgemesh-agent is divided into two parts: cloud and edge.
[root@k8s-master ~]# kubectl get all -n kubeedge
NAME                                   READY   STATUS    RESTARTS   AGE
pod/edgemesh-agent-cloud-dkwwc         1/1     Running   0          74m
pod/edgemesh-agent-cloud-rl6pq         1/1     Running   0          74m
pod/edgemesh-agent-edge-jr5xs          1/1     Running   0          23h
pod/edgemesh-agent-edge-kbzhh          1/1     Running   0          23h
pod/edgemesh-server-7f97d77469-dml4j   1/1     Running   0          83m

NAME                                  DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/edgemesh-agent-cloud   2         2         2       2            2           <none>          74m
daemonset.apps/edgemesh-agent-edge    2         2         2       2            2           <none>          23h

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/edgemesh-server   1/1     1            1           83m

NAME                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/edgemesh-server-7f97d77469   1         1         1       83m

My Test Case

tcp-echo-edge and busybox-edge are deployed in the edgezone namespace. tcp-echo-cloud and busybox-cloud are deployed in the cloudzone namespace. busybox-edge is used to access tcp-echo-cloud-svc. busybox-cloud is used to access tcp-echo-edge-svc.
[root@k8s-master ~]# kubectl get all -n cloudzone
NAME                      READY   STATUS    RESTARTS   AGE
pod/busybox-sleep-cloud   1/1     Running   0          23m
pod/tcp-echo-cloud        1/1     Running   0          23m

NAME                         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/tcp-echo-cloud-svc   ClusterIP   <IP_ADDRESS>   <none>        2701/TCP   23m

[root@k8s-master ~]# kubectl get all -n edgezone
NAME                     READY   STATUS    RESTARTS   AGE
pod/busybox-sleep-edge   1/1     Running   7          23h
pod/tcp-echo-edge        1/1     Running   0          24h

NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/tcp-echo-edge-svc   ClusterIP   <IP_ADDRESS>   <none>        2701/TCP   24h

Start Playing
Cloud access edge
[root@k8s-master ~]# kubectl exec -it pod/busybox-sleep-cloud sh -n cloudzone
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.
/ # telnet tcp-echo-edge-svc.edgezone 2701
Welcome, you are connected to node ke-edge1.
Running on Pod tcp-echo-edge.
In namespace edgezone.
With IP address <IP_ADDRESS>.
Service default.
I'm cloud busybox
I'm cloud busybox

Edge access cloud
[root@ke-edge1 ~]# docker exec -it b4f1820ba1f6 sh
/ # telnet tcp-echo-cloud-svc.cloudzone 2701
Welcome, you are connected to node k8s-master.
Running on Pod tcp-echo-cloud.
In namespace cloudzone.
With IP address <IP_ADDRESS>.
Service default.
I'm edge busybox
I'm edge busybox

Traffic forwarding path
hole-punching successful: busybox --> edgemesh-agent(1) --> edgemesh-agent(2) --> tcp-echo
hole-punching failure: busybox --> edgemesh-agent(1) --> edgemesh-server(reley) --> edgemesh-agent(2) --> tcp-echo
All yaml files in PR: https://github.com/kubeedge/edgemesh/pull/57 🐱🏍

/close

hello, I wonder what are  the differences between k8s-master and k8s-node1? It seems that they are both cloud nodes.
Are there some differences when they are configured? I only configure a kubeedge cluster with the roles of master and edge.


apex_test_tools: 0.0.1-1 in 'dashing/distribution.yaml' [bloom]

Increasing version of package(s) in repository apex_test_tools to 0.0.1-1:

upstream repository: https://gitlab.com/ApexAI/apex_test_tools.git
release repository: https://gitlab.com/ApexAI/apex_test_tools-release.git
distro file: dashing/distribution.yaml
bloom version: 0.9.1
previous version for package: null

Ah merge conflicts. The curse of blooming alphabetically adjacent packages concurrently.

Rebased. Once CI comes back this should be mergeable again. Unless I bungled the rebase.

@nuclearsandwich thanks!


Fix an index error in KernelWordpieceTokenizer

It looks the CI pipeline takes long now, should we move the BuildWheels into another pipeline.

We could also remove the job for python 3.6. Overall, it takes 48 minutes right now.


Typo in import statement

Typo in import statement

CLA signed.

There is a duplicate pull-request of #938

See #938


At the end of the model must have two full-connection layers?

Dear author:
First of all, thanks for your great work!
In the paper i find FC layer is out of model.But in the code,i find the two FC-layer is inside the model. When I want to visualize the yolov4 that has pretrained,I find it dont have two FC behind the model.If I add two FC  which is not pre-trained.It will random initialization what causes each layer of visualization map is not fixed.I don't know if I understand right or do you have any good suggestions.Looking forward to your reply!

Hi, @wwwnnn668,
I don't fully understand what you means, could you clarify? Do you mean in YOLOv4, you have 2 FC layers after the final Conv layer?
I don't think the number of FC layers matters. You only need two steps, extract the feature maps first, then do forward propagation with masked input. Why you need to add two FC layers?

Hi, @wwwnnn668,
I don't fully understand what you means, could you clarify? Do you mean in YOLOv4, you have 2 FC layers after the final Conv layer?
I don't think the number of FC layers matters. You only need two steps, extract the feature maps first, then do forward propagation with masked input. Why you need to add two FC layers?

In your structure of the paper,I find two FC layers were added after the model.In the code ,I find the model output a one-dimensional vector,for example (1000,1).I am wondering whether the model can be finished without FC.

In your structure of the paper,I find two FC layers were added after the model.In the code ,I find the model output a one-dimensional vector,for example (1000,1).I am wondering whether the model can be finished without FC.

If there is no FC, how would you know the confidence score of the target class?

If there is no FC, how would you know the confidence score of the target class?

You mean the model must be followed by FC layer, right?Now I want to visualize the pre-trained YOLOV4 model, which has no FC layer behind it.Therefore, a FC layer must be added to the model, and the parameters of the FC layer are randomly initialized.Does this cause the visual image to change?Thanks for your reply.

You mean the model must be followed by FC layer, right?Now I want to visualize the pre-trained YOLOV4 model, which has no FC layer behind it.Therefore, a FC layer must be added to the model, and the parameters of the FC layer are randomly initialized.Does this cause the visual image to change?Thanks for your reply.

As a detection model, if I remember correctly, it will assign a confidence score for each bbox which indicates the class of this bbox, right? You can use the score here.

As a detection model, if I remember correctly, it will assign a confidence score for each bbox which indicates the class of this bbox, right? You can use the score here.

I see. Thank you.

I see. Thank you.

Closed. Please let me know if you have further questions.

Closed. Please let me know if you have further questions.


how to run this project

please give me environment ,such as the version of pytorch, python, cuda

Thank you for your interest in our paper.  We run the code with torch version: 1.10.0, python version: 3.9.7.

Could you give the version of cuda because it a cuda error that it may be raelated to the version of cuda,thank you!

Could you give the version of cuda because it a cuda error that it may be raelated to the version of cuda,thank you! The version of cuda I use is 11.3, but it reported that "AssertionError: Torch not compiled with CUDA enabled"

I run your project, but it reported that "No such file or directory: '/data/datasets/300K_random_images.npy'" . In your RandomImages class, you set self.data = np.load('/data/datasets/300K_random_images.npy').astype(np.uint8), but  there is no 300K_random_images.npy in your project.

https://people.eecs.berkeley.edu/~hendrycks/300K_random_images.npy


adding example to process Kafka messages using Spark Streaming C# app

along with doc updates and other minor changes

Current coverage is 82.13%

Merging #398 into master will not affect coverage as of 8651317

@@            master    #398   diff @@
======================================
  Files           45      45
  Stmts         4645    4645
  Branches       416     416
  Methods
======================================
  Hit           3815    3815
  Partial        190     190
  Missed         640     640

Review entire Coverage Diff as of 8651317

Powered by Codecov. Updated on successful CI builds.

LGTM


Circular reference when building with Rollup

I'm trying to build my project that uses redux-saga using Rollup, and it reports the following:
(!) Circular dependency: node_modules/redux-saga/es/internal/io.js -> node_modules/redux-saga/es/internal/sagaHelpers/index.js -> node_modules/redux-saga/es/internal/sagaHelpers/takeEvery.js -> node_modules/redux-saga/es/internal/io.js
(!) Circular dependency: node_modules/redux-saga/es/internal/io.js -> node_modules/redux-saga/es/internal/sagaHelpers/index.js -> node_modules/redux-saga/es/internal/sagaHelpers/takeLatest.js -> node_modules/redux-saga/es/internal/io.js
(!) Circular dependency: node_modules/redux-saga/es/internal/io.js -> node_modules/redux-saga/es/internal/sagaHelpers/index.js -> node_modules/redux-saga/es/internal/sagaHelpers/throttle.js -> node_modules/redux-saga/es/internal/io.js

Is this a known issue or something I should not worry about?

It's both 😄 You don't have to worry about - this circular dep shouldn't cause any trouble, but it's also a known issue. Would you maybe like to restructure the project here to get rid of it? We would appreciate the PR!

Ok I'll try to take a look but I'm expecting some redux-saga swag if I fix this! 😆

There is already pending PR (#1453 ) for this by @ziel5122

Ah ok... guess I'll have to wait for my redux-saga coffee mug then.


Set token file mode to 600

This file contains credentials, and should not be readable by other
local users. This fix is UNIX-only, considering that on Windows systems,
a user's home directory should not normally be readable by other users
(whereas that is occasionally the case on UNIX systems, e.g. on Ubuntu).
Fixes this security vulnerability: https://trello.com/c/N7O9D7Dw

cc @fancyremarker

Codecov Report

Merging #231 into master will increase coverage by 0.49%.
The diff coverage is 92.3%.

@@            Coverage Diff             @@
##           master     #231      +/-   ##
==========================================
+ Coverage   93.15%   93.64%   +0.49%
==========================================
  Files          48       49       +1
  Lines        1943     1967      +24
==========================================
+ Hits         1810     1842      +32
+ Misses        133      125       -8

Impacted Files
Coverage Δ

spec/aptible/cli/helpers/token_spec.rb
100% <100%> (ø)

lib/aptible/cli/helpers/token.rb
86.95% <50%> (+39.33%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4dc45ec...dc8098d. Read the comment docs.


Please invite me to the Recode-Hive GitHub Community Organization

Name
Pranathi khandavalli
Discord Username (if applicable)
shinobu0885
Additional Context
i want to contribute

Invitation sent to join the GitHub Organisation. Welcome to the community 🎉
Don't forget after accepting to make it public so it appears on your GitHub profile for everyone else to see. You can do this by finding your name in the GitHub organisation list and change the dropdown to public https://github.com/orgs/Recode-Hive/people


Made listener thread use a queue rather than spinning up many new thr…

Problem
The Airflow task listener collects and emits lineage information in a separate thread that's spun up on each state change event. The thread is terminated after a couple of seconds so that a) the thread doesn't prevent the process from terminating and b) we try to give the thread a chance to actually emit the lineage information.
Unfortunately, as described in #807, the extra thread introduces the possibility of deadlock, especially since we terminate the thread manually without the opportunity to release any locks held.
Closes: #807
Solution
This change alters the way we manage the lineage thread, starting a single thread at the beginning of the process and using a Queue to manage tasks that need to be executed in that thread. It also adds an exit hook that waits for the queue to be empty and then terminates the thread (with a short timeout). Since it's an exit hook, we can be sure that the Airflow task itself has completed before we attempt to terminate the thread. This should completely eliminate the possibility of deadlock.

Note: All schema changes require discussion. Please link the issue for context.

[ ] Your change modifies the core OpenLineage model
[ ] Your change modifies one or more OpenLineage facets

If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports S3 and GCS filesystem operations, tested with AWS EMR).
Checklist

[X] You've signed-off your work
[X] Your pull request title follows our guidelines
[ ] Your changes are accompanied by tests (if relevant)
[X] Your change contains a small diff and is self-contained
[ ] You've updated any relevant documentation (if relevant)
[ ] You've updated the CHANGELOG.md with details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary)
[ ] You've versioned the core OpenLineage model or facets according to SchemaVer (if relevant)

Codecov Report

Merging #820 (3c0d6c5) into main (4830cfd) will decrease coverage by 0.17%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##               main     #820      +/-   ##
============================================
- Coverage     83.59%   83.42%   -0.18%
============================================
  Files            69       49      -20
  Lines          2981     2618     -363
  Branches         19        0      -19
============================================
- Hits           2492     2184     -308
+ Misses          462      434      -28
+ Partials         27        0      -27

Impacted Files
Coverage Δ

...ntegration/airflow/openlineage/airflow/listener.py
0.00% <0.00%> (ø)

integration/dagster/openlineage/dagster/cursor.py

...o/openlineage/client/transports/ConsoleConfig.java

...o/openlineage/client/transports/HttpTransport.java

...penlineage/client/transports/ConsoleTransport.java

...in/java/io/openlineage/client/OpenLineageYaml.java

...openlineage/client/OpenLineageClientException.java

...lineage/client/transports/ApiKeyTokenProvider.java

integration/dagster/openlineage/dagster/utils.py

.../openlineage/client/transports/KafkaTransport.java

... and 11 more

:mega: Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

Also, I would love some kind of test for this. For example, to start non-daemon thread in extractor, that could sleep for like 10 seconds, then return results.

I did go ahead and add a test for this - the sleeping executor was a good idea. I added a 30 second sleep and included the test in the failures suite. Despite the Github report, the latest build did pass - https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/3542/workflows/e06be856-0f9d-40d2-8f48-014df603a85e


Launching the application

Hi Team,
I am getting the following error stack upon running the launcher.
Can some one help me provide a direction to a fix.
Thanks in advance.
C:\Users\ssahoo\Downloads>java -jar excelastic-1.2.6.jar run Launcher
Apr 11, 2018 4:27:01 PM com.codingchili.ApplicationLauncher
INFO: Starting excelastic 1.2.6..
Apr 11, 2018 4:27:01 PM com.codingchili.ApplicationLauncher
INFO: to import files without the web interface use:
Apr 11, 2018 4:27:02 PM com.codingchili.Model.ElasticWriter start
INFO: Started elastic writer. tls = false
Apr 11, 2018 4:27:02 PM com.codingchili.Controller.Website lambda$start$7
INFO: Started website on port 50510
Apr 11, 2018 4:27:02 PM com.codingchili.ApplicationLauncher lambda$new$1
INFO: Successfully started application
Apr 11, 2018 4:27:02 PM com.codingchili.ApplicationLauncher importFile
INFO: Loading file run from filesystem..
Apr 11, 2018 4:27:02 PM com.codingchili.ApplicationLauncher lambda$importFile$4
SEVERE: Failed to load file run
io.vertx.core.file.FileSystemException: java.nio.file.NoSuchFileException: run
at io.vertx.core.file.impl.FileSystemImpl$13.perform(FileSystemImpl.java:740)
at io.vertx.core.file.impl.FileSystemImpl$13.perform(FileSystemImpl.java:732)
at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:275)
at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Unknown Source)
Caused by: java.nio.file.NoSuchFileException: run
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.newByteChannel(Unknown Source)
at java.nio.file.Files.newByteChannel(Unknown Source)
at java.nio.file.Files.newByteChannel(Unknown Source)
at java.nio.file.Files.readAllBytes(Unknown Source)
at io.vertx.core.file.impl.FileSystemImpl$13.perform(FileSystemImpl.java:736)

Hello,
If you start the application with one or more parameters it will attempt to import files from the command line. The following is the syntax to run from the commandline:
java -Xmx1g -jar excelastic-1.2.6.jar <filename> <index> <mapping> --clear
Where mapping and --clear are optional.
To run with the web interface, run the following in your terminal:
java -Xmx1g -jar excelastic-1.2.6.jar
I have updated the readme to clarify this.
Thanks for submitting an issue, let me know if it works out for you.